Published in The Edge Malaysia, 6 - 12 April 2015, as "The Counterfactual Factor".
On March 23 2015, the world bid farewell to a true statesman in Mr. Lee Kuan Yew, an iconic human being and one of the greatest politicians of the 20th century. All around the world, people paid tributes and crafted obituaries to Mr. Lee Kuan Yew, focusing mostly on his achievements in the development of Singapore as a first-world nation. This article does not seek to recap those tributes and obituaries; rather, those tributes and obituaries raise an important question in discussing Mr. Lee Kuan Yew’s legacy – what is the counterfactual?
Briefly, in social science speak, the counterfactual scenario is the scenario that would have happened had the ‘treatment’ or ‘program’ or ‘policy’ not taken place. Dan Levy, a senior lecturer at the Harvard Kennedy School, gives a personal angle on the counterfactual. He asks his students, “What would you be doing if you were not at the Kennedy School? That is your counterfactual.” Similarly, each reader may ask herself, “If I were not at my present job right now, what would I be doing?”
The reason the counterfactual is important to know in any empirical research project, especially in the social sciences, is that it gives us a basis in which to compare outcomes of policy. For example, take Teach for Malaysia’s fellowship program, which places highly-qualified young ‘fellows’ as teachers in schools in underserved communities for a period of two years. Consider, for instance, the outcome of average examination scores of students in classes in which a Teach for Malaysia fellow teaches. For us to conclude that that fellow was effective in her work, we should not merely look at whether the examination scores for her students increased, but by how much more they increased relative to the examination results of those students had they not been taught by that Teach for Malaysia fellow. The latter is the counterfactual.
An astute reader would notice something very peculiar, but very true. In real life, we can never observe the counterfactual. We can only observe what happened, and make conjectures on what would have happened. This is the primary problem in empirical social science research on program evaluation – known as the identification problem – and there is a wide variety of ways that economics methodology (econometrics) has introduced that try to replicate the ideal counterfactual. For the purposes of this article, those methods are relatively less important than the general idea of evaluating some outcome against its counterfactual.
What does this have to do with Mr. Lee Kuan Yew? Well, the true measure of how great Mr. Lee Kuan Yew’s achievements are is the difference between the Singapore as we know it today and the Singapore in which some other Prime Minister – the Prime Minister position was always going to be filled, it just happened to be filled by Mr. Lee Kuan Yew – helmed the development of the island nation. In other words, we need to compare how Singapore is today with how Singapore would have been under the stewardship of an average leader.
In basketball, there is a term for this called, “Wins Above Replacement Player” or WARP. WARP tells us how valuable player X is by measuring the number of wins achieved by a team in which X plays (alongside four other average teammates) versus the number of wins that would have been achieved by a team in which a replacement-level player (average player) replaced X. Speaking nerd-ily, it is an incredibly elegant statistic – if some other average player would have generated the number of wins that you generated for your team, how valuable can you really be? On the other end of the spectrum, replace Lionel Messi in Barcelona with some average attacking player like Adam Lallana and watch Barcelona battle it out for a Champions League spot in Spain’s La Liga.
Turning to the case of Malaysia, when we evaluate government policy, we should not evaluate the efficacy of policy simply by what happened, but rather by the difference between what happened and what would have happened in the absence of that policy. Consider, for example, the Economic Transformation Program (“ETP”). On PEMANDU’s ETP website, it states that Malaysia’s GNI per capita has increased from $6,700 in 2009 to $10,600 in 2013, on track to achieving the $15,000 required to become a high-income nation by 2020. That Malaysia’s GNI per capita has increased is very clear, assuming the statistics are right. What is less clear is what the ETP had to do with it.
If Malaysia’s GNI per capita was going to increase from $6,700 to $10,600 without the ETP anyway, then the ETP has been, overall, underwhelming. The government cannot prove the ETP’s effectiveness by simply quoting how much GNI per capita increased. The proof of the ETP’s efficacy is how much higher GNI per capita is today versus a scenario in which there was no ETP. If the economy was going to grow at 5% anyway with or without the ETP, why waste time on the ETP? A rising tide lifts all boats. This is also true of Government Transformation Program indicators. What we need to know before continuing to commit to the ETP and GTP is whether they have added value above and beyond what would have happened in their absence.
It may seem tautological to say that the common is more likely to occur than the uncommon. Yet, it is a concept that is not well-grasped by many, especially those with an agenda to push. Of course the CEO that saw a 30% increase in firm revenues is going to say that it was because of him. But we should think more deeply. If another average CEO took his place, would firm revenues also have increased by 30%? If the answer is yes, then that CEO deserves no additional plaudits than the average. If the answer is no, then he deserves his bonus.
However, given that the common is more likely to occur than the uncommon, we should expect, on average, everything we observe in terms of outcomes to norms rather than exceptions, just by cold, hard probability. In statistics, occurrences that are significantly different from status quo (either positive or negative) occur only 10% of the time. This means that, 90% of the time, we should expect to see outcomes that are not really different from the average outcome. This is true of examination scores, of goals scored, of firm performance, of policy outcomes and of life. Going back to where we started, we can ask if Mr. Lee Kuan Yew was an outlier as a politician or was he not significantly different from the average politician. I would like to suggest that in the case of Mr. Lee Kuan Yew, he was statistically significant. Mr. Lee Kuan Yew was the exception rather than the norm and the world is poorer without him in it. His choices and his actions worked because he was who he was; those same choices and actions, cited as models of governance by many, may not work out as well in the hands of another politician, who – as we know – is likely to be a merely average politician with 90% probability