You may remember our previous article about Actuaries Making Mistakes?
Well, apparently it is not just actuaries and the stakes are quite high in this case. You may have heard about the mistakes made in the Reinhart and Rogoff study, which was uncovered by the Herndon, Ash and Pollin paper, published on April 15, 2013. You can read more details on the discovery here.
To summarise the issue, in 2010 Reinhart and Rogoff, two economists from Harvard, published a paper called “Growth in a Time of Debt.” Their thesis was that more debt is bad for growth of the country’s economy. This thesis was cited many times in the media and during the political debate.
Thomas Herndon, Michael Ash and Robert Pollin of the University of Massachusetts tried to replicate the results and could not. After requesting the original data and calculations from Reinhart and Rogoff, they found some simple Excel errors as well as some omissions of the data that was not disclosed.
What they did:
Selective Exclusions – the dataset contains 110 years of observations for countries that they wanted to observe. However, they only use 96 of those years! The reasons for these exclusions were not disclosed!
Unconventional Weighting – for some reason, the authors gave equal weighing to the countries with various amounts of observations. Perhaps there was a reason for that, but we do not know what is is!
Coding Error – the Excel spreadsheet that they used for their analysis, contained a simple average formula that excludes several cells for no reason. This is clearly an oversight – something that could have been corrected with proper checks and controls.
What can we learn from this mistake:
Disclose – Actuarial Standard of Practice in any jurisdiction always contain sections on Data Quality. In presenting the calculations an actuary must define the scope of the assignment, define the dataset, state its limitations, describe any sampling techniques that were used, describe the reasonableness of the data and state if any alternatives are available and why they were not used. The original authors clearly did not follow this procedure, otherwise they would have seen their omissions or would have to state the reasons for that.
Check for Reasonableness – some checks are easy to implement. Does the sum of reserves by product equal the sum of reserves by lines of business? Does the output, not the formula look sensible? This can frequently be done within the same spreadsheet, by cross referencing with other data sources or by getting a third party view on the outcomes.
Oversight – six eyes > four eyes > two eyes. Yes, 18 eyes are better that two as well, but one has to be practical about this issue. Why weren’t the calculations checked by others? Most of the times, as you are explaining your calculations to others, or even when you are preparing for such explanations, you find some mistakes!
Let’s hope we all can take something away from this expose and we wish you an error-free day!