Present at the Disruption
How Trump Unmade U.S. Foreign Policy
An MRAP (left) and an M-ATV on the ground in Afghanistan. (U.S. Army)
Chris Rohlfs and Ryan Sullivan’s recent article, “The MRAP Boondoggle,” raised questions about the cost and effectiveness of mine-resistant, ambush-protected (MRAP) vehicles in combat. Rohlfs and Sullivan’s argument, which was accompanied by a statistical analysis, claimed that during the late years of the Iraq War, the Pentagon’s decision to replace up-armored Humvees with MRAPs did not appreciably reduce fatalities. Therefore, they concluded, MRAPs were a waste of money.
That assessment is flawed for two primary reasons. First, Rohlfs and Sullivan misconstrued the data available to them. Second, employing those skewed interpretations, they committed methodological errors in their analyses. The result is an inaccurate report on MRAPs in combat. We can set the record straight. Data from the battlefield, as well as the results of live-fire tests (in which vehicles configured for combat are tested against live munitions), demonstrate that, compared to up-armored Humvees, MRAPs save a significant number of lives and, as a result, are worth the cost.
In comparing the two armored vehicles, it is important to review the history of their development. The Humvee is a light, tactical wheeled vehicle originally produced in 1985 without armor or other ballistic protection. The vehicles were meant to provide on- and off-road transport, away from high-intensity combat. In 2004, the Pentagon began rapidly fielding up-armored Humvees to provide better protection against improvised explosive devices (IEDs). Unfortunately, the increased protection could not match the increasing severity of the IED threat from insurgents in Iraq. So late in 2006, the Pentagon initiated the MRAP vehicle program, which U.S. Secretary of Defense Robert Gates subsequently declared to be his highest priority. By the end of 2008, the result was an array of up-armored designs and add-on kits that were more resistant to roadside bombs. Soon thereafter, in the first half of 2009, the U.S. Department of Defense developed the lighter, smaller, more agile MRAP all-terrain vehicle (M-ATV), which continues to be used in Afghanistan today. To date, 27,000 MRAPS have been produced, and some 23,000 have been deployed into combat in Iraq and Afghanistan.
How do we know they are effective? Consider the combat reports. The Anti-Armor Incident Database, which is managed by the Army National Ground Intelligence Center, details casualties broken down by theater, vehicle, and threat type. These reports are classified and were not available to Rohlfs and Sullivan for their analysis. The data from 2005 to 2011 show that the average number of troops killed per IED attack in Iraq was 14 times higher for Humvees than for MRAPs. In Afghanistan, during that same period, Humvee fatalities were nine times higher.
The Pentagon’s matériel, intelligence, and medical communities regularly cross-check what they have observed in the combat zone with live-fire tests; the Pentagon has conducted more than 200 such trials on MRAPs, Humvees, and M-ATVs. In three significant rounds -- first on the MRAPs, conducted between early 2007 and July 2008; second on the up-armored Humvees, conducted between early 2008 and June 2009; and third on the M-ATVs, conducted between early 2009 and March 2010 -- tests assessed the type and severity of injuries that soldiers would suffer from IED attacks, as well as how the vehicles would perform post-attack (i.e., if they could still be maneuvered). The tests have confirmed that the MRAP program saves lives and is worth the cost. They have also helped the Department of Defense continue to improve MRAPs: The latest innovation to be fielded to combat is an underbody improvement kit for the M-ATV that protects against even larger underbody blasts. More than 6,000 of these kits have already been installed on M-ATVs used in Afghanistan.
In combat, MRAPs provided another benefit, too, one that is not as easy to quantify with statistics but is important in modern warfare. As the ability of the vehicles to keep soldiers safe was proved, troops and commanders became more confident. That confidence increased the number of missions that commanders knew they could assign their troops to clear routes safely of possible roadside bombs. This had a multiplier effect, as clearing paths then further reduced attacks on U.S. service personnel in the war.
But Rohlfs and Sullivan miss many of these points. First, they fail to break out fatalities caused specifically by IED attacks on vehicles from fatalities occurring from any other kind of attack. That aggregation introduces statistical noise and renders the data less reliable. For example, a study of the effects of changing from up-armored Humvees to MRAP vehicles should not include statistics on helicopter-related incidents, but Rohlfs and Sullivan’s does.
Another flaw of their study is that they chose to limit their data to Iraq from 2004 to 2010. The authors acknowledge the fact as a problem and write that there is “very suggestive evidence that MRAP vehicles would have been cost-effective had they been introduced during a more combat-intensive period.” Indeed, MRAPs were, as the authors imply, deployed to Iraq at a time when the overall numbers of attacks were declining. And this makes their data selection problems all the worse. With fewer attacks occurring overall, the 14-fold reduction in fatalities that occurred with the arrival of MRAPs looked less significant, since all fatalities were decreasing.
Rohlfs and Sullivan also incorrectly applied complex statistical techniques to their flawed data set. To discern the effects of introducing MRAPs into Iraq on total combat fatalities, they applied seven different models to their numbers. Each posited that total fatalities depended on the quantities of MRAPs and Humvees in Iraq. The models included coefficients related to the differences between fatalities attributed to MRAPs and so-called soft-skin Humvees and the differences between up-armored Humvees and soft-skin Humvees. A coefficient of zero indicated no difference between total fatalities. A nonzero coefficient indicated a difference. The modeling technique that the authors used adjusted those coefficients until the models’ estimates of total fatalities matched, as best they could, their data set. And because their original data set was flawed, so were their findings.
Further, although the authors tested seven models, they never used appropriate statistical techniques to determine which best fits the data. Instead, they combined the results from all of them. Using averages of coefficients across seven models is simply inappropriate, because all those coefficients were derived from the same data. And it is unsurprising that Rohlfs and Sullivan concluded that MRAPs had little effect, since the actual estimates they arrived at using their seven models fluctuated between positive and negative values. These swings should have indicated to them that their models did not capture important causal effects. Instead, they simply averaged the data and got zero.
Finally, Rohlfs and Sullivan exclude nonfatal casualties, so they miss a huge benefit provided by the MRAP vehicle. The live-fire data and combat data highlight that MRAPs provide improved protection over up-armored Humvees in reducing the total number of casualties (both wounded or killed) per attack.
MRAPs save lives. The men and women in combat who use MRAPs -- and their families -- should have no doubt that that is true. The value of MRAPs cannot be measured using inaccurate models applied poorly to the wrong data as part of a flawed economic analysis. Because U.S. combat troops know MRAPs save lives from their own experience on the battlefield, they have the confidence in their equipment to accomplish their missions in Afghanistan. The Department of Defense is applying the many lessons learned from developing and testing MRAPs to ensure that its combat vehicles protect our troops, enabling them to accomplish whatever tasks they must undertake. And the facts on the ground offer another lesson learned: MRAPs have been well worth what they cost.