New approach to emissions auditing: More honest reports, lower emissions

Overview

An MIT-Harvard study has provided the first-ever evidence of the impact of removing the conflict of interest that arises when regulated firms hire their own auditors—a common practice in many industries. The researchers selected nearly 500 polluting industrial plants in India, and the state’s environmental regulatory body changed the rules for half of them: It randomly assigned auditors to plants, paid them a set fee, and double-checked a sampling of audit reports. Among the striking results: Almost a third of the standard audits falsely reported legal levels of emissions—but under the new rules, the number of dishonest reports dropped by 80%. Best of all, plants responded to the new system by cutting pollution. At least in this case, reform of third-party auditing could lead to more effective regulation.

---

In many parts of the world, rapid industrial development relieves poverty but brings serious air and water pollution. The state of Gujarat in western India is a good example. After two decades of rapid growth in productivity, Gujarat now generates about a fifth of India’s manufacturing output—but the region’s rivers are heavily polluted, and the air in its cities is harmful to human health.

In 1996, the Gujarat Pollution Control Board (GPCB)—the state’s environmental regulatory body—attempted to curb emissions from its 20,000 industrial facilities by instituting a third-party audit system in which an external auditing firm measures air and water emissions at each plant three times a year and submits an annual report to the GPCB. In response to reported transgressions, the GPCB punishes violators at varying levels of severity—from issuing warnings and fines to cutting off water and electricity. Even so, high pollution has persisted.

In 2009, this problem came to the attention of a group of MIT and Harvard researchers affiliated with the Abdul Latif Jameel Poverty Action Lab, or J-PAL, a research network housed within MIT’s Department of Economics. In initial conversations, the researchers learned that the GPCB believed that the current audit system produced unreliable information. The suspected source of the problem? The polluting firms hire the companies that audit their firms’ operations, so there’s a built-in conflict of interest for the auditors. If they skew their findings to suit the firms, they’re more likely to get hired again—and the firms avoid any threat of being penalized for excessive emissions.

“Allowing the entity being audited to hire its own auditor seems a strange arrangement, but it’s common practice—pervasive in corporate financial reporting, the global debt-rating system, health and safety regulation, and many other areas,” says Nicholas Ryan PhD ’12, now a Prize Fellow in Economics at Harvard and J-PAL. Despite periodic calls for reform in various settings, no changes have been enacted to make third-party auditors more independent.

Using Gujarat as a case study, the J-PAL team decided to take the first quantitative look at the third-party auditing system, including the sources and magnitude of problems and their impact on compliance. “While third-party auditing might seem difficult to assess objectively, in J-PAL we specialize in performing such studies,” says Esther Duflo, the Abdul Latif Jameel Professor of Poverty Alleviation and Development Economics and a founder and director of J-PAL. “We use rigorous scientific methods to help us understand exactly what causes such programs to have limited effectiveness and what strategies can make them more effective.”

Assessing the problem

Key to J-PAL studies are randomized controlled trials. Often used in testing pharmaceuticals, the technique involves applying changed conditions—whether a new medicine or a new regulatory procedure—to one of two otherwise identical groups and then comparing the outcomes for the two groups. Working closely with the GPCB, the J-PAL researchers set up the following two-year field experiment to run in Ahmedabad and Surat, Gujarat’s two largest cities.

First they identified a sample of 473 industrial plants. Then they randomly assigned 233 plants to serve as the “treatment” group and kept the remaining 240 as the “control” group. The control firms continued to hire and pay their own auditors, while the treatment firms were subject to a new auditing procedure. Each treatment firm was randomly assigned an auditor, and the assigned auditor was paid a flat fee from a central pool to perform the audit. In 20% of all cases (selected at random), the auditor’s visit was followed within a few weeks by an unannounced visit from a “backchecker”—an independent company that performed the same measurements, thereby checking the accuracy of the auditor’s report. Finally, at the beginning of the second year, the auditors were notified that their pay would be linked to the accuracy of their reports, as measured by the backcheckers’ reports.

The new arrangement altered the incentives of all parties. “With the new procedure, the auditor has no reason to misrepresent the findings to benefit the plant because the plant has no influence over the auditor’s subsequent assignments,” says Ryan. “Set fees for the audits are important here, so assigned auditors can’t extort firms, and the auditor receives extra pay for doing a good job.” Best of all, faced with the possibility of an honest audit, the plants should be more motivated to clean up their emissions to avoid being penalized by the regulator.

Outcomes

Results of the experiment confirm that false reporting is common in the control group, where auditors are hired by firms. For all pollutants, 29% of the audits in the control group reported that emissions were below legal limits when in fact they weren’t. And they systematically reported plant emissions just below the legal limit—levels that are presumably less likely to attract regulatory attention than would be reports showing compliance by a wide margin.

Auditors for plants in the treatment group reported more truthfully. When the treatment auditing method was used, inaccurate reports of plants complying with the law dropped by about 80%. “So it’s not that the auditors are now perfectly telling the truth, but they’re more than three-quarters of the way there,” says Ryan.

A review of the behavior of auditors who worked under both regimes shows that they reported pollution more accurately at treatment plants than they did at control plants that they were auditing at the same time. That observation suggests that the increased accuracy was due to the new audit system. Another factor may be how much the auditors were paid. The fixed fee under the modified system was 40,000 rupees—according to the regulator, enough to cover the cost of performing the measurements plus a small profit. In contrast, auditors who negotiate their own fees are paid on average 24,000 rupees per audit. At that level, they could not conduct all the tests needed for a proper audit.

The figures above present more detailed results for suspended particulate matter (SPM), measured partway through the second year. On each graph, the vertical line marks the regulatory standard of 150 milligrams per cubic meter, and the gray shading shows the region between 75% of the standard and the standard—the zone where false reports might not look too conspicuous.

The two charts on the left show SPM measurements from audit reports (top) and corresponding backchecker reports (bottom) at the control plants. According to the auditors, 93% of all plants are compliant, and 73% of them fall within the gray shaded area just under the pollution standard. Data from the backcheckers look quite different. Here, 59% of the readings exceed the standard—as opposed to just 7% in the audit reports—and there is significant dispersion in the distribution. Only 19% of the plants have readings in that narrow gray band—54 percentage points less than claimed by auditors. Taken together, these data provide striking evidence that auditors fabricate SPM data so that plants look narrowly compliant with the regulatory standard.

The right-hand charts show measurements by auditors at the treatment plants (top) and their backcheckers (bottom). The distributions of the audit and backcheck data now look more alike. The increased similarity is especially evident for readings above the standard, which were very sparse among the control audits. In the auditors’ data for treatment plants, 39% of the readings are within the gray area just below the standard—well below the 73% in the control group reports. But while the treatment increased truth-telling by auditors, it did not end false reporting. The 39% share of audit readings in the shaded area still exceeds the 14% share in the backcheck measurements.

Changes in pollution

“The ultimate goal of improving auditor honesty is, of course, to motivate individual plants to reduce pollution,” says Duflo. And measurements taken six months after the final audits show that the treatment plants did cut their emissions relative to the control plants. The figure on the right shows changes in pollutant levels using units of standard deviation—a method of accounting for all air and water pollutants in a single index. Considering all pollutants, the treatment plants reduced their emissions by a statistically significant 0.211 standard deviations on average. That overall effect was driven by a large decrease in concentrations of water pollutants—a top regulatory priority for the GPCB. The pollution reductions occurred mainly at the highest-polluting plants— the ones that would suffer the harshest penalties if they were caught violating the standards and might end up paying more to deal with regulatory sanctions than they would to abate pollution.

Another question is whether the benefits from such pollution reduction outweigh the extra costs of the new auditing system. Performing a cost-benefit analysis requires putting a monetary value on reductions in pollution. India has not set prices on emissions, so as a first cut, the researchers used emissions prices in China. Based on the data they have available thus far, they calculate a net gain of US$6,000 at each treatment plant. While that result is not definitive, it does suggest that the benefits of the new auditing procedure may outweigh the costs.

Encouraging regulatory change

The findings of the study have already begun to spur change in Gujarat. The GPCB has approved the adoption of a new audit system modeled on the one used in the field experiment. Implementation of the recommended changes awaits approval by the supreme court of Gujarat—the entity that originally issued the call for better regulation that led to adoption of the third-party auditing system in 1996.

Will the Gujarat experiment also motivate reform in other settings? There are enormous differences between auditing financial reports or bond ratings and auditing industrial pollution in Gujarat. But whenever the company being audited hires its own auditor, conflict of interest is present. The researchers hope that regulators and policy makers will at least take notice of their findings. They note that the experiment underscores the value of designing third-party audits so that the first-order incentive is for auditors to tell the truth. And they conclude that—at least in this one instance—it was possible to do so.

---

This research was supported by the MIT Center for Energy and Environmental Policy Research, the Harvard Environmental Economics Program, the International Growth Centre, the International Initiative for Impact Evaluation, the National Science Foundation, and the Sustainability Science Program at Harvard. Further information can be found in:

E. Duflo, M. Greenstone, R. Pande, and N. Ryan. “Truth-telling by third-party auditors and the response of polluting firms: Experimental evidence from India.” The Quarterly Journal of Economics, pages 1499–1545, 2013.

---