With GiveWell's Charity Reviews due in early July, the lack of available evaluations of aid agency work is once again highlighted.
After the tsunami in Thailand over 200 aid agencies of all types and sizes flooded into the area to help out. Of those 200 agencies, I know of less than a handful that actually did an independent evaluation of their work, and none of these were made available to their donors. I also saw several questionable aid projects that were either not evaluated, or if evaluated, the results were not made available to the public.
Case #1 - Proof of projects was required after a spot check revealed incorrect reporting
An international aid agency was required to provide proof that a large percentage of projects had actually been implemented after a spot check by head quarters (HQ) revealed that many of the projects reported as completed hadn't even been started yet.
This example highlights the problem with aid agency reporting. Very rarely are reports actually verified, even within the agency itself. The lack of verification coupled with the pressure to show positive results creates the possibility for creative reporting at every level of an organization. Field staff are motivated by a desire to please their supervisors, and not get fired. Managers are motivated by the requirement to meet goals and deadlines. Country directors are motivated by a desire not to lose their country's funding and to keep HQ at bay. And HQ is motivated by the desire to please donors. Without independent verification, false information could all too easily occur.
Case #2 - An independent evaluation was completed but kept in-house
In another instance, an international aid agency paid for an independent evaluation of it's work. However, the report's negative findings resulted in the organization limiting the report to "internal use only". There was also wild speculation amongst the staff that the report was softened by the researchers after pressure from senior management.
By keeping the report for internal use only, donors are prevented from knowing what the mistakes were and whether or not management had adequately addressing any findings. In addition, donors will not be able to hold the aid agency accountable for improved practices in future aid efforts.
Case #3 - The evaluation was sunk to prevent negative findings from affecting future funding
An organization actually deep-sixed an entire evaluation when findings showed considerable problems. This was done because the implementing partner, part of the Thai government, was afraid that the report would prevent it from receiving future funding. After attempting to significantly alter the report, the government office convinced the funding agency to not publish the report. Instead of improving the quality of the project based on the findings, the findings were simply put aside for political reasons.
Case # 4 - Two instance of poor construction of temporary housing, no apparent evaluations
One international aid agency built several temporary camps for tsunami survivors. Temporary housing consisted of a row of one room houses built on a slabs of cement. Bathrooms were located either off the backside of each room or in a communal area.
In one camp the cement slabs were laid flush with the ground, this caused the rooms to flood during each heavy rain storm. Because the disaster victims had lost all their furniture, their personal belongings were stacked on the floor, this meant they were damaged each time the place flooded. Another aid agency had to build a series of trenches around the temporary housing to prevent flooding.
This same agency had problems with another temporary camp it built. There the sewage system was inadequate for the population. After a few months, raw sewage began to flow in the trenches surrounding the houses. The aid agency that built the camp refused to pay to improve the system because they said the government was now responsible for the camp.The government office overseeing the camp did not have enough available funds to fix the problem. Eventually another party had to be brought in to pay for improving the sewage system.
The website of the aid agency that built both temporary camps does not include any mid-project or end-of-project evaluations. The aid agency website did, however, publish many "pretty stories" as well as requests for funding other projects. Because no evaluations are on the website, I question whether the aid agency has learned from it's mistakes. I do know that the donor public has not.
Case #5 - Project fails at changing traditional habits, causes additional problems
In another instance an international aid agency built houses for Moken people (these are traditional migrant populations that live on their boats and sail up and down the coast between Myanmar and Malaysia during part of the year, and have recently been living on land during the monsoon season). Moken houses were traditionally built on tall stilts over the high tide mark. Mokens used one corner of the house for the bathroom, with everything falling through the slats. The refuse was washed away with each high tide.
The agency built the new houses further up shore, away from high tide. To compensate, a row of toilets were built behind the houses. However, culture does not change that easily. The toilet rooms were used as storage sheds for donated goods, and the Moken continued going to the bathroom in the corner of their houses. However, the high tide could no longer wash away the refuse, leading to problems with insects and concerns over sanitation. Another aid agency sent staff regularly to spray insecticides and other chemicals below the houses to try and prevent the spread of disease.
This aid agency's website makes some project evaluations available, as well as several filmed "pretty stories". However, I was unable to find any evaluation of work in Thailand after the tsunami.
Case # 6 - Houses built without local consultation
A third instance of poor housing design was by an international an aid agency that built houses without consulting the local people. The houses were two parallel rooms on either side of an uncovered cement pad. Because that region of Thailand has an eight month long rainy season, this meant that the houses were continually flooded as rain poured off the roof onto the cement slab and into the rooms. Also, walking from room to room meant going out into the rain.
Another aid agency had to be brought in to build gutters along the roofs to try to decrease the flooding. Unfortunately, although the gutters improved things a little, they still did not address the problem with the center of the house being open to the rain. Also, the gutters hung so low you had to duck to get in or out of the house.
This aid agency's website had an impressive number of mid-project and end-of-project evaluations. However, there was no evaluations of the work in Thailand.
Why do so many aid agencies either not evaluate their projects or hide negative findings?
There are two main factors. First, a poor evaluation may hurt an aid agency's chances of receiving funding in the future. All aid agencies are essentially in competition for funding. By making negative findings public, an aid agency risks losing funding to an aid agency that does not share it's negative findings. Second, evaluations can be expensive and can be difficult to fund because donors often prioritize low operational expenses when choosing aid agencies.
This dearth of accurate and available project assessments has several negative consequences. Donors have no way of judging the quality of an aid agency's work, nor are they able to hold the aid agency accountable for improving it's practices. Additionally, a lack of accurate research makes it very difficult to establish industry best practices because so little information is available to researchers and practitioners.
Donors are key to making project evaluations a standard part of every project and to improving aid practices.
If donors place a priority on independent evaluations - and fund them - then aid agency practices will change. Before giving, donors should request findings from past evaluations. If an aid agency does not share these findings it means that either they have not been evaluating their work, or are unwilling to share any negative findings. If a donor decides to fund an agency, the cost of evaluating that project should be factored into the project budget and donors should expect to pay for it.
Comments