Contradictory posts

Some people may find my last two posts to be contradictory. In one I argue why randomization is important for studies, in the other I argue we should be careful of randomization as it could be unethical. The point though is that researchers must prove why randomization is necessary, and everyone else should expect a good reason.

So, here are four reasons why a randomized study of NUSAF is not only useful, but necessary:

1. For most districts, we didn’t have to look for over-subscription. At $10,000 for a group of 15 to 30 (equaling more than a years income on average), the money is a lot, and so the demand is very high. The youth program though is just one part of a larger program, and so funds were limited.

2. A randomized study is the ONLY way to know the impact of NUSAF. I am looking into designing a larger impact evaluation of NUSAF without a randomized design (or baseline). I thus need to find an appropriate comparison group, but because the districts have approached the program in their own unique way, combined with the extremely subjective approach to identifying projects to fund, I am having a very hard time. Randomizing after selection into the program presents the only consistent way of identifying a control. The lesson: a consistent and perfectly transparent program would be much easier to design a control group.

3. While a previous post of mine suggested we didn’t get the most vulnerable, I don’t think this is related to over-subscription. All projects went through the same appraisal process, and the non needy were weeded out. I think our failure to target the most needy is due more to the complicated procedures of NUSAF funding, which requires a smarter, more motivated person to get the money.

4. Because of selection issues by the district personnel (i.e., there have been a lot of problems with corruption), randomizing may actually be the fairest way to choose funding.

Never-the-less, I still wish we could have gotten everyone. I have a hard time thinking about those that could have gotten the money, but missed out because their random number didn’t come up.

This is because I have no doubt that the money has a positive effect. Some people in NUSAF though have serious disagreements about the usefulness of the youth program. Unlike some who claim the benefits of randomization, I don’t consider this in itself reason enough to randomize. In many instances, non-randomization can answer our questions as well (though I don’t think this is one of those instances).

I also am not convinced by those that argue randomization helps to identify not just if it works, but why. My one and a half years of experience with NUSAF makes me believe the most important determinants of success are how well the groups get along, leadership quality, facilitator involvement, and basic entrepreneurial knowledge. These things will all be studied through the randomized trial, but in some instances they could be addressed without randomization.