The Wondering Mind

The NY Times had a wonderful article last week on the Section 8 housing program and how it’s not really working as people expected. I’m not sure if the article was inspired by a story in the Atlantic Monthly, but that piece is also an excellent report on Section 8.

The Atlantic Monthly story is like a little history lesson on the development of econometrics in the last several decades and the limitations of using econometrics in social sciences. The cited study by James Rosenbaum appears to suffer from many of the limitations of econometrics my professors admonished me against in my econometrics and economics classes.

First time the study is mentioned, the Atlantic article mentions that the study started with 114 families, but were left with only 68 by the time the study was released. Researchers are now generally very leery of studies with such high attrition rates for good reason. Chances are good that the reasons for the attrition are not random, meaning that the types of families who dropped out are likely to be systematically different from those who did not. It’s extremely difficult to estimate how the results would have been had you not experienced such attrition.

It is also possible that such problems of self-selection existed in the selection of the original sample of families. That is, the families who participated may have been more motivated to move, and more motivated to do something to better their lives than the “average” family who now qualify for Section 8. Obviously, you might expect such a motivated family to be different from the average family and for the effects of Section 8 on them to be different also.

Examples of such limitations are documented in the piece, with later studies showing that the initial euphoria over the results of the original study were premature and not accurate. One sentence is especially telling:

For instance, while Gautreaux study families who had moved to the suburbs were more likely to work than a control group who stayed in the city, they actually worked less than before they had moved.

This tells us that it is more than just possible that the apparent effects observed in the results of the study of Section 8 policies were probably more a function of the characteristics of the participating families, rather than being the result of the policies.

These types of studies are notoriously difficult to conduct well because of the problem of obtaining good counterfactuals. Counterfactuals are the controls you use to test the effect of your policies on the treatment group. The easiest illustration of this idea is in trials of medicines. In such trials, a group of patients are randomly assigned to receive the real drug or to receive a placebo (a “dummy” drug that has no medicinal effect but which is otherwise indistinguishable to a casual observer, such as the patient). You compare the difference between the patients given the drug versus those patients given the placebo to try to ascertain the effects of the drug.

The problem with social science is that social experiments are very difficult to conduct. For one thing, the subjects knowing that they are part of an experiment tends to contaminate the results. And it is rare that natural conditions exist which proximate a good experiment (what social scientists call a natural experiment). For example, if you change a law or regulation and observe how people react, it is often difficult to know how they would have acted if nothing had happened (the counterfactual).

For these reasons, the original study appears to have been not as strong as the initial reaction to it might have you believe.

(To be fair, I have not read the original study. It is published in a book, and I currently have neither the time nor the money to buy and read it. So it is entirely possible that the authors were well aware of the limitations of their study and pointed them out; others may have simply chosen to ignore them. It is also possible that social scientists at the time were either unaware of these limitations in econometrics, or they knew but had no better techniques to use and people got carried away with the results.)

Which tells us why perhaps the Section 8 program was never going to have the spectacular results its advocates expected. But that’s not the whole story. As Rosenbaum says,

“People were moved too quickly, without any planning, and without any thought about where they would live, and how it would affect the families or the places.”

As the Atlantic Monthly article and the NY Times articles both illustrate, the Section 8 program pretty much just gives poor families a housing subsidy, and not much else. They do not provide any additional support programs to help the welfare recipients to get jobs, or get better jobs, support for the family, etc. The attitude seems to be, “we’ve taken you out of the ghettos, so now pick yourselves up and join the middle classes.”

I’m not sure when or why we started to believe that we could do welfare on the cheap, but this is an appallingly lazy, and bad, way to practice public policy.