(Via the Bad Astronomy Blog of all places.)

Well, Kelsey, you can believe as you like, but the question at hand is whether or not that’s something that has been proven by this study. And while it was touted as proof that AOE does not work, closer examination indicates that there are a lot of questions regarding it’s validity that don’t have good (or at the very least, clear) answers.

In fact the study reported mostly identical numbers for both groups for nearly every characteristic—no variation at all, random or otherwise. This result could only be obtained by manipulation whether accidental or intentional.

You are absolutely correct. This is intentional, and they describe within the study report what the manipulations are: weighted regression and estimation.

First, the results are biased using weighting methods which cause the figures to appear to be less variable than they are. This is done to highlight statistically important variations, causing the variations you describe to disappear.

Second, thoughout virtually the entire study, and in particular in the Executive Summary where all the figures you refer to are found, you are not looking at the statistical data at all. You are looking at “Estimated Impacts”. The studies first collected the data, weighted the results to smooth variation due to random factors, then used these results to create *estimates* of the effectiveness of the results. This is what you are looking at.

This results in figures in which all unimportant statistical variances, such as the ones you rightly point out should (and do) exist in the data itself, are deliberately eliminated.

For a table that shows the variances in the actual data, rather than the estimated impacts, see pg. 112 of the original study, Table A.14: Impacts on Perceived Effectiveness of Birth Control Pills for Preventing Pregnancy, Overall and by Site. Note that these are still weighted figures. However, despite the weighted figures, this shows variances of up to 16% between the two groups.

Didn’t it strike you as odd that all the results were the same?

The study involved a total of 2,057 students in both the control and program groups, about equally divided, which means that a random fluctuation in some result by ten students would shift the reported figures for a particular characteristic by one percentage point. Even if the populations were completely identical, one would expect variations of several percent in the results obtained based solely on chance.

Even if two samples were accidentally drawn from the same group one would expect variations of several percent in the results obtained based solely on chance.

In fact the study reported mostly identical numbers for both groups for nearly every characteristic—no variation at all, random or otherwise. This result could only be obtained by manipulation whether accidental or intentional.

Examples, the percentage or scale mean for the program group comes first, then the number for the control group: (some of this data was in the first post in this thread)

Remained Abstinent Always: 49%, 49%
Abstinent Last 12 Months: 56 %, 55%
Had Sex-Always Used Condom: 23%, 23%
Sometimes Used Condom: 17%, 17%
Never Used Condom: 4%, 4%
One Partner: 16%, 16%
Two Partners: 11%, 11%
Three Partners: 8%, 8%
Four or more Partners: 17%, 16%
Identification of STD risks: .69, .67 Scale Mean
Knowledge of Unprotected Sex Risks: .88, .88 Scale Mean
Knowledge of STD Consequences: .52, .51

Perceived Effectiveness of Condoms:
Usually: 51%, 52%
Sometimes: 38%, 38%
Never: 3%, 3%
Unsure: 7%, 7%

And so forth. Almost all of the numbers are the same in both groups.

Something is really fishy with this study.

Assuming that 49% is the correct number for the populations that these groups represented, what is the probability that a random sample of a little over one thousand members would return a result of 49%?

This is pretty close to a coin toss, so just simplify the problem by assuming that exactly 50% of both populations were abstinent. If you threw a thousand coins, what are the chances that you would come up with 495 to 505 heads, inclusive?

And then what’s the probability that you would come up with 495 to 505 heads if you did it again?

You can try it at this website: http://www.matti.usu.edu/nlvm/nav/frames_asid_305_g_3_t_5.html

More formally, as Newton explained over 300 years ago, you can determine the probability of coming up with 495 to 505 heads in 1000 tosses by expanding the binomial (x y)^1000, summing the coefficients of terms 496 to 506, and dividing by the total of the coefficients.

You would come up with 495 to 505 heads about 28% of the time. The probability of obtaining this result twice in a row when the sample size is 1,000 is .28*.28 = about 8% or about one chance in twelve.

But, in the first sixteen results reported in the Mathematica study, eleven were identical, four differed by one point, and only one by two points. The probability that these results would occur in random samples of this size is on the order of .08^16 or about 3 chances in 100 million billion (10^17) trials.

This is essentially an impossibility. The study results could not have occurred in the way described in the report.

I suspect that they screwed up and entered the same data twice. What little variation exists is because of other errors in data input, processing or rounding errors.

I find the whole thing kind of disturbing. Like the Emperor has no clothes.

I think that even if we like the results of this study, it’s better to know the truth–they screwed up royally.

Bob Unferth

And for that matter, it’s a lot harder to get rid of a teacher, staff member or principal.

I’ve said it before, and I’ll say it again; the biggest problem with the public school systems is the home environment. Teachers and schools can help but ultimately cannot overcome the issues of children that come to school unprepared to learn.

a running experiment which compares schools without federal interference with ones that do have federal interference. It’s called “private schools”

As hf points out, one difference between private and public schools besides how much “federal interference” they have is the different admission standards.

Another is that private schools can kick out students who misbehave or simply don’t measure up academically. It is much harder to expel a student from public school.

One could fill a book with the differences between private and public schools, without once resorting to anything having to do with the role of the federal government in the latter.

Amy
another herder of cats

Actually, what’s at issue is not that the programs refuse to be evaluated, but that the schools using the programs and the group of kids in them had to fit certain not-well-described study criteria.

I’d accept this study as valid for the programs and locations it covered, but extrapolating it to 7 other programs that weren’t looked at seems a bit of a stretch.

essentially, “they chose only 4 of the 11 commonly used programs and excluded ones that has research backing up their effectivness” still needs to be addressed.

Pretty much what everyone so far has said seems reasonable to me, but I’d also like to add that claiming, in effect, “Hey, so it’s just a coincidence that every program they chose to examine (over 1/3 of our total programs) was ineffective. The others are totally useful,” seems kind of . . . not . . . good.

I mean, “Some of our programs aren’t useless!” Isn’t exactly the greatest rallying cry in history.

—Myca

The programs to be evaluated were chosen in advance and therefore could not be specifically chosen for their efficacy or lack thereof: no one knew apriori how effective they were likely to be. If you have any evidence that any other AOE program is more effective, please cite it because all the literature that I’ve previously read suggests that AOE is not just null value but acutally dangerous in terms of increasing the number of teens engaging in risky sex, without actually reducing the number engaging in sex of any sort. So if anything I am suspicious of this study because the AOE appears to do better than the literature suggests that it should.

Not that the mess the Bush administration created in 8 years is going to get cleaned up quickly, but at least it will *hopefully* stop getting worse once he’s out of office.

I’m beginning to second guess my decision NOT to move to Canada after the last election.

Why anyone with the least bit of common sense would think abstinence ONLY education is an effective tool for fighting STDs and unwanted pregnancy is beyond me. I am not coming at this from an academic perspective, but rather as one from my own experiences as a teenager and with Sex ed. Annecdotal evidence/personal experience opinion all the way - take it for what it’s worth.

I had pretty comprehensive Sex Ed. Age appropriate sex education began in 5th grade and continued through high school. I didn’t run out to have sex because of it, nor did most of my peers. In fact, I didn’t have intercourse until I had graduated college - and I wasn’t the only one of my peers to do that and none of us are/were particularly pious and religious. But the ones who WERE having sex (who would have done so anyway more than likely given historical estimates of rates of teen sexual behavior) certainly were having safer sex than they otherwise might have.

At one point, the student council of my HS lobbied to put condom machines in the school bathrooms at the high school - they made a compelling argument (which was squashed of course), but the fact that they were even able to voice the argument to begin with was pretty good progress I think - and that was over 10 years ago.

For the record, I’m not a person who thinks teenagers should be having sex. It may in fact be my most conservative viewpoint. I just don’t think before the age of 18-20 or so most people are really able to competently handle the mental, emotional and physical consequences of sex. And I plan on teaching my daughter about the importance of waiting until she’s really ready, but I’m also not going to refrain from teaching her about the importance of proper condom use and other birth control. I suspect I’ll even discuss/demonstrate the proper application of a condom - can’t say if I really will or not, she’s only two, so who knows, but I’d like to think I will. That being said, I think it is the job of parents to instill the confidence and sense of self worth required to be able to abstain from sex in adolescence.

The school’s job is to educate - and basic reproductive knowledge and understanding how to prevent disease and unwanted reproduction should be part of every basic Health class. Isn’t that common sense?

This report focuses on 4 of these 11 programs. These four programs are called impact sites because they had program features and staff capable of supporting a rigorous, experimental-design impact evaluation.

What this means is that these four programs were chosen because they fit the specifications of the Mathematica Policy Research design. To say it another way, these four programs were chosen because they were situated in schools/communities which had the staff and the diversity which Mathematica needed to carry out the study.

Other programs with established and successful research to prove their effectiveness were not chosen because they did not happen to fit the research design.

Yes, but Mathematica’s research design is the absolute best research design; random assignment into program groups and control groups, with a demographically diverse group of students. It’s not unreasonable for Mathematica to design their study to use the most rigorous methods, nor does it show that they were biased, except insofar as “best research design” is a bias.

If the “other programs” are not willing or able to be evaluated using random-assignment of a diverse group of students, then I question whether those “other programs” have really proven themselves to be worthwhile and effective.