A reader builds on these dissents:
I can barely contain my discontent with the Red Cross study you featured. I am teaching a statistics class this semester and I was in the middle of prepping my lecture on sampling when I took a break and saw the post. I have to say the study will be a perfect example for my class on how NOT to sample or interpret survey results.
So, the actual Red Cross post on the study starts by chastising the "youth" of our country for having such wrong, no-good, immoral preferences when it comes to torture and that they do not know what the Geneva Convention is. It moves on to mock them because they have the audacity to believe people should be educated before they can vote and proceeds to lecture us, and the youth, on the importance of the Geneva Convention. It ends with some psycho-blabber on why youth these days are so messed up. Of course, it's because they were exposed to horrible, terrible things, which makes them hold horrible, terrible beliefs about the morality of torture and such.
Yes, it takes a long time to get to the point, because it takes the "study" a long time to tell me what kind of "youth" we are talking about. It's 12- to 17 year olds! Yes, we are made to read through all that moralizing and finger wagging, add some Dr. Phill theorizing, to finally learn that the sample is composed of 12- to 17 year olds, it is half of the sample size of the adults surveyed and, as a result, has a wider margin of error (+/- 4.4 as opposed to +/- 3.1). I don't need to be a statistician to know that you cannot compare the two samples that straightforwardly!
In fact, as I will explain to the "youth" in my research methods class tomorrow, they would have to at least account for the different sample sizes in their interpretation, if not do the correct and decent thing and conduct some significance testing – like simple z-score, which my students have learned all about yesterday and will calculate tomorrow in class. My guess is that if you actually take the time to analyze the results properly, the difference between the responses of the two groups will come out as barely statistically significant.
I mean, just eye-ball it! If you take 4.4 percent out of the "youth" answers in favor of torture, you pretty much get very close to the response rate of the adults. If you keep playing around with the margin of error like that, you will notice that more often than not the confidence interval between the two results overlap. That tells you that more often then not, the results cannot be distinguished from one another. Which is actually pretty darn surprising, given what I will say below.
Aside from the boring statistical stuff above (which my students will get tomorrow anyway), how wise is it to draw such harsh conclusions based on answers given by 12- to 17 year olds on such an abstract topic? Really, what do you expect to get? If you expect to get some meaningful, fully formed, stable statement of preference, you have obviously managed to avoid meeting "youth" in your adult life and/or stay away from serious research on child development. Has anyone taken the time to notice how odd it is that the "No opinion/Dk" category is selected by only 1% of 12- to 17 year olds in the sample? These "youth" do what all youth do when confronted by a grown-up and asked about some "important" grown-up issue: they give you a gut instinct answer. Maybe it's the answer they think you want to hear; maybe it's the answer they think is most popular; maybe it's the answer that will make them look tough; or maybe they just don't want to look stupid in front of a grown-up with a clip board. Hence the 1% of "youth" who courageously admit they don't know, don't care, or otherwise have better things to do.
Don't get me wrong; the same problems outlined above are common among adult respondents as well. Which only proves my point: if we expect (and we do!) that adults, when confronted with questions on issues they know little about, will make stuff up (literally) or just give you the answer they think you expect or makes them look good, then how likely is it that 12 to 17 -year olds will exhibit more of that behavior? When you deal with "youth" who are just that young, you should be overly careful not to oversimplify and overstate your results. The study asks teenagers a bunch of complicated questions about scary things (like terrorists and wars) and then mocks the youth when they don't give a rational, correct answer.
I am not saying that the "youth" is stupid. I am saying that basic brain development studies (the kind adults would be expected to know if they theorize about youth) tell you that the frontal cortex – the one responsible for calculating risks and self-control – is not fully developed until the age of 25, which is why "youth" do not get to drive (and if they do, they pay higher insurance rates, or their parents do), don't vote, and some are not even left home alone yet. Knee-jerk solutions like torture to get to the truth would sound like a good idea EVEN IF the "youth" knew about the Geneva Convention or read studies showing torture doesn't work; they are just young and restless. The irony is that a large portion of adult respondents (presumably with a fully functional frontal cortex) do the same thing, while having very high expectations of today's "youth."
In my opinion, what this study does is entrapment – loosely defined, aside from methodological malpraxis. The condescending tone and painting a whole generation with a broad acusatory brush just adds insult to injury.