Gifted Issues Discussion homepage Forums GT Research The Elite Illusion (public exam schools)

http://papers.nber.org/papers/W17264
The Elite Illusion: Achievement Effects at Boston and New York
Exam Schools
by Atila Abdulkadiroglu, Joshua D. Angrist, Parag A. Pathak - #17264 (ED LS)

Abstract:
Talented students compete fiercely for seats at Boston and New York
exam schools. These schools are characterized by high levels of peer
achievement and a demanding curriculum tailored to each district's
highest achievers. While exam school students clearly do very well
in school, the question of whether an exam school education adds
value relative to a regular public education remains open. We
estimate the causal effect of exam school attendance using a
regression-discontinuity design, reporting both parametric and
non-parametric estimates. We also develop a procedure that addresses
the potential for confounding in regression-discontinuity designs
with multiple, closely-spaced admissions cutoffs. The outcomes
studied here include scores on state standardized achievement tests,
PSAT and SAT participation and scores, and AP scores. Our estimates
show little effect of exam school offers on most students'
achievement in most grades. We use two-stage least squares to
convert reduced form estimates of the effects of exam school offers
into estimates of peer and tracking effects, arguing that these
appear to be unimportant in this context. On the other hand, a
Boston exam school education seems to have a modest effect on high
school English scores for minority applicants. A small group of 9th
grade applicants also appears to do better on SAT Reasoning. These
localized gains notwithstanding, the intense competition for exam
school seats does not appear to be justified by improved learning for
a broad set of students.

******************************************************

The paper is at http://econ-www.mit.edu/files/6856 . This will be a paper that advocates of gifted education may not welcome but will need to respond to, as was the paper discussed in the thread "Are gifted education programs a waste of money?" http://giftedissues.davidsongifted....2430/Are_gifted_education_programs_.html

Economists are so cute, with the assumptions they make, and the way they try to apply their fanciest math to the behavior of adolescents! Yes, that's what we need to analyze how well a school is doing--more sophisticated math!

[quote]
Our strategy in a nutshell is to compare the scores of exam school applicants
who barely clear the admissions cuto to the scores of those who fall just below.
[quote]
It may well be that being a big fish in a less competitive pond is actually better for outcomes like SAT score then being the littlest fish in the most competitive pond. I think it depends on the student. It may well be that the measurable effects would be on the students who score the highest going into a selective school - because they have been sandbagged the most. Of course finding good outcome measurement for that group is tricky - probably not enough headroom in the SATs to demonstrate some very real differences.

((shrugs))
Grinity

As someone who attended a fiercely competitive exam school, and is also 2E, I would really hate to have my school experience reduced to my ability to perform on the SAT! My SATs were good, and good enough to get me where I wanted to go to college, but it had nothing to do with my school. My SAT scores went up due to Princeton Review - my achievement tests scores are entirely related to actually attending my school and the fabulous educational opportunities finally experienced in high school.

I understand the rationale of comparing those who got in with those who did not in terms of data that is readily available to the researchers. But it totally misses the point of these schools - what these schools provide is the opportunity for breadth and depth of subject in an environment where it is acceptable to be smart and to enjoying learning. the SAT tests none of that.

Letting go of enormous pet peeve now and stepping down of the soapbox . ..

DeHe

The paper is discussed at the Boston Globe:

http://boston.com/community/blogs/rock_the_schoolhouse/2011/08/historically_many_of_massachus.html
Do exam schools add value?
by Jim Stergios
August 7, 2011 10:31 AM

Obviously SAT scores were a bad place to look for differences in education between schools.

What's interesting is that analysis failed to reveal a difference in the number of AP tests taken, and passed, despite the fact that the exam schools offer so many more AP courses. It seems that instead of taking more tests, exam school students are simply selecting a similar number from a larger domain. I don't see that the study attempted to make any distinction between taking and passing a Calc B/C test rather than a Calc A/B test though, which seems like a flaw to me.

On the other hand, is it reasonable to expect that it's no better to be in a class where the pace is a little too fast than a little too slow?

If we want to know the value of exam schools, shouldn't we focus on the students who are more typical of exam schools? The students who got in (not borderline) but didn't go because of geography, etc?

I whole-heartedly agree. They also provide the experience of going to school with a group of peers of similar ability. It probably gives many of them the experience of being "average" for the first time in their lives -- something I didn't have until I went to college. I think that the social and emotional benefits of such an environment should not be discounted.

Another recent paper finding similar results:

http://www.people.fas.harvard.edu/~dobbie/research/ExamSchools_July2011.pdf

Exam High Schools and Academic Achievement: Evidence from New York City
Will Dobbie, Roland G. Fryer, Jr.
NBER Working Paper No. 17286
Issued in August 2011
Abstract: Publicly funded exam schools educate many of the world's most talented students. These schools typically contain higher achieving peers, more rigorous instruction, and additional resources compared to regular public schools. This paper uses a sharp discontinuity in the admissions process at three prominent exam schools in New York City to provide the first causal estimate of the impact of attending an exam school in the United States on longer term academic outcomes. Attending an exam school increases the rigor of high school courses taken and the probability that a student graduates with an advanced high school degree. Surprisingly, however, attending an exam school has little impact on Scholastic Aptitude Test scores, college enrollment, or college graduation -- casting doubt on their ultimate long term impact.

Except.

They used Bronx HS of Science v Stuyvesant as the means to check for a discontinuity.

The programs at Bronx are every bit as good as Stuyvesant and all three schools used are exam schools - so I do not see this as a good test.

If you wanted to test, you would have to send the kids who did not get into the exam school to a "normal" school.

I would think one way to do this would be to track kids who moved out of NYC to schools that produced just a handful of high SAT scoring students each year.

An interesting history of New York's exam schools is

http://www.city-journal.org/html/9_2_how_gothams_elite.html
How Gotham�s Elite High Schools Escaped the Leveller�s Ax
Heather MacDonald
City Journal
Spring 1999

Detractors may argue all they want, but if nothing else, graduates of theses elite school are disproportionately represented at the Ivies, at least when I was in college in the eighties.

I would suggest that their representation in the Ivies may demonstrate the social benefits of these schools rather than any academic benefit. In other words, the Ivies actively recruit from these schools. Therefore, there is a social capital advantage to attending them, rather than a purely human capital advantage to doing the coursework there.

So do they make a difference? Yes, but not simply because of the academic program, but because attending one gets you into a certain social world that includes the Ivies, which someone may not otherwise have access to.

Unless things have changed from the mid-1990's, it's never been that hard to get into an Ivy if you are in the 140ish+ I.Q. range (and used this ability in high school).

Affording an Ivy? That's a different story.

That may be true, but the school setting and that social cultural knowledge ABOUT the Ivies plays a role. I grew up in rural South Texas in a town of about 6000 people. No one I knew went to an Ivy and they seemed VERY far away. Even with the academic ability to get in and succeed there, you need a certain amount of social and cultural capital to know how to go about it and even see it as a possibility. Now, most of the kids from these schools probably already have this from their families, but attending an exam school just adds to that social capital because the Ivies are already actively looking for students from these schools.

If you lived in an urban area on the East Coast in a middle to upper middle class family, a 140+ IQ may be all it takes and an exam school can only enhance those opportunities. But without those social benefits, one's intellectual ability and academic success can easily go unnoticed and unsupported. To me, the representation within the Ivies and the findings of the study that these students do not perform significantly better (methodological questions notwithstanding) suggest that any "value added" by these schools is social/cultural not academic.

And then of course, there's the money...

Isn't there's some sort of quota system within Ivy admissions, so they are allowed to take X number (generally) from these schools, whereas other regions have a lower quota?

I think there is often a benefit in the application process for being "geographically diverse". Is this what you mean? So if I had applied from my rural Texas school, I would have gotten a boost for being diverse on that count. But that is because it is so much less common for such students to apply. But I would have lost out in any legacy points or letters of recommendation from alum or esteemed educators. Plus my school did offer less in terms of extracurriculars and more advanced curriculums (no AP classes offered at all). So I think, despite my IQ, my application would not have looked as good as someone with a lower IQ and more access to these social and cultural extras.

For the most part though, I would guess that private prep schools and magnet schools across the US are the usual pipeline to elite universities. I went to the University of Chicago for grad school and I know they recruited alot of their undergrads from particular private schools across the country.

That was part of what I meant. I also expect that the pipelines are "known quantities" so I would guess that there is some sort of soft quota where you don't have to do as well as your peers as you would at a standard issue school.

I think that things have changed a lot.

Last year ~35,000 students applied to Harvard and only ~2,100 were admitted. The Ivies reject straight-A students and students with perfect SAT scores. This page has admissions statistics for all the Ivies.

If you scroll down the page at the second link, you'll see how admissions statistics have changed in the last few years alone. For example, Harvard used to admit around 10% of its applicants, and now takes only around 6%. Cornell is down from 30% to 18%. Etc.

College admissions have become an arms race, with increasing students enrolling in SAT prep courses, seeking summer experiences for use as application essay fodder (as opposed to personal development or learning), and so on. I have actually seen advertisements for college essay writers (and PhD thesis researchers and writers) on craigslist. I flagged one of the ads, but craigslist didn't take it down. It's reasonably common.

Anecdote: someone in my family is very bright, had amazing scholarship offers for two good private colleges, graduated at the top of her class, had lots of extracurriculars and awards, and was admitted to exactly ONE Ivy out of five applications. And that was several years ago.

That's the kind of person I was talking about, meaning what would be expected of a 140+ish person.

And she got into an Ivy (I didn't say "get accepted to all the Ivies").

So, while things are harder, you should still be able to get into an Ivy.

The story was more complicated than that. Sending a PM to avoid giving out too much info publicly.

At any rate, the statistics I provided tell much more than a single anecdote. As the news story said, "Valedictorians are a dime a dozen" in the applicant pool.

Add the quotas (they want the same proportion of racial groups each year, they reserve places for alumni children, etc. etc.), plus a higher bar to acceptance for financial aid applicants, and it's easy to see that a high IQ, good grades, and high SAT scores are no guarantee of admission.

I know a story similar to Val's - valedictorian, rural high school in the SE, no AP classes available, near perfect SATs, plenty of extracurriculars, and 0 for 2 on Ivy admissions back in the mid-80's.

I agree that IQ, grades and SAT scores aren't necessarily enough. And it's not recent.

As someone admitted to an Ivy, and who went through the process, here is my take.

Most kids who have the skills and life story to get in do not apply. That is changing and this is part of the reason why applications have risen at top universities.

Another reason applications have risen is that the influx of Asians and their desire for social acceptance is also fueling the number of applicants.

Another reason is that the new SAT is much easier than the old one.

And a lot of kids are using the SAT prep programs and also attending "chinese school." Most of the top kids at the top public schools in DFW do this. This fuels the high SAT scores and high GPAs.

When I got accepted, I was the first in five years from my HS. The class before me had a number of exceptional grads, but only one applied to an out of region school and she went to Stanford. In my class, my classmates applied to public universities and later did their grad schooling locally as well. Looking at the grad list today from the same school, 1-4 a year go to ivies, MIT, or Caltech. A change.

The exceptional kid 25 years ago had AP classes, SAT > 1500, and some outside activities. Today, this is the norm. Kids are so much smarter and parents so much more prepared that the normal AP stuff is not a separator at all.

As far as life stories go, most kids will have the same one. The exceptional applicant today will have a history of independent pursuits or significant real world achievements. Something that the committee does not see on most apps.

For instance, six years in the US Army Rangers with three combat tours. Or an established byline in the local newspaper. Or a dozen published papers in a peer reviewed journal. Or 30 college credits taking senior level courses at the local university. Or manager in some business.

Whereas that was me in the early 90's, and I was 1 for 1 on Ivy admissions.

I think part of it is that there's not enough headroom in those sorts of descriptions to distinguish the 99.999th percentile valedictorian-with-high-SATs from the 99th percentile one. And part of it is that even with 17% yields, test scores and grades have never been sufficient to demonstrate that you can successfully transition from big-fish-in-small-pond to small-fish-in-big-pond.

The Ivies are generally considered to be much harder to get in than to stay in, and they don't want to admit kids who will struggle unsuccessfully. IMHO you need some way of demonstrating that you have the ability to struggle successfully, not just that you're smart.

Demographics might be coming into play here.

Aren't the Millenials the baby boomers' kids who are currently in college?

They aren't making any more Ivies.

If that's one of the issues, then Ivy percentage acceptances should peak, then decline.

The internet-driven common application form is another reason. It's a lot easier to complete one application for six colleges than six completely different applications for six colleges that also have to be done by hand or typewrter, except for the essays.


I don't agree that kids are smarter today. Rather, I think they're better prepped for an easier SAT. Plus, AP courses have become less demanding in order to meet requirements that "everyone be able to have a chance to try them." When I was in school, most kids weren't capable of taking AP courses and most didn't (and that was okay from what I could tell). Today, everyone deserves a chance to "try" and the system has ended up making them easier.

None of this makes today's teenagers smarter than their parents.

There was a program on NPR, new york station about the exam high schools and other good high schools. A principal at one of the other good high schools weighed in. He remarked that the other good high schools didn't have the AP classes offered at a Styvescant. Styvescant is 31 in the nation and second in National Merit scholars. If top students want to get in there, they are going to have a great peer group. That is hard to beat, when you are looking for great peer groups. And Harvard loves (which means other good colleges love) Styvescant and the quality of candidates.

Recently a 15 year old left Styvescant and went to Harvard. Do you want your kid with those kinds of peers?

Now, just like Juilliard, which became known as Little Asia, Styvescant is becoming an Asian majority. The Asian kids are getting the scores to get in. Our friend, who had a child in Sty, up until last year, said when he went to the AP math parent's meeting, he and his wife were the only non Asian parents there. That may be reflect in the Language portion of the SAT.

Ren

The exceptional kid 25 years ago had AP classes, SAT > 1500, and some outside activities. Today, this is the norm. Kids are so much smarter and parents so much more prepared that the normal AP stuff is not a separator at all.




Sigh. As someone who researches educational inequality for a living, this thread is now depressing me. This is only the norm for SOME schools and the degree of difference between those schools and the rest is increasing, making it much less likely that a talented student who doesn't go to those schools or have a family that can afford the extra tutoring and test prep, etc. will have a shot at attending a top university.

http://en.wikipedia.org/wiki/Flynn_effect says today's kids have 10 IQ points over 30 years ago.

I know Mr W is ahead of my milestones by 9 months or so. And he has loads more opportunities to learn than I did - and far more knowledgeable and sympathetic parents.

Multiply this across the whole population, and it will have an impact.

I think many of the so called elite schools are echo chambers that are overpriced and which saddle their students with too much debt. A different topic, for sure.

The school's name is spelled "Stuyvesant".

Uh, that's the point of the elite schools. They're top quality credentials! And you learn what echos you need to echo in order to fit in.

Look at law schools, for example. You've got your T3, T6, T14, and, generic law schools. The better the school, the less you actually have to work because the less your grades will matter. It's nearly impossible to fail out, even if you kind of actively try. If you go to a generic law school, you will have to actually attend class and work. They won't just hand you a diploma. Plus, you have to compete to win so that you become employable. All of them cost $150,000.

You're not paying for an education. You're paying for a global brand. Actually learning things is a side effect and purely incidental to meeting relevant people and getting a diploma with the right name on it.

I think you are correct, but there is a population of educated and affluent parents that is more "hands-on" and academically focused in their childrearing than parents were 30 years ago, and they use the Internet (for example this forum) to find opportunities for their kids (such as Johns Hopkins summer programs) and their high incomes to afford them. Their children may thus be able to accomplish more (or at least assemble more impressive resumes) before entering college. My parents did not try to advance me in math before a summer program prior to 7th grade, and they knew about it only because I had a math prodigy friend. Now anyone interested can learn about EPGY, AOPS, Singapore math, etc.

An NYT article states that British college admissions are more academically focused, which I agree with:

http://thechoice.blogs.nytimes.com/2011/09/22/applying-to-college-in-the-united-kingdom/
September 22, 2011, 10:05 PM
Applying to College in the United Kingdom
By REBECCA R. RUIZ

...

Ms. Burn of Oxford stressed that universities in Britain are more interested in academically directed personal statements than they are in reflections on personal memories or demonstrations of character.

Of the essay, she said: �It�s not the place to talk about the time you ate a pot brownie. It�s about what you want to study and why.�

Ms. Burn emphasized that recommendations should be similarly to the point. �This isn�t a time to talk about how charming students are, or how they�re always on time,� she said. �Talk about their academic performance and be specific.�

She also advised that people providing recommendations be wholly truthful and not write anything they would not want a student discovering: by law in Britain, applicants are allowed to read submitted recommendations of them.

Applicant interviews are uncommon in Britain, excluding Oxford and Cambridge, both of which require them, Ms. Burn said. �There are just too many applicants to interview at most schools,� she said. �And the concept of the alumnae interview or student interview doesn�t exist for us.�

Most prospective college students are unable to answer questions about "what they want to study" and "why".

Unless the "why" is money, and the what is "whatever major pays the most."