The continuum of Analytic Journalism
by
JTJ
on Sun 15 May 2005 12:17 PM MDT
The past eight days have presented
Americans with two extremes of Analytic Journalism, the bad and the good.
The bad is Newsweek’s’ cover
story that hit the stands on Monday, May 9, 
2005, boldly headlined “2005 American’s Best
High Schools: Ranking the Top 100.”
Inside, we are told that “Public schools are ranked
according to a ratio devised by Jay Mathews: the number of Advanced Placement
or International Baccalaureate tests taken by all students at a school in 2004
divided by the number of graduating seniors.” The table accompanying
the story also includes a second variable: percent of the student body
“eligible for free and reduced lunches, an indicator of socioeconomic status….”
But the backstory is even
stranger. Remember, Newsweek is making
a semi-big deal out of the fact that Washington Post reporter Jay Mathews
“devised” the ratio. Yet Mathews himself
wrote something somewhat different on December 27, 2004 in a piece headlined “Are
Bonus Grade Points for Hard Courses Unfair?”
Here is Mathews quoting from his own story [emphasis
ours], which quotes a large-sample study on AP/IB testing conducted by Saul
Geiser and Veronica Santelices of the University of California at Berkeley.
“Here is the news story I wrote about it last week.
For those who don't have time to read the whole thing, this quote from the
report sums it up well:
"’The
main finding . . . is that, controlling for other academic and socioeconomic
factors, the number of AP and honors courses taken in high school bears
little or no relationship to students' later performance in college. The
study is based on a sample of 81,445 freshmen entering the University of
California (UC) [including eight campuses] between 1998 and 2001. While student
performance on AP examinations is strongly related to college performance, many
students who take AP courses do not complete the associated AP exams, and
merely taking AP or other honors-level courses in high school is not a valid
indicator of the likelihood that students will perform well in college.’”
So then tell us again why
the percent of students who merely take the exams is indicative of the “best”
high schools? Mathews is an experienced
reporter who has covered education for a number of years. We wonder just how much direct involvement
he had with the editors over at Newsweek.
We appreciate, however, the
editors at least showing their secondary data, if not the raw numbers. (The online version ranks more than 1,000
U.S. high schools.) Where we disagree
with Newsweek and Mathews is the use of any single variable to describe
something as complex as measuring the quality of education. Here’s why:
·
This index only
measures test-taking; it says nothing about performance, either on the
exams or once the students get to university.
In fact, any graduating senior willing to pay $82 per AP test
would be counted in the Mathews index.
Ergo, should a school district want to rise in the ranks, it would
allocate the funds to pay for 12 percent of its graduating seniors to take the
AP exams and, presto, it becomes the No. 1 high school in the nation. [The No. 1 school reported 10.755 percent of
its graduating seniors took the exam(s).]
Unlikely? IAJ fellow and former high
school teacher Pat Mattimore reports: “A chairman of one of the ‘ranked
public high schools’ back East e-mailed me that at her school classes are
bribed with things like harbor boat cruises to get 100% participation on the
exams as a result of Mathew’s ranking system.”
[Click here
for Mattimore’s essay on the topic.]
·
This kind of
"journalism" is only a step or two removed from the "Best Of
…" lists so beloved by ad sales folks.
Anybody can vote and vote often for their favorite pizza restaurant. For Newsweek to pull some cheap shot like
this -- aided and abetted by an experienced reporter from the WP -- only
compounds the Best Of… promotional sins and adds to the perceived shabbiness of
journalism. Newsweek even misses opportunity with this approach: its
website, that has the school rankings, has no search engine so readers can find
their schools of interest (or NOT find their schools). This is just the kind of stats that cries
out for a GIS server that would draw appropriate maps and attach “drill-down”
data, the kind of thing USA Today and other newspapers have used to present voting results.
·
More serious is that
this single-index approach also feeds into the public's instinctive longing for
a magic-bullet method of analysis and decision-making. There are few, if any, social phenomena that
can be adequately described by one or two indices. This sort of Newsweek thinking supports simplistic solutions,
e.g. the way to stop excessive drinking or abortions simply is to pass a law
against them. The way to make people do
the right thing is to post The Ten Commandments in prominent public
places.
Responsible journalism in the Digital Age (aka, the Age of Data Access) should
be trying to not over simplify but explain/illustrate complexity and the work
required to understand highly complex issues like "educational
quality."
Good work at The Times
Fortunately a week later, The New York Times opened what may prove to be a fascinating series, “Class
in America: Shadowy Lines That Still Divide”
“This series does not purport to be all-inclusive or the
last word on class. It offers no nifty formulas for pigeonholing people.
Instead, it represents an inquiry into class as Americans encounter it.”
Reporters Janny Scott and David Leonhardt do a
fine job in the opening article addressing – and illustrating – the complexity
of understanding the role of class in the United States’, or any, society. They write:
“The series does not purport
to be all-inclusive or the last word on class. It offers no nifty formulas for
pigeonholing people or decoding folkways and manners. Instead, it represents an
inquiry into class as Americans encounter it: indistinct, ambiguous, the
half-seen hand that upon closer examination holds some Americans down while
giving others a boost.
“The trends
are broad and seemingly contradictory: the blurring of the landscape of class
and the simultaneous hardening of certain class lines; the rise in standards of
living while most people remain moored in their relative places.”
No “nifty formulas”?
Perhaps not yet in the story, per se, but the first-rate infographics, innovative
ways of presenting the data
and analysis.
supporting the story sure draw on a lot of data, data analysis and provide some
The bottom line: Top-notch analytic journalism from
The Times that informs readers while illustrating the complexity of the
topic. Oh, but had Newsweek been
able to invest in the same effort, what a good week it might have been.■
