Institute for Analytic Journalism

Web scraping with Excel [Saturday highlights from the Global Investigating Journalism conference]

JTJ — 2005-10-01T04:22:06-06:00

Tommy Kaas, of the Danish International Center for Analytical Reporting, just presented a fascinating session on how to use Excel tools to "scrape"data off the web an import it into Excel, at least Excel XP. This is typically helpful where one needs to extract data from standardized tables on dynamic web sites, for example those with demographic, economic or crime data.

He has posted some handouts at dicar.org/global2005 or http://www.dicar.org/global2005/exercise_macroscraper2.htm
. It's not yet clear to us if this is more efficient than writing PERL or PHP scripts, but it's still an elegant hack.

AJ Tool-of-the-Week: Furl - Online bookmarking tool

JTJ — 2005-05-25T17:10:28-06:00

We've been using a variety of web-based bookmarking tools for the past four or five years, tools like the now-departed Blink and Backflip. They were all OK (so long as they remained financially viable), but never quite seemed to meet all our needs. Recently, though, we learned about Furl (www.furl.net) and we like what we see. Furl is in beta, so we don't know what the ultimate price will be, but journalists will like the ease with which we can pull URLs off a web page, markup those savings with keywords, copy-and-paste webpage annotations and then save the citation in a folder of your making. Oh yeah, you can also save and e-mail the link(s) to anyone. In fact, we like Furl so much, we will be demo-ing it next week at the IRE conference in Denver.

As the Furl gang says:
"Furl will archive any page, allowing you to recall, share, and discover useful information on the Web. Browse your personal archive of Web pages, and subscribe to other archives via RSS."

Check it out.

Doing well by doing good

JTJ — 2005-05-19T15:50:02-06:00

Here at the IAJ we believe one of the reasons people come to newspapers or broadcast stations is to get the data which, upon analysis, they can turn into information that helps them make decisions. Ergo, the more meaningful data a journalistic institution can provide, the greater value that institution has for a community.

A good example arrived today thanks to Tara Calishain, creator of ResearchBuzz. She writes:
** Getcher Cheap Gas Prices on Google Maps
<http://www.researchbuzz.org/getcher_cheap_gas_prices_on_google_maps.shtml>

"Remember when I was saying that I would love a Gasbuddy / Google Maps mashups that showed cheap gas prices along a trip route? Turns out somebody has already done it -- well, sorta. You can specify a state, city (only selected cities are available) and whether you're looking for regular or diesel fuel. Check it out at http://www.ahding.com/cheapgas/ "

The data driving the map is ginned up by GasBuddy.com It's not clear how or why GasBuddy gets its data, but it offers some story potential for journalists and data for news researchers. It has an interesting link to dynamic graphs of gas prices over time.

Surely the promotion department of some news organization could grab onto this tool, tweak it a bit, promote the hell out of it, and drive some traffic to and build loyalty for the organization's web site.

That's the obvious angle, but what if some enterprising journo started to ask some questions of the data underlying the map? What's the range in gas prices in our town/state? (In Albuquerque today, the range was from $2.04 to $2.28.) Are there any demographic or traffic flow match-ups to that price range? How 'bout the variance by brand?

Would readers appreciate this sort of data? We think so, especially if there was an online sign-up and the news provider would deliver the changing price info via e-mail or IM much like Travelocity tells us when airline ticket prices change by TK dollars.

AnyLogic: Tool-of-the-Week

JTJ — 2005-05-11T22:33:30-06:00

A talented band of coders in St. Petersburg, Russia has put together a nifty simulation modeling application written in Java. Anylogic™ "supports virtually all existing approaches to discrete event and continuous modeling, such as process flow diagrams, system dynamics, agent-based modeling, state charts, equation systems, etc. With this incredibly rich toolset you are not limited with the technology anymore – analyze the problem, identify the best approach, and find the solution!"

The package is relatively affordable, especially if one can qualify for the educational discount. It could make for a handy tool to model and/or illustrate a variety of dynamic aspects in an urban setting -- ambulance response time, crowd movement during an anti-war demonstration.

Today, too, the roll-out of the LA Times re-designed web site includes an intuitive interactive map of freeway traffic flow -- real time -- of the greater LA basin. Perhaps some enterprising news organization will figure out a way to tie these maps from SigAlert into the dashboard-mounted GPS navigation devices. Or will SigAlert itself deliver those goods?

Tool of the Week: The geography of libraries

JTJ — 2005-04-21T18:06:16-06:00

Gary Price's Resource Shelf indirectly pointed us to a new resource that can provide journalists with a quick and not-so-dirty GIS tool for their community or any other in the U.S. GeoLib is the Public Library Geographic Database (PLGDB) Mapping site. But its mapping engine delivers far more than just push-pining public libraries. The GIS server can display a variety of physical, demographic and political boundaries and data that can be turned on and off in traditional GIS layers. The resulting maps could be easily copied with a tool like Snag-It and dropped into a document or web page.

"Florida State University's GeoLib Program ( www.geolib.org ) is developing the National Public Library Geographic Database. The database includes the locations of 16,000 public libraries, data sets from the US Census, and library use statistics from the National Center for Educational Statistics. The goal of this database is to provide consolidated information on public libraries nationwide, easily accessible over the Internet. The project is sponsored by the Institute of Museum and Library Services ( www.imls.gov )

A key ingredient to the long term success of the PLGDB is the ongoing incorporation of other relevant data. We need your feedback as to what data you would like to have in the future. We hope the database will offer public librarians access to data that can shape local, regional and national library policies. For more information about GeoLib and its programs and to find out about the latest status and data releases of the National Public Library Geographic Database check back on this website regularly."