Big Data and Culturomics
Friday, August 09, 2013
Big Data — and how we use it — is changing the way we understand our culture and history. Research scientists Erez Lieberman Aiden and Jean Baptiste Michel (Uncharted: Big Data as Lens on Human Culture) teamed up with Google to create the (highly addictive) Ngram Viewer: it sifts through millions of digitized books and charts the frequency with which words have been used. "Geek" and "nerd" started proliferating around 1980. “Women” began to spike in the 60s, surpassed “men” in the 80s, then fell behind again in 2005.
The Ngram Viewer also exposes notable gaps. “If you take Marc Chagall or Pablo Picasso, Kandinsky or any virtually any intellectual, especially Jewish, especially artists,” Michel explains, “and you look for the trajectories of their names in books published in German, you will see huge dips, holes really, literally, at about the time of the Nazi regime. So you can visually see the thumb of oppression pressing down the trajectories of these people.”
Your Ngram Viewer Discoveries
Use the Ngram Viewer to search a word, phrase, name, over any time period you want. Post your best discovery as a Comment below; we may call you to be on the air.
The deadline to be considered is August 15 at 11:59pm ET.
Aiden and Michel call their method of combing through text to map cultural trends “culturomics.” “It’s like genomics but with culture,” Aiden tells Kurt Andersen.
Stanford French professor Daniel Edelstein calls his data-driven project “Mapping the Republic of Letters” a kind of Facebook for the Enlightenment. He and a team took tens of thousands of letters written by intellectuals like Voltaire, Jean-Jacques Rousseau, and Jonathan Swift during the 1700s. Using maps, their system shows how new ideas spread through Europe and the world. And in some cases, the data doesn’t follow accepted history. Voltaire is thought to have been influenced by his English contemporaries. But when Edelstein mapped his correspondence, “Where are the letters to England? There’s such a myth around Voltaire as having transmitted English thoughts about politics about philosophy about science into France and there was just this gaping hole in the map of his correspondence.” Edelstein says this discovery has lead to a new research project to explain the hole.
And then there’s the question of proper spelling: you say doughnut, I say donut. Teddy Roosevelt favored the latter method, even trying to standardize that spelling in 1906, but it wouldn’t stick. What did? According to the Ngram Viewer, “it seems like [“donut”] is relatively contemporaneous with the creation of Dunkin Donuts.” We can’t say we’re surprised.
Special thanks to Maneesh Agrawala, our scientific advisor on this story.