Google’s founding philosophy is that we don’t know why this page is better than that one: If the statistics of incoming links say it is, that’s good enough. No semantic or causal analysis is required. That’s why Google can translate languages without actually “knowing” them (given equal corpus data, Google can translate Klingon into Farsi as easily as it can translate French into German). And why it can match ads to content without any knowledge or assumptions about the ads or the content.

Speaking at the O’Reilly Emerging Technology Conference this past March, Peter Norvig, Google’s research director, offered an update to George Box’s maxim: “All models are wrong, and increasingly you can succeed without them.”

…faced with massive data, this approach to science — hypothesize, model, test — is becoming obsolete.

There is now a better way. Petabytes allow us to say: “Correlation is enough.” We can stop looking for models. We can analyze the data without hypotheses about what it might show. We can throw the numbers into the biggest computing clusters the world has ever seen and let statistical algorithms find patterns where science cannot.

There is certainly value in sophisticated data mining and an inductive approach to research, but to dismiss the deductive approach (construct theory>deduce testable hypotheses>empirically verify or falsify hypotheses) would be shortsighted.  Modern data mining may be enough to authoritatively establish a non-random relationship, and in some cases (translations and advertising) more than suffices for useful application.  However, even the largest data sets still represent only a sample–and, therefore, an approximation–of reality.  Moreover, establishing correlation still doesn’t get you to the underlying causal mechanisms that drive causation.  Even if Google, with enough data and advanced statistical techniques, can claim that a causal relationship exists it can’t tell you why it exists.

For some subjects, “why” may not matter–do we care why Google’s program is able to accurately translate between languages, or is the practical effect enough for us?  But for others it is crucial when thinking about how to construct an intervention to alter some state of being (e.g. a medical condition, poverty, civil war, etc).   Understanding causal mechanisms can also help us think through the consequences of an intervention–what are some potential side effects?  Are there other, seemingly unrelated, areas that might be affected by the intervention in a negative way?  When we are dealing with more interconnected, complex systems (like human physiology or society) it behooves us to go beyond relationships and understand what levers are being pulled.

Update: My friend Drew pointed me to his reaction to Anderson’s piece when it originally came out–worth a read.

Now and in the future, we will have unprecedented access to voluminous amounts of data.  The analysis of this data and the conclusions drawn from it will have a major impact on public policy, business, and personal decisions.  The net effect of this could go either way–it can usher in a period of unprecedented efficiency, novelty, and positive decision making or it can precipitate deleterious actions.  Data does not speak for itself.  How we analyze and interpret that data matters a great deal, which puts a premium on statistical literacy for everyone–not just PhDs and policy wonks.

Thompson notes a number of statistical fallacies that many, including members of the media, fall prey to.  Using a single event to prove or disprove a general property is one spectacular one that we see all the time, particularly with large, macro-level events.  Regardless of what side of the climate change debate you are on a single snow storm or record-breaking heat wave does not rise to the level of hypothesis-nullifying or -verifying evidence.

There are oodles of other examples of how our inability to grasp statistics–and the mother of it all, probability–makes us believe stupid things.  Gamblers think their number is more likely to come up this time because it didn’t come up last time.  Political polls are touted by the media even when their samples are laughably skewed.

Take correlation and causation.  The cartoon below nicely illustrates the common fallacy that the correlation of two events is enough to prove that one causes the other:

Correlation and Causation

Bottom line: the importance of statistical literacy will only increase. Statistics will come to permeate our lives, more so than ever before. And if we want to truly take advantage of this we had better learn to speak the language.

Combine sophisticated data analytics with the fact that people, it turns out, are actually rather predictable in their routines and you end up with a myriad of possibilities for business and public policy.

Via Alex Lundry

[Note: I will be traveling over the next few days, so posting will be light-to-nonexistent]

[...] IBM is unveiling a new internal analytics product that the company is touting as the “largest private cloud computing environment for business analytics in the world,” which launches internally with more than a petabyte of information. Along with this internal product, IBM will launch a companion product for clients to build upon this cloud-based architecture, called IBM Smart Analytics Cloud.

The internal product, dubbed Blue Insight, will provide 200,000 employees in IBM’s sales and development department with the ability to extract data and information to make decisions and gain further insight at the point of sale. Blue Insight will gather information from nearly 100 different information warehouses and data stores, providing analytics on more than a petabyte (1,000 terabytes or 1,000,000 gigabytes) of data. For example, sales execs may use customizable queries of real time data to understand revenue opportunities and how many sales in their region are closing to help improve prediction. Or a manufacturing process engineer can evaluate real-time data on the plant floor to identify trends and data to improve yield and reduce shipment delivery times.

IBM Smart Analytics Cloud offering for clients will similarly deliver powerful business intelligence via the scalable, private cloud. The product lets the client import data and than transform this information into insights to develop strategies and decisions. The service will offer the ability to create reports, analysis, dashboards, and scorecards to monitor business performance and measure results.

The key to good analytics is not just the accumulation of relevant data, but also the ability to manipulate and visualize the data in meaningful ways.  The ability to create custom dashboards, etc, is an important part of making analytics a useful and profitable tool.  IBM has certainly dedicated itself to expanding its service and software offerings relative to its traditional hardware business, and analytics has emerged as a huge part of that strategy.  In doing so, it should be well positioned to profit from the coming analytically-driven world.

Authorship over time

So what are the implications for universal authorship?  SEED mentions one and I have another that comes to mind.

The first implication is in the article: as more and more people become creators of content it will become increasingly difficult for organizations of all kinds to control their messaging and their brand.  Provide a bad customer experience and it will be on facebook, immediately broadcast to hundreds if not thousands of people.  Discriminate against someone because of their race or sexual orientation and they will tweet about it, likely prompting dozens of re-tweets which allows the issue to reach exponentially more people, who then might blog about it, and so on, and so on.  Organizations are already finding it hard to control their brand, imagine when the number of authors increases 10 fold (which, according to SEED, will now happen yearly).  Theoretically, firms and technologies that can monitor content related to an organization will be big winners.  Additionally, organizations will need to invest more heavily in their own networks and crowds to help combat negative content (whether true or false).

Another implication that immediately came to my mind is the increased difficulty in separating signals from noise.   As the cost of entry into the market for content heads towards zero and the tools of creativity are fully democratized, there will be an even greater explosion in content from which individuals and organizations will have to separate relevant, accurate pieces of information.  There is already a flood of information to wade through and it is becoming more difficult to do so.  Exponentially increase the amount of content and the variety of sources and you’ve taken the problem to an even greater level.  All things being equal, more content and a more fractured supply source will only increase the amount of noise and make identifying the signals (the accurate pieces of information) more difficult.  What we will need, and what will become valuable, are services that don’t simply aggregate content, but also determine their level of accuracy and credibility.

Nate Silver

I am thinking here of services that mimic the approach of Nate Silver at FiveThirtyEight.  The world didn’t lack political polls, but it did lack a methodology for cutting through the noise created by dozens of polls, many providing contradictory predictions.  Silver came up with a way to not just aggregate polls, but to increase the ratio of signal to noise, allowing for a more accurate portrayal of public opinion and ultimately a prediction of Presidential elections.  Silver made polls better by developing what I would call a sophisticated aggregation methodology (note: I will be writing more on this soon).  The key will be to develop a scalable, replicable approach that can be applied to a variety of domains.

Soon enough, we may all be creatives.  And that means there will be a heck of a lot more chaff to wade through.

This underscores the comparative advantage of individuals with skills amenable to collecting, coding, manipulating, and visualizing Big Data in the current labor market.  Additionally, as not all data will be easily collected and coded via computer programs, services that can efficiently harness the crowd in support of Big Data will also be critically important.

A visualization of thousands of Wikipedia edits that were made by a single software bot. Each color corresponds to a different page.

Photo via Wired

As I was watching the video I thought about the potential to leverage such a platform for large-scale coding of qualitative data. Coming from the social sciences, often we find the need in large scale research for the massive coding of data, whether it is language from a speech, the tenor or sentiment of quotations (or newspaper articles in media studies), the nature of cases (i.e. did country A make a threat to country B, did country B back down as a result, etc.), or the responses from an open-ended survey. Coding is an issue whether you conducting qualitative or quantitative analysis–especially where you have captured large amounts of data. Often times the data is not inherently numerical and needs to be translated so that quantitative analysis can be conducted. Likewise, with a qualitative approach one still needs to categorize various data points to allow for meaningful comparisons.

The interesting thing about a service like Crowdflower is that it can leverage a ready group of workers globally who are ready and willing to conduct the coding at a reasonable price. Additionally, Crowdflower utilizes various real-time methods to ensure the quality of the coding. Partially this is achieved through the scoring of coders relative to their past performance, how they fair on tasks that are “planted” by Crowdflower (i.e. salting with tasks where the correct answer is known ahead of time), and how much agreement there is between coders on various items.

The final method comes up quite a bit in social science research when you have to determine how to categorize a given piece of data. The level of agreement is crucial to confidently coding a particular case. I would imagine that a platform such as CrowdFlower could make that task easier and more robust by quickly tapping into a larger pool of coders.

Has anyone used a service like CrowdFlower in this way (i.e. coding data from qualitative research)? Would be interested in your perspective.

Stanley Milgram

Gartner Vice President Kathy Harris discusses in some detail four areas of jobs needed in the near future. Though she never really uses the words “social networks” the implication is that most companies aren’t really geared toward taking advantage of the impact of these online communities, and that the numbers will be too large to ignore, regardless of the business you are in.

“Many of the needed technical capabilities originate in the social sciences and are aimed at usability and adoption of technology-related business services,” Harris said in a release. “These capabilities embody the notion of ‘action at the interface’ between the enterprise and its markets or between business management and technology management. Therefore, organizations are likely to shift the responsibility for leveraging technology outside centralized IT organizations and into the business units responsible for growth and innovation of revenue, products and services.”

Erving Goffman

To me, if you combine the plethora of data being generated by Web 2.0 technologies with the inherent social and behavioral aspects of these technologies, it screams for individuals that have training in sophisticated research methodologies (both quantitative and qualitative) as well as substantive subject’s that relate to sociology, psychology, and behavioral economics.  It may be creating a perfect storm where individuals with this particular skill set finally find themselves in high demand outside of the Ivory Tower.  As a trained social scientist myself, I also hope it puts to bed, once and for all, the short-sighted notion that the social sciences don’t really belong in the category of ‘science’ compared to their physical cousins.

(Via Jason Spector)

The first comes from GigaOM, where Jennifer Martinez looks at the emerging problem of trying to keep up with the constant flow of data via status updates.  As our networks grow, and our use of various social networks increases, we are inundated with updates which often times leads to missing particular updates that we may be most interested in.  Additional, she notes that besides missing out on information you care about, this stream overload can lead to “disjointed conversations that lack context, making it hard to piece together and decipher what it all means”.  I can relate to this problem, and my ‘immersion’ in social networks is average to above average.  I haven’t figured out an optimal way to keep up.  I try to utilize a few useful tools (e.g. Seesmic), but between social networks and Google Reader I find myself constantly playing catchup.

Michael Driscoll at Dataspora follows up on this theme providing a more high-level discussion of how the rise of data (vs. documents) conflicts with the architecture that underlies the web today.  Current mark-up languages are geared towards, and ideal for, documents (e.g. HTML and XML), not the kind of streaming data that will come to dominate content.  To explain this point he provides a comparison of metaphors where documents=trees and data=streams:

Trees are rooted and finite: you can’t chop up a tree and easily put it back together again (while XML has made concessions to document fragments, it is not a natural fit).

Streams can be split, sampled, and filtered. The divisibility of data streams lends itself to parallelism in a way that document trees do not. The stream paradigm conceives of data as extending infinitely forward in time. The Twitter data stream has no end: it ought have no end tag.

Conceiving of data as streams moves us out of the realm of static objects and into the realm of signal processing. This is the domain of the living: where the web is not an archive but an organism.

Finally, Ben Lorica at O’Reilly Radar discuss the challenges with trying to analyze large amounts of data in near real-time. As there are a number of potential solutions for the structured data that we are generating, there is a less obvious way to deal with the immense unstructured data. He notes recent work by a team at UC Berkeley that was able to take unstructured data and, leveraging entity extraction, turned it into structured data for a SQL database.

As suggested by Daniel Pink’s assertions on the rise of a right-brained working elite, the ability to extract stories from a world of increasing and abundant data will be increasingly critical to many industries. Indeed, the opening of U.S. federal government data at data.gov (and the appointment of Sir Tim Berners-Lee to similarly open the UK’s data archives) implies a new societal and cultural importance for data wranglers. (my emphasis)

The article also included some great links for those looking to get started examining this new trend.  They include:

1. The recently published book “Beautiful Data” brings together essays some of the world’s most cutting-edge data practitioners — such as Stamen Design — on subjects as diverse as DNA analysis, crime maps and crowdsourcing.
2. Ben Fry’s PhD thesis “Computational Information Design,” which outlines the need for a new field based on multiple disciplines.
3. The post “Three Sexy Skills Of Data Geeks,” which explains statistics, data munging and visualization — or studying, suffering and storytelling, as the author jokingly suggests.
4. Blogs such as Dataspora and Flowing Data.

Some people may be asking what the big deal is.  Statisticians have been around forever and their techniques have become more sophisticated over time.  The big deal is that it isn’t just about statistics and crunching numbers.  It is about combining multiple disciplines–such as statistics and graphic design–at a time of unprecedented data accumulation so as to glean better insights through the collection, analysis, and visualization of data.  Most companies claim to have a focus on ‘analytics’, but in my experience this term and its sophistication in a business setting varies widely.  Getting the most out of data requires leadership to think deeply and strategically about what kinds of data would be most useful, what kind of measures most illuminating, and how potential insights gleaned from that data might change their go-to-market strategy as well as R&D.  This should be correlated with a serious commitment to creating the necessary infrastructure (i.e. processes, systems) for collecting, analyzing, and visualizing the relevant data.  Like most things, it is a question of whether data and analytics are viewed as simply a nice feature or critical to growing and maintaining a business.  Will and vision, not just resources, are crucial.

For those that are interested in the new frontier of data and analytics I would also recommend the following:

• AI and Social Science
• John Udell’s Blog
• Data Wrangling
• O’Reilly Radar
• FlowingData Beginner’s Guide
• Visual Complexity

If anyone has additional links or recommended reading feel free to leave it in the comments section or email me.