[M]ost people respond to a critical question by arguing against the reasoning of whoever asked the question. They offer all of the evidence they can think of, hoping to make their case overwhelming. They shoot at an attack sixteen times with bullets of data to make sure it is dead. But in so doing, they are arguing not on their own but on the naysayer’s territory, opening themselves up to counter-attacks with each piece of evidence they dispense — and simultaneously putting other listeners to sleep!

I have seen far more success when people offer a quick, direct, common sense answer that shows respect for the naysayer but moves the discussion along. It is important to strike a balance between addressing a naysayer’s concern and keeping each question-and-answer brief in order to hold your audience’s full attention. To use economics terms, there are diminishing marginal returns to data-dumping in your answers. Great leaders throughout history, from Gandhi to Sam Walton, have always employed this principle to maximum effect. They knew the power of clarity and simplicity. And they found that using it allowed them to connect with more people and win more hearts and minds.

The next time you present an idea on an important new marketing campaign, for example, and someone rebuts it by citing five previous times that your company tried a new marketing campaign and it was unsuccessful, you have two options. You could go through each of the five examples, explain their flaws in detail, and demonstrate how each of those flaws does not apply to your idea. Or you could say, “There are always examples of failed attempts to do anything of real importance, and we did indeed learn from the experiences you cite. But we cannot allow these past failures to keep us from adapting to a changing world or else we would never move forward on anything.”

I am sympathetic to the notion of using less data in presentations as well as being less verbose, but Kotter seems to be conflating the two.  Short, concise statements that lack adequate evidence and data are just as likely to get shot down as long, laboring statements that include reams of data.

Kotter is right that data dumps are a bad thing, but his example has it’s risks.  If your audience includes folks with great BS detectors they won’t let you get away with that statement.  Sure, it’s short.  But it lacks any rationale for why the campaign being pitched shouldn’t be judged by those previous failures.  How hard would it be to briefly state that the factors that led to the previous failures do not apply to the current case?  You don’t need to get into the weeds on each factor, but you have to give people a reason to buy in to your current proposal other than “we need to keep trying new things and taking risks”.  Statements devoid of evidence are just as useless as data without purpose or context.

For me, it’s about right-sizing and being selective with the data you use, not banishing data and evidence in favor of simple statements or platitudes out a fear of alienating your audience.

Apple-polished Y-axis

Correct Y-axis

The book is packed with great examples.  However, Seife spends a bit too much time on some cases.  More variety would have made the book better.  Seife also tends to focus on the intentional manipulation of data while ignoring the unintentional instances.  There is no doubt that people use many of the tricks he describes to bend data to their advantage, but often times misleading data is the result of people simply making bad calculations rather than purposeful manipulation.  Additionally, Seife’s suggestion as to how to combat proofiness, mathematical sophistication, doesn’t seem capable of solving the problem on its own.  While I agree that the public could benefit from a more robust understanding of numbers and their manipulation, Seife basically ignores the issue of perceptual bias.  Even the most sophisticated consumers of data are subject to fundamental perceptual biases.  Given that we are “predictably irrational”, to quote Dan Ariely, any solution must also take into account that we are hardwired in many ways to be manipulated by proofiness.

The book is not for deep subject matter experts in mathematics or statistics, but it is a fantastic primer for the lay person.

The monthly gas-production data, known as the 914 report, is used by the industry and analysts as guide for everything from making capital investments to predicting future natural-gas prices and stock recommendations. But the Energy Information Administration (EIA), the statistical unit of the Energy Department, has uncovered a fundamental problem in the way it collects the data from producers across the country—it surveys only large producers and extrapolates its findings across the industry. That means it doesn’t reflect swings in production from hundreds of smaller producers. The EIA plans to change its methodology this month, resulting in “significant” downward revision.

The gap in output between what the 914 report has been predicting and what is actually occurring has been growing larger and larger.  Many analysts have long suspected the methodology underlying the reports was faulty, but the EIA has been slow to revise it.  The overestimation of output has depressed prices, the lowest in 7 years.  Any revision to the methodology will bring about a “correction” in energy markets and particular states will surely see their output dip significantly.

So what can we learn from this from a methodological perspective?  A few things:

1. How you cast the die matters: The research methodology that we employ for a given problem significantly impacts the results we see and, therefore, the conclusions we draw about the world.  The problem with the DOE’s 914 report wasn’t simply a matter of a bad statistical model, it was the result of unrepresentative data (i.e. relying only on the large producers).  This isn’t simply an issue of noisy or bad data, but of systemic bias as a result of the methodology employed by the EIA.  The data itself is seemingly reliable.  The problem lies with the validity of the results, caused by the decision to systematically exclude small producers and potentially influential observations from the model.
2. Representativeness of data doesn’t necessarily increase with the volume of data: More than likely the thought went that if the EIA collected data on the largest producers they’re extrapolations about the wider market would be sound–or close enough–since the largest players tend to account for the bulk of production.  However, as we see with the current case, this isn’t necessarily true.  At some point in history this methodology may have been sound, but it appears that changes to the industry (technology, etc) and the increased importance of smaller companies have rendered the old methodology obsolete.  Notice that the EIA’s results are probably statistically significant, but achieving significance really isn’t that difficult once your sample size gets large enough.  What is more important is representativeness–is the sample you’ve captured representative of the larger population?  Many assume that size and representation are tightly correlated–this is an assumption that should always be questioned and, more importantly, verified before relying on the conclusions of research.
3. Hypothesis-check your model’s output: The WSJ article notes that a number of independent analysts long suspected a problem with the 914 reports by noticing discrepancies in related data.  For example, the 914 report claimed that production increased 4% in 2009.  This was despite a 60% decline in onshore gas rigs.  If the 914 report is correct, would we expect to see such a sharp decline in rigs?  Is this logically consistent?  What else could have caused the 4% increase?  The idea here is to draw various hypotheses about the world assuming your conclusions are accurate and test them–try to determine, beyond your own data and model, whether your conclusions are plausible.  Too often I’ve found that business fail to do this (possibly because of time constraints and less of a focus on rigor), but academics often fall into the same trap.

David McCandless is a “visual journalist” who specializes in visualizing data across numerous subjects. In his latest work for The Guardian’s Data Blog, David visualize a ton of data regarding troop deaths and injuries, size of forces by country, as well as the civilian toll in Afghanistan.

Like all good visualizers, David tells a story with his infographics, putting many issues in perspective (likely creating some “oh, I didn’t realize” moments for readers) through the use of relational data. For example, David compares absolute measures of troop fatalities by country to fatalities as a percentage of total troops deployed. What one sees is that while the US has lost the most troops by far, the Canadians have lost the most troops as a percentage of those they have deployed:

The piece is chock full of infographics like this. One issue I have with David’s analysis comes less from the data (and it’s visualization), and more with his commentary. One infographic depicts the number of troops in Afghanistan by country or organization (e.g. NATO, etc). David includes a bubble for private security contractors (PSC), which I think is great as it is a key statistic that we should be taking into account. However, while his own data shows that only 3,000 of these contractors are armed he makes the comment that “that’s a huge amount of hired guns”. Now David may just be using the common phrase of ‘hired gun’ to refer to all PSCs, but when talking about PSCs such a phrase implies something very specific (i.e. contractors that carry and use weapons in theater–not just logistical support, etc). If he isn’t just using the phrase in the generic sense then the claim is overblown. By his own numbers, armed PSCs only make up a little over 1% of the entire fighting force in Afghanistan (3,000 out of 292,486). I am not sure 1% constitutes a lot of hired guns.

The best part about how David operates is that he provides links to all his data (including what was used for this article) via Google Docs. This is a great practice, one that encourages readers to check his work, look for additional patterns in the data, and ensures that any errors in presentation or interpretation can be brought to light and discussed. (David has altered other infographics based on reader feedback.) I wish more people would adopt the practice.

In any event, be sure to check out David’s blog and other work.

How can we then decide that a campaign was “better” than another one? We rarely look at a campaign data – partly because the actual metrics data is proprietary and not available to anyone beyond walls of an agency and of their clients…

General, and generally available, feedback mechanisms and benchmarks for success don’t really exist. While it may not have been possible before to know exactly if a TV/print/outdoors/radio campaign influenced particular brand affinity and purchase decisions, digital lets us do things differently.

This means that we don’t have to judge works of others purely on elusive criteria of “creativity”, but on actual data on how this creativity fared with people (what did they do? and what did they do next?).

Read the whole thing.

For those that don’t know him, Rosling is something of a superstar at the intersection of public health research and data visualization.  Rosling’s recent work focuses on dispelling the misconception that there is a binary distinction between the developed and developing worlds.  While Rosling is not the first to argue against this dichotomy (and all its political ramifications), his approach is both fascinating and affective.  Rosling utilizes an impressive visualization tool–that he developed and Google bought–to present data on the progress of the developing and developed worlds in a relational manner.  The video below was shot in June of 2009 at a TED-sponsored talk at the US State Department.

Rosling’s presentation drove home two points for me: the importance of challenging our assumptions, the impact of data visualization, and the power of relational data.

1. Too often we take for granted what the facts are around a given issue. What was true yesterday may not be true today; however, what was true yesterday undoubtedly contributes to our mindset today and the prism through which we view today’s issue. How does one combat this characteristic of human perception? The review of data. We should be challenging our assumptions, testing them against meticulously collected data to ensure that what we once that of as conventional wisdom still applies. If not, change it.
2. How data is visualized can have as big an affect on helping us to alter our views. The data Rosling relies on is not new or proprietary (heck, I used some of it when I lectured on the political economy of development). What is innovative and impactful is how we presents the data.  Rosling’s software visualizes the data in a way that is more powerful than simply providing a list of child mortality rates or the percent change over time for each country.  It allows him to tell a story, to weave a narrative which connects the audience to the data in a way a regular bar chart or data table simply could not do.
3. The most powerful way to visualize data is relationally.  Rosling’s visualizations make full use of this principle, presenting country data relative to other countries as well as across time.

When I think of the difference a data-driven world can and will make, it is this–the ability to view the world and make decisions in a fact-based manner.  We can better separate the wheat from chaff, signal from noise.  We can make our mind set match our data set.  And we are just scratching the service.

Yup, looks about right.  Of course, as Nate points out, there are ways to widen the valley.  But I’ll leave that for another post.

Authorship over time

So what are the implications for universal authorship?  SEED mentions one and I have another that comes to mind.

The first implication is in the article: as more and more people become creators of content it will become increasingly difficult for organizations of all kinds to control their messaging and their brand.  Provide a bad customer experience and it will be on facebook, immediately broadcast to hundreds if not thousands of people.  Discriminate against someone because of their race or sexual orientation and they will tweet about it, likely prompting dozens of re-tweets which allows the issue to reach exponentially more people, who then might blog about it, and so on, and so on.  Organizations are already finding it hard to control their brand, imagine when the number of authors increases 10 fold (which, according to SEED, will now happen yearly).  Theoretically, firms and technologies that can monitor content related to an organization will be big winners.  Additionally, organizations will need to invest more heavily in their own networks and crowds to help combat negative content (whether true or false).

Another implication that immediately came to my mind is the increased difficulty in separating signals from noise.   As the cost of entry into the market for content heads towards zero and the tools of creativity are fully democratized, there will be an even greater explosion in content from which individuals and organizations will have to separate relevant, accurate pieces of information.  There is already a flood of information to wade through and it is becoming more difficult to do so.  Exponentially increase the amount of content and the variety of sources and you’ve taken the problem to an even greater level.  All things being equal, more content and a more fractured supply source will only increase the amount of noise and make identifying the signals (the accurate pieces of information) more difficult.  What we will need, and what will become valuable, are services that don’t simply aggregate content, but also determine their level of accuracy and credibility.

Nate Silver

I am thinking here of services that mimic the approach of Nate Silver at FiveThirtyEight.  The world didn’t lack political polls, but it did lack a methodology for cutting through the noise created by dozens of polls, many providing contradictory predictions.  Silver came up with a way to not just aggregate polls, but to increase the ratio of signal to noise, allowing for a more accurate portrayal of public opinion and ultimately a prediction of Presidential elections.  Silver made polls better by developing what I would call a sophisticated aggregation methodology (note: I will be writing more on this soon).  The key will be to develop a scalable, replicable approach that can be applied to a variety of domains.

Soon enough, we may all be creatives.  And that means there will be a heck of a lot more chaff to wade through.

This underscores the comparative advantage of individuals with skills amenable to collecting, coding, manipulating, and visualizing Big Data in the current labor market.  Additionally, as not all data will be easily collected and coded via computer programs, services that can efficiently harness the crowd in support of Big Data will also be critically important.

A visualization of thousands of Wikipedia edits that were made by a single software bot. Each color corresponds to a different page.

Photo via Wired