Schwarz traces the history of baseball’s obsession with statistics to Henry Chadwick, a journalist and baseball writer widely acknowledged as the grandfather of baseball statistics.  Chadwick’s work in the mid- to late-19th century laid the foundation for much of the statistical framework through which we appreciate the game today.  Chadwick was adamant that the new game of baseball required a fair accounting of player performance:

In order to obtain an accurate estimate of a player’s skill, an analysis, both of his play at bat and in the field, should be made, inclusive of the way in which he was put out; and that this may be done, it is requisite that all…contests should be recorded in a uniform manner.

Anyone who has paid even scant attention to the debate regarding traditional and sabermetric approaches to the game of baseball will recognize Chadwick’s logic–not only do position players contribute to their team’s success by creating runs through their offense; they can also prevent runs by the other team through their defense.  Traditionally, players came to be valued and compensated based largely on their offensive production.  When it came time to arbitrate salaries or negotiate free agent contracts, offensive statistics carried the most weight.  (Whether or not the offensive statistics being used were the most accurate is a much larger debate, and Schwarz gives ample space to this history as well).

As the analysis of baseball became more sophisticated, analysts were finally able to measure a player’s total value by incorporating runs produced through offense as well as those saved by defense.  Rather than relying on the traditional fielding percentage (which simply measured the number of chances a fielder converted to an out), more sophisticated measures allowed for talent evaluators to look at how many runs a player saved in the field. Power hitting shortstops surely contributed to their team’s success by creating runs, but their light hitting counterparts could conceivable contribute just as much by saving runs.

Chadwick’s quote highlights two critical issues for any organization, and it is a theme that runs through The Numbers Game; ensuring that the metrics you rely on account for all of the ways a person contributes to success and that the data used to calculate those metrics is collected in a consistent, uniform manner. For years, teams overpaid for free agents whose offensive statistics were not viewed in the context of their defensive statistics. In some cases, a player’s offensive production was either tempered or cancelled out by their poor defensive play–in short, they were giving up as many runs with their poor defense as they were earning through their offense. While many organizations today have embraced the “analytics revolution” one wonders to what extent they truly know what metrics matter.  Like traditional owners and managers in baseball, it is likely that many businesses assume they know what the proper metrics are–what behaviors contribute the most to success.  The lesson from baseball is that accepted wisdom, personal experience, and “gut feelings” may be right, but they must be subjected to rigorous analysis to determine the best way to evaluate performance and drive success.

The Numbers Game is full of examples of how certain metrics came to be the accepted wisdom and the difficulty of any endeavor to unseat them in favor of better measures.  Besides baseball fans, anyone who is leading a business analytics initiative or responsible for performance and compensation will find this book rewarding.

This year I am in three leagues–one for work, one for “stat-geeks”, and one for family and friends.  I decided to tweak the way I approached the draft this year and wanted to share a bit of the strategy with readers.

One of the biggest problems with standard pre-draft rankings, particularly those of the big fantasy hosting sites (e.g. ESPN, CBS, etc) is that the rankings are based solely on aggregate measures of performance such as total projected points for the upcoming season.  Now, of course the goal is to assemble a team with players that end up scoring lots of points throughout the season, but total points scored ignores the fact that teams compete head-to-head, week-to-week.  In order to make the playoffs a team has to outscore opponents on a consistent basis in order to accumulate wins, not just points.  That means drafting players that not only score a lot of points, but score a lot of points week in and week out.  When it comes to deciding between which players to draft, managers would be better off selecting consistent scorers versus boom-or-bust players (at least, that is my hypothesis).

Let’s take a look at two hypothetical players:

Over the course of four weeks both players score the same amount of total points. However, Player A is clearly a boom-or-bust player while Player B is more consistent week-to-week.  Player A gives you a great chance to win Weeks 1 and 4, but makes it much hard to win in Weeks 2 and 3.  On the other hand, Player B is the model of consistency, giving you a great chance to win each week.  On most pre-draft rankings, Players A and B will look like equally valuable picks, but this is misleading.

This year, I decided to see whether a player’s penchant for boom-or-bust performances was at all consistent and predictable.  The initial answer seems to be yes.

I developed two metrics; one to evaluate high scoring consistency and one that takes predicted points and combines them with scoring consistency.  The first, ConBoom, measures, weights, and then combines the number of times a player scored >=20 points, >=15 points, and <10 points per game over the course of a season.  This is the foundation of the consistency metric.  ConRank combines the ConBoom score for a player with a weighted measure of that player’s predicted total points for the upcoming season.  (How am I weighting each component of the measures?  Well now I can’t reveal the entire secret sauce, now can I?)

I validated the measures against the past three years of actual player data and found that ConBoom scores from one year were highly correlated with ConBoom scores the next year (.70).

I am going to use the new metric to guide my drafts in all three leagues and essentially test how my teams fair against other teams over the course of the season.  With data and predictions for every player I’ll be able to test the method over three league scoring systems, 32 teams, and 256 games as well as validate the measures predictive attributes over another year.

Draft number one is tonight.  Let the games begin!

A loose consensus has formed around the idea that basing CEO pay on, say, five years of stock returns would eliminate some of the reckless decision making that led to the Great Recession. But I suspect that even if you could build a compensation plan that focuses on long-term shareholder value, you’d solve only part of the problem.

That’s because such a scheme still ties CEOs’ motivation to one fickle number—company share price—and assumes that pay alone motivates chief executives to perform.

Any number of things can motivate CEOs—peer recognition, for example, and even a desire to change the world. In fact, CEOs usually have all the money they need. Why then does it seem that they care more about stock value and the compensation it produces than those other forms of motivation?

The answer is almost uncomfortably simple: CEOs care about stock value because that’s how we measure them. If we want to change what they care about, we should change what we measure.

Ariely goes on to argue that, in general, people align their behaviors with the criteria that others judge them on.  He suggests a number of additional metrics that CEO’s performance should be judged by, such as jobs created, pipeline of new products and patents, satisfaction level of customers, and trust in your company and brand, but says that these are difficult to measure.

I largely agree with Ariely, but slightly disagree on two points:

First, it isn’t simply that what gets measured gets done, but rather what gets measured and valued gets done.  It is a slight difference, but an important one.  Businesses today are flush with data, statistics, metrics, and dashboards.  The decision as to which of those metrics to value, i.e. to use as a measuring stick of employee (and, in this case, CEO) success is an important one.  Take an example from baseball.  For decades, managers, talent scouts, and front office executives paid little attention to walks and on-base percentage.   Both were measured, but they weren’t properly valued–players with a high number of walks and a robust on-base percentage were not compensated at a high level because of their performance in these two areas.  It isn’t enough to measure–specific measures must also be publicly valued in order to affect behavior in the way that Ariely suggests.  The issue of what gets valued is an issue of culture change to a large extent–it certainly isn’t easy, but without it measurement will not affect behavior.

Second, many of the items that Ariely suggests are quite measurable.  My current employer, Gallup, is a global leader in the science of measuring customer satisfaction (or, as we would argue, customer engagement) and brand confidence or trust.  Specifically, Gallup has developed metrics for both that have been validated in relation to business outcomes (e.g. customer retention, profitability, etc).  Often times we assume that things are inherently difficult or impossible to measure and, therefore, fail to measure them at all (to be fair, Ariely doesn’t suggest this, he simply states that it is more difficult).

Combine sophisticated data analytics with the fact that people, it turns out, are actually rather predictable in their routines and you end up with a myriad of possibilities for business and public policy.

Via Alex Lundry

[Note: I will be traveling over the next few days, so posting will be light-to-nonexistent]

[...] IBM is unveiling a new internal analytics product that the company is touting as the “largest private cloud computing environment for business analytics in the world,” which launches internally with more than a petabyte of information. Along with this internal product, IBM will launch a companion product for clients to build upon this cloud-based architecture, called IBM Smart Analytics Cloud.

The internal product, dubbed Blue Insight, will provide 200,000 employees in IBM’s sales and development department with the ability to extract data and information to make decisions and gain further insight at the point of sale. Blue Insight will gather information from nearly 100 different information warehouses and data stores, providing analytics on more than a petabyte (1,000 terabytes or 1,000,000 gigabytes) of data. For example, sales execs may use customizable queries of real time data to understand revenue opportunities and how many sales in their region are closing to help improve prediction. Or a manufacturing process engineer can evaluate real-time data on the plant floor to identify trends and data to improve yield and reduce shipment delivery times.

IBM Smart Analytics Cloud offering for clients will similarly deliver powerful business intelligence via the scalable, private cloud. The product lets the client import data and than transform this information into insights to develop strategies and decisions. The service will offer the ability to create reports, analysis, dashboards, and scorecards to monitor business performance and measure results.

The key to good analytics is not just the accumulation of relevant data, but also the ability to manipulate and visualize the data in meaningful ways.  The ability to create custom dashboards, etc, is an important part of making analytics a useful and profitable tool.  IBM has certainly dedicated itself to expanding its service and software offerings relative to its traditional hardware business, and analytics has emerged as a huge part of that strategy.  In doing so, it should be well positioned to profit from the coming analytically-driven world.

Well, it is performance review time at work and this reminded me of a post I’ve been meaning to write for a while.

An issue that has always interested me is how organizations measure individual performance.  Organizations have finite resources and therefore must deploy those resources in the most efficient manner, maximizing their value.  Given the large percentage of resources invested in personnel, organizations have a huge incentive to get those investments right.  However, calculating an accurate ROI on employees is probably one of the hardest things to do.  To explore why–and how it might be done better–I turn to the world of sports, baseball in particular.

There has long been a debate within the Sabermetric community (and between purists and Sabermetricians) regarding the statistical relevance of “clutch”: the ability of a player to elevate their performance in key situations in a manner significantly different from their performance in normal situations.  Early research by such pioneers as Bill James found that the attribute of clutch didn’t exist–much like the idea of a “hot-hand” in basketball, the appearance of a clutch performance (e.g. a usually mediocre batter managing to hit .500 in a playoff series) was nothing more than a statistical artifact.  If you were to look at any 5-7 game stretch during the 162-game regular season you are just as likely to find even average hitters going 4 for 8 or 8 for 16 as you would be to find them going 2-8 or 0-16.  Over a long enough time period, these streaks even out and players regress to their mean performance.  Basically, if you are a .250 hitter, over a long enough period of time your performance will return to its mean, despite occasional swings to the extreme left and right sides of the bell curve.

More recent studies have looked to expand upon earlier research and refine how we search for clutch performances.  A common way to do this is to not look at single games that were more important (e.g. post-season play), but rather particular moments that alter the probability of a team winning that particular game.  This approach has been termed “leverage“:

the swing in the possible change in win probability. If there is a game with one team leading by ten runs, the possible changes in win probability, whether the event is a home run or a double play, will be very close to negligible. That is, there won’t be much swing in any direction.

But, in a late and close game, the change in win probability among the various events will have rather wild swings. With a runner on first, two outs, down by one, and in the bottom of the ninth, the game can hinge on one swing of the bat—a home run and an out will both end the game, but with vastly different outcomes for the teams involved.

From left to right: A-Rod and Jeter

An excellent example of this debate over which players are clutch and which are not is the case of Alex Rodriguez. Long touted as one of the greatest hitters of all time, A-Rod has consistently been criticized as a non-clutch performer, accused of accumulating the bulk of his stats in low-leverage (i.e. low-pressure) situations. One could not listen to NY sports radio over the past 5 years without hearing fans decry the ‘superficial’ performance of the Yankee slugger while praising the clutchness of his statistically less impressive teammate, Derek Jeter. Part of the reason for this perception is the effect of high-impact events on perception and the framing of the debate by the media. But when we look at both players’ statistics in leveraged situations a much different picture emerges:

[As of 2008] Mr. Rodriguez has hit for the clutch throughout his remarkable, surefire Hall of Fame career. His career OPS in high-leverage situations is .975. In medium-leverage, it’s .960. And in low-leverage, it’s .972. That’s consistent with the American League as a whole during his career, when each year batters in high-leverage situations hit somewhere between 1% worse and 6% better than they did in low-leverage situations.

Additionally, if you look at the playoffs (where most look for clutch performance), A-Rod has accumulated the following statistics (including this year): .299BA/.393OBP/.958OPS. Here’s Jeter’s: .309BA/.379OBP/.858OPS. In series where his team is playing for a spot in the World Series, A-Rod’s stats are even better–Jeter’s go south.

The point is not to cheerlead for Alex Rodriquez, but rather to point out two points that can be applied to any organization and field when thinking about the value of personnel:

1. When evaluating performance we should pay attention to results over the long term, not an arbitrary chunk of time. Anyone can ‘get hot’, but that doesn’t mean the individual willed themselves to a better performance or that current success is a predictor of future success. Discrete outcomes are quite dependent on factors outside of an individual’s control as well as the randomness of performance. This works both ways–for successes and for shortcomings.
2. Single, high-impact events can significantly skew our perspective. Objectively evaluating the quality of individuals is difficult since we tend to use anecdotes as mental shortcuts. Big wins are important, but they shouldn’t blind us to the overall performance of an individual. Those big moments could easily be the result of random chance rather than talent. Now, we all benefit from luck from time to time. However, much time and treasure has been wasted on individuals who ‘came up big’ at one time and management is just waiting for them to rise to the occasion yet again. This applies equally to ‘hot’ baseball prospects, CEO’s, coaches, sales personnel, programmers, managers, etc.

So what should we do? How do we better measure human capital performance? I think partly we should take a page from baseball. New measurements have emerged over time to better evaluate player performance. Taking an analytical approach to measuring performance is step one. It shouldn’t be the only measurement, but without appropriate data organizations are simply basing their evaluations on subjectivity and intuition–you need a balance. With new measurements comes a new mindset; the prism through which we view the world changes so that instead of looking for those ‘big moments’ we now look for consistent performance over the long term. A healthy appreciation for randomness, chance, and methods for accurately measuring performance are all good first steps towards more accurately evaluating human capital.

For those that made it this far, thanks for sticking with me on this one.  Would love to hear your thoughts.

This underscores the comparative advantage of individuals with skills amenable to collecting, coding, manipulating, and visualizing Big Data in the current labor market.  Additionally, as not all data will be easily collected and coded via computer programs, services that can efficiently harness the crowd in support of Big Data will also be critically important.

A visualization of thousands of Wikipedia edits that were made by a single software bot. Each color corresponds to a different page.

Photo via Wired

The first comes from GigaOM, where Jennifer Martinez looks at the emerging problem of trying to keep up with the constant flow of data via status updates.  As our networks grow, and our use of various social networks increases, we are inundated with updates which often times leads to missing particular updates that we may be most interested in.  Additional, she notes that besides missing out on information you care about, this stream overload can lead to “disjointed conversations that lack context, making it hard to piece together and decipher what it all means”.  I can relate to this problem, and my ‘immersion’ in social networks is average to above average.  I haven’t figured out an optimal way to keep up.  I try to utilize a few useful tools (e.g. Seesmic), but between social networks and Google Reader I find myself constantly playing catchup.

Michael Driscoll at Dataspora follows up on this theme providing a more high-level discussion of how the rise of data (vs. documents) conflicts with the architecture that underlies the web today.  Current mark-up languages are geared towards, and ideal for, documents (e.g. HTML and XML), not the kind of streaming data that will come to dominate content.  To explain this point he provides a comparison of metaphors where documents=trees and data=streams:

Trees are rooted and finite: you can’t chop up a tree and easily put it back together again (while XML has made concessions to document fragments, it is not a natural fit).

Streams can be split, sampled, and filtered. The divisibility of data streams lends itself to parallelism in a way that document trees do not. The stream paradigm conceives of data as extending infinitely forward in time. The Twitter data stream has no end: it ought have no end tag.

Conceiving of data as streams moves us out of the realm of static objects and into the realm of signal processing. This is the domain of the living: where the web is not an archive but an organism.

Finally, Ben Lorica at O’Reilly Radar discuss the challenges with trying to analyze large amounts of data in near real-time. As there are a number of potential solutions for the structured data that we are generating, there is a less obvious way to deal with the immense unstructured data. He notes recent work by a team at UC Berkeley that was able to take unstructured data and, leveraging entity extraction, turned it into structured data for a SQL database.

Your.flowingdata is a great interface that allows you to track your daily behavior through quick, direct messages on twitter.  Want to track how far you walk, how much you eat, how often you spend online versus reading printed media, etc?  The program allows you to interact with the data in a number of helpful ways and, if you like, share your data with friends or download the data and conduct your own analysis.  Nathan (the creator of FlowingData.com and, of course, your.flowingdata) has a post detailing several updates to the program since the July launch.

Like any data analysis exercise, the payoff can be uncovering patterns in your behavior that you simply weren’t aware of since we a) tend to lose perspective in the moment,  and b) we also our memories tend to be biased (i.e. recalling more recent events prominently, or emphasizing extreme events over mundane ones).

The key is that the program also provides various visualization tools so that you can very easily view the patterns as they emerge.  This kind of feedback is critical if you are serious about altering your behavior.  Maybe you eat too much, too late at night, aren’t getting enough sleep, spending too much time online.  In some cases you likely have an idea that your behavior is out of whack, but in other cases you may not realize the impact that certain behaviors are having.  Visualizing your personal metrics can be a real eye-opener and help you to make productive changes in your life.