Decomposition

Talk on Ranking NFL Teams

Fri, 22 Feb 2013 20:34:02 -0500

I gave a talk last month at the New York Open Statistical Programming Meetup on ranking systems, specifically applied to the NFL. You can find slides, code, and an IPython notebook which contains most of the information. I encourage you to look at the slides, which I spent a lot of time on. They contain two embedded interactive visualizations. I did get my Super Bowl prediction wrong, though.

There were about 200 attendees, but unfortunately there is no video of the talk. Thanks to everyone who came; it was incredibly fun for me.

The talk was mostly a review/comparison of different methods:

Pythagorean wins
Eigenvector methods
the Bradley-Terry-Luce model
optimal rankings

The last one warrants more explanation. I had previously reviewed the optimal descriptive ranking problem and my solution. It’s a fascinating application of graph theory to a problem that most people wouldn’t consider to be graph-theoretic. Once the ranking problem is posed as a topological sort of a graph which contains cycles, it’s easy to describe an exact (if non-unique) solution as well as find an algorithm to approximate it. The results are quite stunning: a 10+% increase in the number of correctly described games from the other models.

NFL Fans on Facebook

Mon, 28 Jan 2013 19:18:00 -0500

NFL Fans on Facebook:

I made a promise to myself when I interned at Facebook that I would write at least one blog post on the NFL. Today I got to publish it! Technically this was all quite trivial, mostly just aggregating users by teams and geography, then producing the maps using D3 (I had to rasterize them to post them on Facebook).

The NFL friendships thing involved a join on the social graph with the Like graph. If I had had more time, I would have looked for rivalries. My plan was to look for friendships that were unlikely holding relative geography fixed but changing the two teams. The idea was to look for some pairs of teams that despite having pairs of fans in close proximity are unlikely to have fan friendships.

The finding that winning is correlated with Likes is also interesting. To an economist, this would seem to be an excellent instrumental variable. (This is not a new idea). Conditional on the spread of the game, wins and losses are basically coin flips, and yet they seem to be correlated with big increases in team fan bases. This strategy could potentially be used to identify peer effects if one were willing to make a bundle of assumptions about dynamics.

Real scientists make their own data

Fri, 25 Jan 2013 16:13:00 -0500

Around budding social- and data scientists, a question you often hear is “where can I get data?” It happens so often that people like Hilary Mason, who I’m sure gets this question all the time, have posted pages with resources. Getting new data can be just what you need to practice a technique you are learning or complete a project that you can publish or add to your portfolio.

Here I argue that if you want to make a bigger impact as a scientist, you should make your own data instead of downloading it. Here are my points:

Historically science has been about observing (and sometimes manipulating) phenomena. Some of the most important contributions across all fields of science have occurred through actual hands-on data collection.
Making your own data means you are creating new facts about the world which gives you privileged access to scientific findings. Novel data is a source of competitive advantage that is sustainable, unlike being clever about your analysis.
When you invest in making your own data, you are forced to consider your research question long before diving into analyses. Having data that were costly to gather or create conveys that you are a thoughtful, careful researcher who knows a lot about her domain. And it probably means that you are addressing an important question.
If you are the creator of your data set, then you are likely to have a great understanding the data generating process. Blindly downloading someone’s CSV file means you are much more likely to make assumptions which do not hold in the data.
Making your own data means you can run randomized experiments, which enable you to make causal inferences. This is a big deal if you want to make any claims about policy decisions.

Think about a famous scientist who has inspired you. It’s likely he or she invested heavily in data collection. From Aristotle to famous natural philosophers—such as Galileo—practicing what evolved into modern science, there is rich tradition of making careful observations of interesting phenomena. Without modern methodology and theory, these proto-scientists were making important discoveries through careful record-keeping.

Gregor Mendel spent years breeding bees and pea plants in order discover the laws of modern genetics. Ronald Fisher, who we tend to regard as a statistician due to his methodological contributions, was actually coordinating large-scale experiments with crops at the Rothamsted Experimental Station. They had to use elbow grease and waited a long time to make important new discoveries.

Stanley Milgram is perhaps the best example of making your own data. The creativity in his work was apparent by how he approached difficult but important questions. He didn’t throw his hands up and resort to casual theorizing, he invented novel empirical strategies like sending chain letters to find answers.

This is my thesis:

Your best chance to make a serious contribution as a business or academic researcher is to find, make and combine novel data.

Almost everybody in your field will be as well-trained as you are. They will be able to run a Google search for data sets just as you can, and they will be able to apply methods like regression, clustering, visualization, etc on the data they find. If you want to compete, I suggest you allocate a substantial portion of your effort toward both 1) asking excellent questions and 2) constructing your own data which are suitable to directly answer them.

Many social/digital scientists are reluctant to invest in making data because it’s much more costly and risky than analyzing data you already have available. Sure it’s a gamble, but the payoffs can be substantial, both for science and for your reputation. As a thought experiment, I urge you to consider the last time you were really wowed by a finding that wasn’t produced using new data.

But how can you make your own data? Here are my suggestions:

If you already work as a data scientist for a company with a product or service, add your own instrumentation instead of relying on logs. Get permission to run randomized experiments which can tackle tough questions.
If you don’t work for a company, ask to partner with or intern for one. If you are answering a question they find interesting, and you are willing to help plan the data collection and analysis, you might just get their attention. Help out at DataKind and you’re almost guaranteed to get new data that no one has ever seen before.
Experiment on or survey your friends. With social networks and kind friends (and maybe some promises of free drinks), you can often get a large enough convenience sample to test a hypothesis.
Buy some new friends. Many excellent behavioral studies have been conducted on Mechanical Turk, including some in progress by famous data scientists.
Build your own website or application. This is the costliest route, but allows you the most control. I spent two years building Creds so I could study how beliefs about uncertainties are correlated in social networks. Some researchers at Boston University built a news reader application to study how humans allocate attention to news articles. This is a serious gamble that could pay off in a big way.

I’m sick of seeing the same old data sets recycled with a slightly new analysis done. Use old data to practice techniques, but if you want to get serious about being a scientist, make your own data.

Creds Blog: The Simon-Ehrlich Wager and Another Role for Markets

Mon, 14 Jan 2013 17:25:06 -0500

Creds Blog: The Simon-Ehrlich Wager and Another Role for Markets:

Creds is a side-project I’ve been working on for awhile now. I encourage you to 1) check it out and 2) read this post to see my inspiration.

credsblog:

In 1980, an economist named Julian Simon and a biologist named Paul Ehrlich had a disagreement about the future scarcity of natural resources on Earth. Being gentlemen, they decided to stake some token money-as well as their reputations-on what they believed.

The Statistics Software Signal

Thu, 03 Jan 2013 11:52:00 -0500

Last night on Twitter, I went on a bit of a rant about statistics packages (namely Stata and SPSS). My point was not that these software packages are bad per se, but that I have found them to be correlated with bad quality science. Here is my theory why.

When you don’t have to code your own estimators, you probably won’t understand what you’re doing. I’m not saying that you definitely won’t, but push-button analyses make it easy to compute numbers that you are not equipped to interpret.
When it’s extremely low cost to perform inference, you are likely to perform a lot of inferences. When your first regression gives a non-result, you run a second one, and a third one, etc. This leads untrained researchers to run into multiple comparisons problems and increases the risk of Type I errors.
When operating software doesn’t require a lot of training, users of that software are likely to be poorly trained. This is an adverse selection issue. Researchers who care about statistics enough should have gravitated toward R at some point. I also trust results produced using R, not because it is better software, but because it is difficult to learn. The software is not causing you to be a better scientist, but better scientists will be using it.
When you use proprietary software, you are sending the message that you don’t care about whether people can replicate your analyses or verify that the code was correct. Most commercial software is closed source and expensive. We can never know if the statisticians at Stata have a bug in their code unless we trust them to tell us. Also consider researchers from schools or companies which can’t afford expensive commercial software. Should they not be able to reproduce your results?

I do think these packages are valuable can be used for good. I have used Stata and it has saved me plenty of time. My main point is that there are a number of mechanisms through which bad science can be correlated with using push-button statistics software, not that one is a direct consequence of the other.

What your statistical software says about you (to me):

R: You are willing to invest in learning something difficult. You do not care about aesthetics, only availability of packages and getting results quickly.
Python or JVM languages: You are a hacker who may have already been a programmer before you delved into statistics. You are probably willing to run alpha or beta-quality algorithms because the statistical package ecosystem is still evolving. You care about integrating your statistics code into a production codebase.
Julia: You are John Myles White.
Stata: You are an economist who doesn’t care to code your own estimators, probably because your comparative advantage lies elsewhere. Possibly you are doing sophisticated work with panel data where Stata is the only game in town. You don’t care that you can’t do proper programming because you’re not a programmer.
SPSS: You love using your mouse and discovering options using menus. You are nervous about writing code and probably manage your data in Microsoft Excel.
Matlab: You definitely know what you’re doing and you care about performance. You know Matlab is expensive but you aren’t the one paying for it. You live in a bubble where everyone you know uses Matlab.
Mathematica: You are an aesthete who believes everything Stephen Wolfram says.
SAS: You are an analyst for a large pharmaceutical company, and SAS is all you have ever known. You have a large library of custom SAS macros, so that (clearly) makes you a programmer. That anyone would want to hand-code statistical methods leaves you utterly baffled. If SAS does not ship with a particular statistical method, then it probably isn’t important. (h/t Chris Fonnesbeck)

Trouble in the Sandia Mountains

Thu, 06 Dec 2012 11:12:00 -0500

Last night I made a series of decisions that seriously jeopardized my life.

The Sandia mountains rise up to the East of Albuquerque and provide views of a beautiful Southwest sunset. You can take a cable car up to a lodge at the crest of the mountains. When we arrived at the departure point, I decided I would go for a quick trail run when we got to the top. Having been trapped in the car all day on our way from Flagstaff, I was excited to move. I had been running mostly in San Francisco the last few months and missed the rugged terrain I frequented while I was living in Palo Alto over the summer.

But more than that, I had hiked on South Crest Trail before, six years ago when visiting Albuquerque with a then-girlfriend. I thought the trail was amazing and wanted to push further, but she was pragmatic and made me turn around earlier than I wanted. From the cable car you can track the crest of the trail and see dozens of spectacular Western-facing viewpoints along the way. I thought this was a great opportunity to see some of the views that I had missed a few years ago.

We arrived with only 75 minutes of sunlight left, so I dressed quickly and left my phone in the car. It was 65 degrees, sunny, and calm at the base of the mountain, but in the waiting room for the cable car, there is a weather station showing that going up to 10,000 feet of elevation would drop the temperature to 36 degrees and increase the wind to 28mph. I looked down at my shorts and running hoodie and decided it would be ok but that I was pushing it.

When we got the top I quaffed some water at the fountain in the tram station and took off to the South. It was immediately familiar so I stopped to enjoy a few views very briefly but pushed on to see what was further. The trail was rough: lots of rocks and quick elevation changes, surrounded by dense pine trees. I figured I would run about 25 minutes out and then turn around in time to use dusk-light to find my way back. The trail didn’t seem complicated enough to get lost with only a couple of obvious-seeming turns.

Sadly, the trail meandered to the left along the Eastern side of the ridge. This meant two things: no views of the sun setting to the West and it was getting darker faster than I thought. I figured if ran a bit further eventually it would come back up the crest and I could catch the beginning of the sunset. I picked up my pace.

This actually worked-as the trail winded to the right almost 180 degrees and ended up giving me some of the views I was craving. Perhaps it was altitude-induced optimism, but I thought that maybe it would provide me an alternative route home that wouldn’t require retracing all my steps. I pushed on for longer than I should have before realizing I was on a completely separate ridge in the mountain. I would need to run even further to get home.

I ran faster as I realized I was running out of daylight. The Eastern side of the mountain was already quite dark. As I ran out of light, I had to slow my pace to avoid turning my ankle on a rock. After 20 minutes of hurrying to get back I realized I was actually descending the mountain on the Eastern side. I had made a wrong turn at some point and it wasn’t obvious to me where. The difference between the trail and the woods was now barely visible.

I began yelling “Hello” and “Help” at regular intervals, hoping that I could find another hiker. I never heard a response. I turned around and headed back, straining my eyes to look for landmarks. I knew that keeping calm was the most important thing but it was a struggle to do so. It was only getting colder and darker. The one upshot was that I knew I couldn’t be more than a couple miles from the cable car station and that my dad and sister would be wondering where I was. At worst I would need to be found by rangers with flashlights. The questions were whether it would get to that point and how long that would take.

Doubling back, I felt more lost than before because I was headed in the wrong direction. Perhaps the scariest part was finding a turn in the right direction, following it for awhile and realizing I was looping back in the wrong direction. Then I experienced the old “getting lost” trope of seeing the same landmark twice, in this case a pair of white logs crossed in an X which were visible in the moonlight.

At this point it was so dark I was barely staying on the trail. I had to reach down and feel the ground to make sure it was soft and loose to make sure I was walking where I was supposed to. Despite this strategy, I made countless mistakes and tore up my legs walking through brush. Losing the trail like this was the scariest thing I experienced. At some points I would get on my hands and knees and crawl around until I found soft dirt to walk on.

I backtracked more and found a signpost I knew was on my original route. This was where I had made the wrong turn. Following the trail out from here was harder than I expected, but from here it was only a matter of time until I found my out. I knew this and started deliriously belting out “99 bottles of beer on the wall” as I marched toward the station. This last bit was firmly on the ridge and dimly lit by Albuquerque city lights. I knew I would be ok.

I arrived at the cable car station at around 7:15pm, two hours after I expected to be getting home. I was deliriously happy until I saw how upset my dad and sister were. They were on the phone with the state police and desperately trying to get them to send out a search party. I had gotten there just in time to prevent that.

The lessons in this story can only be described as common sense. I went running too late, pushed too hard, and underestimated how dark it would get. It was pure trail-runner arrogance that got me into this situation. I hadn’t brought a flashlight or a phone, either of which would have made this situation far less hopeless.

Despite the stupidity I demonstrated, I feel like it was a valuable experience. Sometimes you need to make stupid mistakes and suffer the consequences to learn lessons. Escaping the danger I faced in the woods, in the dark, has made me more appreciative than ever at how wonderful my life is.

Optimal Descriptive NFL Rankings

Tue, 20 Nov 2012 13:47:00 -0500

Most NFL fans, like myself, obsess over who’s going to win games or which players to start in our fantasy football leagues. One of the fundamental tools we use to look at this are rankings. Rankings are the simplest possible model that can represent a total order, which you can think of as a function that allows you to compare all possible pairs in your set.

Describing the NFL season

In the NFL regular season, each of the 32 teams plays 16 games. The result is 256 binary outcomes, which if we assume are the same as coin flips (independent trials of a fair coin), then the entirety of the season contains at most 256 bits of information. One way to think about this is that if you could send yourself a string of 256 1s or 0s from January back in time to Septmeber, with a simplistic coding scheme, your past self could correctly predict all 256 games.

I began to think: we all love rankings, but how well can you describe the season with a ranking of teams? There are 32! possible rankings, so a ranking of all 32 teams contains log2(32!) bits of information, or about 118 bits. This is the smallest amount of information you could use to describe what happens in all possible pairings of teams. How accurate is it to describe 256 bits with 118 bits? How many games would you get wrong?

This is also a good way to look at how random outcomes are. If rankings can’t describe the season well, then it’s hard to have a good intuition about which teams are the best in the league.

Ranking algorithms

Given my training in statistics and love of the NFL, I have tried ranking teams before. The model I used then is a variant of the BTL model. These models essentially posit that teams can be assigned a real number that represents their strength (they exist in a latent “quality” space), and that the probability of team A beating team B is proportional to the differences in their strengths. Essentially this is a logistic regression with dummy variables for teams.

As I looked into the mathematics of ranking, I found that many smart people have thought long and hard about this problem. The abstract problem is how to take a series of pairwise comparisons and generate a total order relation.

The tricky bit is that these problems have a recursive structure: first you rank teams naively, then you re-rank adjusting for strength of schedule from your naive ranking, then you re-rank using the new strengths, and so forth. It’s no surprise then that most sophisticated ranking procedures use something like the power method to compute eigenvectors of a square matrix that encodes the pairwise comparisons.

The BTL model is probabilistic, while the eigenvector model is derived using linear algebra. They are computed differently but there is a deep relationship between the two in that they optimize very similar objective functions.

Optimal ranking with graphs

The pairwise comparison matrix (the element at row i, column j is 1 if i beat j) is also a representation for a directed graph. Using this representation, the ranking problem can then be posed slightly differently.

In the BTL model and the eigenvector model, we wanted to project each team onto the real line. If we think of our pairwise comparisons as constituting a directed graph, then the ranking is actually a path through the graph. At first I thought this path might be related to the TSP with appropriate edge-weights, but this was a bit off.

The algorithm to find this path is called a topological sort, the directed graph equivalent of a minimum spanning tree for an undirected graph. The problem is, you can’t perform a topological sort when there are cycles in the graph. Cycles are a generalization of transitivity violations: situation where A beat B, B beat C, but C beat A. We can’t sort the graph if these exist. If they don’t, the sort can take place in linear time.

The set of edges (games) which introduce cycles into the graph is called the feedback arc set (FAS). You can think of this as the set of cases the ranking model is too simplistic to explain. It seems a bit backwards, but if we can get rid of all the upsets in advance, then we can rank the teams easily and quickly. Finding the minimum FAS is an NP-hard problem (of course!), but for a 256 edge graph, it’s not a problem to compute it with igraph among others packages. The minimal FAS set is not guaranteed to be unique (for a trival example, the minimal FAS set for a 3-cycle can be any of the three edges [1]), so unfortuntately the ranking may not be unique.

After removing the FAS, the topological sort produces the optimal ranking in the following sense: no other ranking will make fewer mistakes when applied to the 256 games. This is essentially minimizing 0-1 loss of the model, which is exactly our objective function if we want rankings that make the fewest mistakes. This ranking may not be unique, but you cannot describe the season better with any ranking model.

Results

I implemented a few ranking systems in Python. You can checkout the code on github. What I find amazing is how poorly BTL and eigenvector methods compare, at least descriptively, to optimal ranking. These rankings only are only about 70-75% accurate, while optimal ranking almost always breaks 80%. This is about 10-20 games which would be considered upsets under BTL and eigenvector methods that would not be considered upsets under the optimal ranking.

Visualizing

I put together a visualization of the win-matrix using d3.js. Squares are colored if the row team beat the column team. Our “loss” comes from wins that are inconsistent with our ranking model—any square beneath the diagonal. You can also look at a bigger and interactive version of the visualization.

Name Wins BTL Optimal Rank

Loss:

A Challenge: minimum description length

I have shown that, at least for the task of description, a graph-based algorithm which minimizes the correct loss function is superior to probabilistic models which don’t actually minimize the right thing. But the ranking is still not a perfect representation of the NFL season, getting about 20% of the games wrong on average. My question to you is: describe a 100% accurate encoding of the NFL season that uses the fewest possible bits. I have thought hard about this and still don’t have a satisfying answer.