The importance of checking the tape
There’s an old joke my father, an Anglican priest, used to tell (apologies if this offends any religious sensibilities but I’m certain he, a true believer, would feel no reservations in retelling it in a sermon).
A catholic, a baptist and an Anglican all die and go to heaven. They meet St Peter at the Pearly Gates, who informs them they can enter heaven if they each answer one question: “Who do you think God is?”
The catholic answers first. “Well, mother church teaches us–” St. Peter quickly cuts him off. “No, I want to know what YOU think God is.” The catholic again replies, “Again, mother church is clear–” And St. Peter instantly refuses him entry.
The baptist goes next. “Who do you think God is?” “Sir, the bible clearly tells us–” “No no no,” St. Peter says, becoming a little more exasperated. “Who do YOU think God is?” The baptist politely answers, “If we look in the bible at chapter–” So St. Peter also denies the baptist entry.
Finally, a disappointed looking St. Peter asks the Anglican, “Who do you think God is?” And the Anglican immediately responds, “God is the creator and ruler of the universe and source of all moral authority, the supreme being.” Delighted, St. Peter reaches down for his keys, when the Anglican suddenly adds, “Now, on the other hand…”
The joke, obviously, is that while the others believe with certainty in the received, authoritative wisdom of their respective faiths, the Anglican finds solace in perpetual theological waffling.
I think analysts sometimes resemble the archetypes in this joke. One group may favour one set of metrics, believing they imply a host of set-in-stone “truths” about the game, like, for example, that finishing is not repeatable and is therefore not a “skill” worthy of deliberate practice. Another group may point to an altogether different set of metrics that support the case that “random variation”, say, in shot conversion rates, is a temporary chimera that better data and analysis will one day reveal. The third group–the one I usually fall into–casts a skeptical, non-committal eye over the whole enterprise, maintaining Wittgenstein’s dictum, “Whereof one cannot speak, thereof one must be silent.”
The problem is that these outlooks don’t go a long way in answering the most pressing question asked by football managers or technical directors: how do I make my team better?
I’ll use the example of the “finishing” or “shot quality” debate as an example.
In football analytics circles, it’s well known that raw shot conversion rates are subject to a lot of variation, not only from year to year but game to game. These conversion rates are one half of the PDO metric (along with save percentages), a metric often used to explain why a team with poor, predictive variables is overperforming.
Likewise, there are often players who score far more (or fewer) actual goals than “expected goals” in a season, the latter being the number based on average conversion rates based on proximity to goal, shot type, etc. that we would “expect” them to score. This over or underperformance tends not to be repeatable from year to year.
From the perspective, say, of the betting modeler, all that matters is that these phenomena tend to be true over the long term, so they can safely conclude that “finishing” involves a lot of noise, ie variation, and for that reason isn’t a good predictor of future performance. What matters for them is that the random variation in shot conversion or Goals vs ExpGs are accurate “descriptive” accounts of how football teams perform. That is an important place for any football analyst to start.
The problem is how analysts move from a descriptive account to a normative account, ie how football clubs should behave to maximize results. It’s at this point in which good football analytics sometimes tends to break down.
For example, let’s say you’re an analyst consulting a club with a great shot ratio (TSR), great overall ExpG ratio, but a low shot conversion rate and a mediocre goal differential. The coach comes to you and asks, “What’s wrong with my team? Why aren’t my strikers scoring more goals based on the kinds of chances they’re creating?” You tell them, “Nothing is wrong! You’re doing all the right things, it’s just bad luck, I’m sure your best strikers will right themselves in no time, stay the course.”
I should stop here and say that even with this curt, exaggerated response, you have already provided the coach a wealth of valuable knowledge prior to this exchange. The coach for example knows through you that his team is taking more shots, and generally better kinds of shots, than their opponents. That helps narrow down a bit of the problem (though the good analyst should keep in mind the “good” numbers may be help causing the “bad” ones). It also may be possible that the team is, indeed, going through a temporary slump and nothing more.
Yet there are a host of things you as an analyst can do here to go further than simply dismissing the team’s poor shot conversion rate as a temporary blip. For example, does your ExpG model for example take into consideration the number and location of opposition players, in addition to shot location and type? Maybe the X,Y player positioning data isn’t consistent enough over your sample to justify including that kind of information in the model.
Even so, you decide to take the bold step to methodically go over video of recent games. Perhaps you notice that while your team is taking more shots and from generally good locations, the forwards and attacking mids tend to be closely marked, or there are generally a lot of opposition players in the final third. This pattern matches the data which reveals your team has a fairly high rate of blocked shots compared to the rest of the league. Additionally, your team is known for relying on a “build up play” approach, and rarely plays for the rapid counter. When it does try to play on the break, the decision making tends to be poor.
You don’t have a model or linear regression to back this visual impression up (you feel a bit dirty), but you’re willing to play video analyst for a day. So you go ahead and let the coach know your hunch, and at that point they say to you, “Okay, interesting. Let me try a few things at practice.”
The idea that a coach might be able to improve on something as simple as finishing is, to date, not exactly a far out, controversial concept among managers. Many coaches, including Sir Alex Ferguson, practiced getting his players to finish quickly and accurately, often with a single touch following an early cross. Additionally, last year Henry Winter detailed Rene Muelensteen’s work with Cristiano Ronaldo while he was still at Man United. While Muelensteen worked in part on his attitude and confidence in front of goal (no doubt important factors), he also worked on that most controversial of ‘f’ words: finishing:
“Ronaldo was thinking: ‘That ball comes to me, I hit it top corner.’ I needed him to get out of that. I told him: ‘It doesn’t matter how you score, where you score, as long as the ball goes in the net.’” It was time to score ugly goals as well as beautiful ones.
“We worked on positions, which zone he was in, 1 (in front of goal), 2 (to the sides) or 3 (further out). We worked on what type of finish. One-touch. Do you need to control it? Volley it. Pass it in. Side-foot it in. Chip it in. We worked on certain goalkeepers. Did they have a certain trend? It’s details. When [post-Ronaldo] we played Schalke away in the Champions League semi [in 2011], we knew that Manuel Neuer, a good goalkeeper, was like Peter Schmeichel and would come out with a star jump [spreading himself]. So we worked on finishes low to either side, low through the legs.’’
Again, none of this discounts the countless situations in which an otherwise decent striker finds himself alone on goal and somehow skies their shot over the bar or tamely hits it into the keeper’s arms. Luck still has a major influence in the game. But there may be instances in which, for certain teams in certain instances, a low shot conversion rate may not simply be “noise.”
I think it’s important to understand that when we measure things like PDO or ExpGRs, we’re not measuring the behaviour of hydrogen molecules or gamma radiation. As Ken Arneson theorized a few weeks ago in relation to baseball analytics, “All high-level sabermetric truths derive from lower-level truths about human biomechanics and psychology”:
“And not vice versa. Things like platoon splits and home field advantage are not Constants of the Universe like the speed of light or the Planck-Einstein relation. The arise from more fundamental truths about human anatomy and psychology.
I do not consider a sabermetric truth to really be a truth unless there is a biomechanical/psychological foundation upon which that truth can rest, and from which that truth is capable of being derived.”
When imagined this way, the high variability in shot conversion makes clear sense and may or may not have anything to do with “blind luck” per se. Think of the relatively low number of shots even the greatest players manage to take over the course of a single game, or even several games. Think of the enormous variations in distance, shot type, opposition player positioning, goalkeeper positioning, quality and weight of the received pass, the possibility of deflection, the ball taking an odd bounce, etc. Think of the small amount of time a player has to make a quick decision, the relative skill of the central defenders, the quality of the pitch etc. etc.
Even the most brilliant players are faced with making the most of a generally limited number of chances under wildly varying circumstances. This is why for most teams (save for 2012-13 Man United), creating more chances than you concede is vital for long term success, more so than creating a smaller number of so-called “high quality” chances.
Yet there is no reason why teams cannot, nevertheless, focus on finishing as a skill. Indeed, there is a school of thought (mine) in which the ability to finish accurately is a reflection of other qualities which also help create more and better chances, like making space or picking the ideal pass, weighting it perfectly, knowing when to break at pace, etc. And while, as a rule, there is a lot of variation in shot conversion in football, this isn’t necessarily a Law of Physics. Each team, in each season, must be looked at individually in context.
The point here is that good analysts will always be willing to check noticeable data patterns against the living breathing thing that is football. They may not be able to tell you who God is, but they can at least go back and check the tape.