Stats vs. Intuition: What scouts can learn from Social Science

“If a tree falls in a forest and no one is around to hear it, does it make a sound?”

Philosophers have been baffled by the perception of trees for centuries. In a 1710 treatise, the Anglo-Irish philosopher George Berkeley wrote of some hypothetical trees in a hypothetical garden: “The objects of sense exist only when they are perceived; the trees therefore are in the garden… no longer than while there is somebody by to perceive them.”

By 1884 the trees had begun to fall. The magazine Scientific American asked its readers: “If a tree were to fall on an uninhabited island, would there be any sound?” A lengthy #IslandOut social media campaign ensued, and, in the early years of the twentieth century, the international philosophy board re-located the conundrum to a forest, which, like many a knee-jerk board response, did absolutely nothing to solve the actual problem at hand.

Tree-gate is ultimately a question of ontology – a subfield within philosophy which questions the nature of ‘being.’ Ontology, thankfully, is a topic far less intellectually intimidating than it sounds, with the alternative perspectives within ontology best examined by a return to our forest of conceptual joy. To an individual with an ‘objectivist’ ontological stance, noise is entirely independent of human subjectivity, so yes, even in the absence of human presence, the tree would come clattering down. An ontological ‘constructivist’ would whole-heartedly disagree. Noise, they would argue, is merely a human perception within a world that does not exist independently of the meaning that individuals attach to various things.

Upon which side of the fence an individual falls has profound ramifications for their methodology selection in pretty much any discipline going. Ontological objectivists have a ‘positivist’ outlook and value only that which can be ‘scientifically’ verified, with quantitative methods favoured by such researchers. Constructivists, on the other hand, favour ‘interpretive’ qualitative methods, with deep, contextual narrative the order of the day.

This ontological schism can be clearly applied to the Stats vs. Intuition divide prevalent in sports analysis. Whilst adopting a scornful and dramatized approach to interpretive analyses, the 2011 Moneyball film captured this gulf fairly well. Just before being fired by Billy Beane, the Oakland A’s head scout Grady Fuson lets his opinion be known of the A’s new statistical recruitment methods: “You don’t put a team together with a computer… Baseball isn’t just numbers. It’s not science. If it was anybody could do what we do, but they can’t ‘because they don’t know what we know. They don’t have our experience and they don’t have our intuition… There are intangibles that only baseball people understand.” Such a perspective again reared its head in this piece by Neil Ashton in The Daily Mail which contrasted the statistics used by ‘laptop gurus’ with the implied deep understanding of ‘good football men.’

The use of data rather than intuition is at the heart of the rise of analytics in football, and indeed in all sporting and business environments, in pursuit of competitive edge. One area in which statistical methods certainly appears to trump intuition, as far as football is concerned, is when it comes to predicting the league. In Simon Gleave’s Premier League predictions competition, 9 of the current Top 10 of predictions are the result of statistical models, whilst the pundit and former footballer Robbie Savage, presumably a ‘good football man’ with bags of intuition, is to be found languishing in 75th place out of the 98 entries. In Mark Lawrenson’s weekly Premier League score predictions on the BBC Sport website, meanwhile, the joint best score to date to have been achieved by a competitor came courtesy of an analytical model.

In political science, a schism is also apparent within the traditional literature which compares quantitative and qualitative methods. In 1971, for example, the leading Dutch political scientist Arend Lijphart published a scheme in which research methodologies were ranked according to their merits and inherent problems. Quantitative methods ruled the roost, with Experimental and Statistical methods suffering, according to Lijphart, only from practical difficulties such as experimental control and data collection. Qualitative methods, meanwhile, were considered hamstrung by their interpretive, ‘unscientific’ status, though were recognised for their ability to extensively examine cases when only limited resources were available.

It would seem perhaps tempting then, based on the case so far, to recommend that all football clubs, and indeed political science departments, marginalise intuitive approaches and make an effort to use statistical analysis as often as possible. This, however, would be to present only half of the story as qualitative research has experienced something of a renaissance within political science in recent years.

This process was arguably kick-started in 1994 by three leading American political scientists named Gary King, Robert Keohane and Sidney Verba. They wrote a book called Designing Social Inquiry which aimed to argue that interpretive qualitative research could be of immense value to social scientists, so long as they were rigorous and aspired to ‘scientific’ positivist values such as Popperian falsification. The authors argued that quantitative methods had the inherent problem of making imperfect analyses. A country’s level of economic development, for example, is about far more than it’s GDP per capita. GDP pc, however, is frequently favoured by quantitative researchers when seeking to measure a country’s level of development. This, then, provided a window of opportunity for more holistic analyses, particularly when samples were small and in any case unsuited to data analyses – the deep, contextual analyses offered by qualitative methods were deemed newly valuable.

The concept of ‘triangulating’ methods – that is, approaching research questions from multiple methodological stances – has accordingly emerged in social science in a bid to combine the ‘science’ of the quantitative methods with the holistic merits of interpretive studies. According to Kimberly Neuendorf, a Professor of Media Arts at Cleveland State University, “triangulation… is the ideal,” with “the conclusions of the researchers… strengthened multi-fold” when research utilises mixed-methods.

‘Nested analysis’ is a prominent method by which research is triangulated within political science. Devised in 2005 by Evan Lieberman, then an Assistant Professor at Princeton University who has since become a Professor of Political Science at MIT, nested analysis attempts to synthesize the merits of statistical analyses with the deep focus offered by qualitative methods. The following flowchart, taken from the American Political Science Review (Vol. 99, No. 3) journal in which Lieberman’s article appeared, represents how nested analysis research may be carried out.

nest.png

The process boils down to an original statistical analysis followed by in-depth case study analysis of specific cases within the studied sample in order to ascertain whether the results of the statistical analysis are reliable. The cases selected for the in-depth analysis depends on whether or not the preliminary statistical analysis yielded intuitive results. If not, then surprising cases are deliberately examined in order to really gauge the model’s precision, or lack thereof.

Nested analysis might provide an interesting, and practically applicable, middle-way methodological solution to the ‘stats vs intuition’ debate within the scouting processes of football clubs. Dan Altman has been fairly prolific (e.g. here and here) in recommending that clubs use data analysis as a ‘first cut’ in the transfer market, and nested analysis offers a great tool by which this may be achieved. For example, a club could build a statistical profile based on any number of KPIs of their perfect hypothetical transfer target, and quantitative analysis would allow the filtering of hundreds or even thousands of potential players against these benchmarks. It is at this point – once a shortlist of players who match the profile has been drawn up – time to hand the reins to the ‘old-school’ intuitive scouts who may then perform deep case-study analyses of individual players in order to test the rigour of the statistical procedure.

If a model says that Messi is terrible and that Glenn Whelan, for the sake of argument, is the world’s best player, then it is certainly worth carrying out extensive ‘small-n’ in-the-flesh scouting analyses to work out why the model’s results are not intuitive and to find which parts may, or may not, need refining. If the interpretive scouts become convinced that a model is indeed satisfactorily capturing the club’s original intentions, then it is time to send the intuitive scouts back out to assess in person those players whom the model recommends.

Such a method, interestingly, is not dissimilar to that utilised by Leicester City, a team who have this season been widely praised for their recruitment strategy. Their process seems to essentially entail the use of statistical analysis to build a shortlist followed by a period spent going to watch the top candidates in person. Rob Mackenzie, Leicester’s former Head of Technical Scouting who has since moved to Tottenham to become their Head of Player Identification, takes up the story following such a process in 2014 when Leicester identified that they wanted to sign a wide player: “We did all the statistical profiling and came up with… three that were really interesting who we wanted to go and watch. We than had a period of going to watch those three and then decided on the one we wanted to buy… We were able to sign Riyad Mahrez from Le Havre… He’s an example of a statistical process that resulted in a player.” (These quotes are taken from this interesting Sky Sports article)

Mackenzie is notable for recognising the importance of philosophy and of having well-defined processes in place when it comes to player recruitment, as these tweets attest.

robm

robm2.png

Clubs would do well to make an effort to understand the significance of ontology in the formulation of individuals’ favoured research processes, and then look to how academic disciplines such as political science have been able to meld positivist and interpretive stances. If they are able to learn from academic debates in this way then not only will they be in a position to overcome the tiring ‘stats vs intuition’ narrative which has seemingly hamstrung the mainstream adoption of analytics in football, but they may also stand to gain a very real competitive edge in player recruitment.

Advertisements

2 thoughts on “Stats vs. Intuition: What scouts can learn from Social Science

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s