It’s by now very well-known that Save% (Sv%) is a flawed means of judging a goalkeeper’s shot-stopping ability. The chief reason for this was established in 2014 research by Sander Ijtsma (here) and Colin Trainor (here), which found a keeper’s Sv% from a single season to have very little predictive power as to what they would attain the next. Such criticisms have been, in part, soothed by Dan Kennett, who has argued that whilst Sv% is to be avoided as a seasonal metric, it still has use over longer periods of time.
It would seem to be all very well, then, to march assuredly ahead and apply Sv% as a metric for goalkeepers’ shot-stopping proficiency once they have faced, say, 200 Shots on Target (SOTs) (approximately 2 seasons worth). Alas, the Gods of Football and Probability have conspired against such a simple solution, with Garry Gelade noting in an OptaPro guest blog that: “Goalkeepers in weaker teams… face on-target shots that are harder to save. Failing to account for this will lead to a systematic bias; goalkeepers in weak teams will be underestimated and goalkeepers in strong teams will be overestimated.”
So, what to do? One method involves the creation of an Expected Goals (xG) model, and calculating how many Saves a goalkeeper has pulled off as compared to the ‘expected’ amount based on the chance quality of SOTs faced, which is essentially what Gelade goes on to do in his article. Paul Riley has notably done excellent publicly available work in this field, such as in this excellent Tableau. Johannes Harkins has also written an interesting OptaPro blog on the topic, with Thom Lawrence also having done some work in this field. Whilst a number of pro clubs use xG models in their assessments of goalkeepers, I’d wager that fewer do than fans of football analytics might suppose. In any case, a good assessment of shot-stopping should probably also consider the quality of the save made in addition to its difficulty – #shamelessplug.
The great stinking problem with xG models is the data required to create them. Shot locations, assist type, speed of attack, and even defensive pressure, are all factors that an xG model ideally includes and the collation of this information ranges between being impossibly time consuming to entirely reliant upon the purchase of expensive Opta data. As a cash-strapped student with an interest in analysing shot-stopping performance, creativity is required!
It is, as noted, possible to merely use Sv% in a bid to identify keepers with consistently high numbers, though this falls foul of Gelade’s critique. It is imperative, then, to correct a goalkeeper’s Sv% for the quality of shots that they are likely to have faced. With Clubelo’s Elo ratings – the Elo rating system is a method for calculating relative skill levels in competitor-versus-competitor games (Wikipedia, 2016) – a respected measure of a team’s underlying quality, I figured that this could serve as a proxy for shot qualities conceded, with the logic as follows: A team with a high Elo relative to their league’s average would concede easier shots for their goalkeeper than would a team with an Elo lower than the league’s average. I am hypothesizing, then, a link between a clubs’ Elo relative to average, and a goalkeeper’s Sv%.
To test this, I threw the Sv% (Saves/(Saves+Goals Conceded)) for 82 keepers to have faced in excess of 200 SOTs in Europe’s Top 5 leagues into a Tableau scatter plot (go and play!) with their club’s Elo relative to the league average (adjusted season by season). Whilst only 24% of the variance (R²) in Sv% was explained by a keeper’s club’s Elo, it was highly statistically significant (P-value: <0.0001). When just those who had faced over 400 SOTs were assessed, the P-value remained, whilst the R² veritably rocketed to 0.56. Over a decent sample of shots, then, it appears that Gelade’s theory on Sv% being affected by team quality is borne out by Elo.
This offers a great opportunity for the cash-strapped goalkeeper analyst to gain an insight into goalkeepers’ shot-stopping abilities. Whether a keeper is above or below the trend line offers an indication as to whether their Sv% is an over or underperformance based on the quality of the shots they are likely to have faced. The Tableau I linked to earlier, then, serves as a public tool with which to assess the shot-stopping prowess of the goalkeepers of the top European leagues (note: goalkeepers are more than just shot-stoppers!).
Who should my club scout?
Within the >400 SOTs camp – the most statistically robust cut-off – there are 4 keepers notably above the trend line: Hoffenheim’s Oliver Baumann (who, aged just 25 and with a Transfermarkt valuation of £4.13m, might well pique the interest of a few clubs), Sassuolo’s Andrea Consigli (29, £5.25m), Athletic Bilbao’s Gorka Iraizoz (35, £1.13m), and Saint Etienne’s Stephane Ruffier (29, £7.5m). Hugo Lloris (Tottenham), Simon Mignolet (Liverpool), Diego Benaglio (Wolfsburg), Sven Ulreich (FC Bayern), and Brad Guzan (Aston Villa), meanwhile, seem to save fewer shots than the keeper of clubs of their respective relative standards would be expected to.
If we scale back to consider all keepers to have faced >300 SOTs (R² = .33), we find some prospects much farther from the trend line, with Real Madrid’s Keylor Navas (29, £13.5m) absolutely dominating proceedings following two seasons with a Sv% in excess of 75% – one of which was attained in goal for La Liga’s lowly Levante. Lille’s Vincent Enyeama (33) looks an absolute steal at £1.5m, and is surely well worth a scout (#mixedmethods), particularly given football’s seeming undervaluation of keepers who are black or under 6ft – Enyeama happens to be both!
Beyond Navas and Enyeama lie three keepers who offer encouragement for the combined use of Sv% and Elo as a low-budget proxy for an xG model. The Premier League’s Adrian (West Ham, 29, £5.25m), Vito Mannone (Sunderland, 28, £1.88m), and Lukasz Fabianski (Swansea, 31, £5.25m) are substantial outliers on my Tableau, and are the Top 3 keepers on Paul Riley’s Premier League-only xG shot-stopping model.
Other impressive performers within the >300 SOTs club and below 30 years of age: Ralf Fahrmann (Schalke, 27, £6.75m), Loris Karius (Liverpool, 22, £5.25m), Marco Sportiello (Atalanta, 24, £6m), Marwin Hitz (Augsburg, 28, £3.38m).
A Tyne-Wear Goalkeeping Derby: 11-point swing
With Newcastle’s Tim Krul one of the picks of the underperformers at >300 SOTs, let’s consider the fate of Tyne-Wear clubs were their keepers to swap. Krul has saved 59.4% of the SOTs he has faced playing for a team with an Elo -4.1% relative to the Premier League average. Mannone has saved 70.3% for a team with a slightly better relative Elo at -3.9%. For ease of analysis, we can adjust these to 60% and 70% with a common denominating Elo of -4%.
For every 10 SOTs that the two face, then, Mannone typically saves one more than Krul. This season, Sunderland conceded 5.8 SOTs per game (220 in the season) and Newcastle 4.6 (175). Whilst I’m aware that neither Mannone nor Krul, for a host of reasons, didn’t play every game, let’s imagine that they did – and saved at the rate anticipated by their >300 SOTs faced.
With Mannone in goal for every game, Sunderland would be anticipated to have conceded 66 as opposed to 88 with Krul. Aston Villa, for reference, conceded 76 goals this season. For Newcastle, Mannone would have conceded 53 – the same as Chelsea – as opposed to Krul’s 70. For a team conceding between 4.5-6 SOTs per game, then, Mannone represents a shotstopping upgrade on Krul to the tune of conceding between 17 and 22 fewer goals.
Based on points gained per goal conceded, and all else being equal in every other aspect of goalkeeping, having Mannone in goal would have seen Newcastle reach 47 points to finish joint 11th with Swansea and Everton (rather than the 37 that got the Magpies relegated). By the same calculations, Sunderland with Krul would meanwhile have gained a paltry 27 points (as compared with 39 in real life). Relying on these armchair calculations, then, it seems that Mannone is worth approximately 11 points a season over Krul to a side struggling to avoid relegation.
With Costel Pantilimon (69.1%, 236 SOTs) having turned in similar numbers to Mannone throughout his time at Sunderland before he upped sticks to Watford in January, it seems entirely plausible that the Black Cats have their keepers to thank for their series of last-minute League survivals in recent seasons. For under £2m, Mannone just might well be one of the most undervalued players about.
Thanks for reading, feedback is as always massively appreciated! @Sam_Jackson94
Now go and play with my Tableau: https://public.tableau.com/profile/sam.jackson#!/vizhome/201213-presentSvELO/Sheet1