DEBUNKED: 4 SPORTS ANALYTICS MYTHS

As someone labelled an “analytics guy,” I cringe when talking heads, who are cast as members of the “analytics community,” speak in absolutes. Their job is to provoke and increase ratings, but it comes at the cost of misrepresenting good analysts. Black and white arguments sell advertising, but working analysts live in the gray reality.

The loudest voices often kindle preconceived notions which can build walls that are tough to break down. Sports’ data explosion led to an enormous – and at times – uncomfortable adjustment for many people. It’s important that leaders take an empathetic approach and recognize that change is hard.

Human Behavior

I once met with twenty leaders of a major investment firm to kick off their discussion of a new “big data” strategy. The idea was to excite these industry veterans (each averaged 20+ years of experience) about predictive analytics by talking about a fun and relatable topic – baseball. I sprinkled in baseball stories, but examples of cognitive bias and other behavioral impediments to transformative change dominated my slides.

Behavioral psych turned out to be extremely relatable, enabling us to explore the audience’s concerns. They laughed at my more absurd examples and even poked fun at their own irrational fears and bouts of imposter syndrome triggered by this big data initiative. We cleared some hurdles that day and hopefully gave them a better shot at a successful endeavor.

“The Only Thing We Have to Fear…”

Unfortunately, fear often drives decision making. It can either paralyze us or send us down a path for the wrong reasons. When the Red Sox hired Theo Epstein at age 28 (youngest GM ever hired at the time), his father advised him to “be bold.” That stuck with me as I watched Theo take calculated risks that led to remarkable success. But sometimes, leaders succumb to an intense fear of failure which begets inaction. To be bold does not mean to act recklessly but rather to not fear failure.

Fear also drives these common analytics misconceptions.

MYTH #1 – “ROBOTS WILL REPLACE US”

Utilizing computing power to crunch numbers affords people like coaches and scouts more time for their most impactful work.

Evolving Job Descriptions

Coaches and players work together in the trenches, and those relationships are crucial. Good coaches know how to best connect with each individual and how to most effectively deliver them instruction or information. Today, analysts and coaches may collaborate more to flag course corrections or craft development plans, but the art of coaching remains essential and invaluable.

Similarly, scouts continue to add tremendous value through qualitative evaluations of players. Today, a scout’s role may look different in more analytical organizations. These scouts may:

• focus more on prospects than established players at the highest level,
• get more looks on a player, giving them a better chance for a more accurate evaluation,
• not use a stopwatch to report info that’s now available via technology,
• drill deeper on the most important player attributes, or
• invest more time researching what motivates a player, how they manage adversity, or their overall character.

Getting the most impact from our resources is always our goal. If we use computers to do what they do best, that allows people to do more of what they do best. If done well, you’re positioned to make better decisions and be more efficient, and your people will feel fulfilled because they’re adding more value.

What Computers Do Best

When I started in baseball (2003), there were around 10,000 data points, and now that figure is more like 10 billion. People can no longer get by “feeling” their way through data because the sheer volume far exceeds our brain capacity.

For example, eyeballing pitch tracking data is risky. Sometimes, very smart, data-savvy people make significant mistakes because they over or under-weight a particular metric or they separate variables too cleanly into buckets. A pitcher’s effectiveness depends on a complex combination of factors, and our brains don’t allow us to properly quantify the interaction effects of variables.

The best organizations always strive for the best processes, and they understand that requires insights from both algorithms and experienced, talented people.

MYTH #2 – DECISIONS WILL BE FORMULAIC

Sitting in the top decision-making seat is hard, with no simple formulas for making tough decisions easier.

A former boss once asked me if I was frustrated that he didn’t base all player acquisition decisions on model output. The answer was an easy no, but his question opened the door for a much more nuanced conversation about knowing when to rely on the output and when to stray.

Crystal Ball

Introducing predictive analytics can cause tension merely by the presence of the word “predictive.” Misinterpretation of its meaning may create the impression that analysts think they will always be able to predict the future. And this feeling can lead some people to relish moments when outcomes differ from model predictions.

It’s important for leaders to recognize when unproductive hole-poking happens. If not nipped in the bud, they risk both sides digging in, causing further polarization. This divide will hinder their efforts to answer the most important questions.

Some may dismiss the tools altogether the first time they’re “wrong.” This is an example of the perfect solution (or nirvana) fallacy – if the model isn’t perfect, it’s useless. A seatbelt may only save your life 50% of the time in a serious car accident, but that doesn’t mean you shouldn’t use one.

Balancing Act

Elite sporting ops departments understand the strengths and limitations of their current tools and always try to improve the next versions. That’s just as true for quantitative models as it is for scouting, coaching, or recruiting methods. No system is perfect but if, for example, it increases your success rate on player acquisitions by 20%, it’s making you better.

I often introduced new models as tools that produce better starting points than their predecessors. I followed that with a thorough conversation about variables that the model accounted for and which ones had yet to be incorporated. That allowed my audience to focus more on factors that needed their attention and less on reinventing the wheel.

Analytical tools have limitations; therefore, decisions cannot be formulaic. There will always be factors in each decision that aren’t yet addressed in a model despite your analysts’ best efforts. But it would be a mistake to dismiss the output or only accept it when it confirms what you already believe.

MYTH #3 – MODELS ONLY INCLUDE PLAYER STATS

The first time we used an amateur draft model (i.e., ranking of draftees by predicted future value), some scouts presumed the model was only useful for college players. They suspected little could be done with high school players since their stats were never meaningful. As we arranged our master draft board, there was confusion when an analyst suggested moving a high school player higher up the board. The analyst highlighted glowing scouting reports, focusing on tool grades the model found to predict higher future value. This became an opportunity to discuss two things:

1. Scouting reports are a collection of qualitative data.
2. Their scouting reports carried immense weight in the model rankings.

More Than Numbers

Data scientists build models to incorporate as much relevant information as possible to better understand what drives outcomes. Besides game performance and scouting evaluations, many other interesting datasets exist, including:

• Medical risk assessments
• Personality evaluations
• Vision and reaction tests
• Biomechanical data
• Functional movement screens
• Financials
• Surveys
• Wearable tech measurements
• Ball and player tracking data

This volume can easily overwhelm decision makers, so it makes sense that they may want analytical frameworks to help figure out what’s important to the decision at hand and what’s superfluous.

Impactful leaders want proven ways that leave no stone unturned. Good analytical tools make it easier to wrap our heads around the value of various pieces of info, allowing leaders to be agile and opportunistic.

MYTH #4 – ANALYSTS ARE KNOW-IT-ALLS

Arrogance exists in every field, but the best analysts, scouts, and coaches show humility and always want to learn and get better.

Some of my fondest memories include time spent picking scouts and coaches’ brains because I had so much to learn in those areas. The late Bill Lajoie was the GM of the 1984 world champion Detroit Tigers and an outstanding scout for us at the Red Sox. I learned a lot about scouting from listening to his draft room discussions and from rare, cherished one-on-one conversations. I was 20-something with a math degree, and he was in his 70’s, an industry legend, and a top scout. We had little in common, but he still actively engaged with me, answering my questions and asking me what I had learned from data. Bill made quite an impression on me because after all his accomplishments and years, he still sought growth.

We All Want the Same Thing

Working in sports is a grind with heavy workloads and unusually long hours, which means those of us who persevere truly love the game. We can easily forget that this passion and our desire to win unite us, regardless of our differences.

By nature, most analysts are intellectually curious and recognize there’s always more to learn. Their process behind the scenes is quite academic, seeking peer reviews and critical feedback from analysts and non-analysts. These collaborations themselves tell us they don’t think they have all the answers, and I recommend giving them that benefit of the doubt from the start.

In Summary

Great leaders create opportunities for growth and healthy debate while preserving respect for our diverse backgrounds and skills. Great analysts, scouts, and coaches share those values and show humility.

The best organizations debunk myths and tune out noise from pundits and the Twittersphere. Instead, they focus internally on breaking down silo walls and navigating the complexities of human behavior. They invest energy into all areas of their operation, maximizing employee impact and streamlining decision making. And these investments will return a thriving culture.