Analyzing Baseball Umpires Using Machine Learning

Abstract

This project develops a probability-based grading system for Major League Baseball umpires using Statcast pitch-level data. Traditional evaluations rely on simple accuracy of umpires’ call correctness, which fails to account for the difficulty of judging high-velocity, high-spin pitches relative to an invisible strike zone. This study employs XGBoost and Logistic Regression models to estimate the probability that a pitch is called correctly, using engineered features such as the distance from the closest edge of the strike zone and pitch velocity. Hyperparameters were optimized via Bayesian search to minimize log-loss, resulting in well-calibrated probability estimates. The predicted probabilities were incorporated into an exponential payoff function, P(𝜂, c), where 𝜂 represents the ease of a call, derived from the log-odds of a correct call, and c denotes correctness. This formulation rewards umpires more for successfully calling difficult pitches and penalizes errors more on easier pitches. This produces a nuanced measure of performance beyond binary accuracy. The XGBoost model achieved a log-loss of 0.254, AUC of 0.875, Brier Score of 0.081, and 88.1% accuracy on the testing set, demonstrating strong predictive capability. Results indicate that incorporating difficulty-adjusted grading better differentiates umpire performance, providing a robust, quantitative framework for evaluating accuracy in high-pressure, real-world conditions.