Transparency in models
Check this page anytime for our current model status.
We track all sports back to their 2010 season and update stats daily with the previous day's games.
We measure the performance of our Elo model via two metrics.
BSS is the most important metric. It allows us to quickly see how our Elo model compares to a baseline model.
There are many definitions of a "baseline" model, but the simplest is to just predict the home team as the winner since home teams have an advantage in every sport.
i.e. if the home team wins 63% of the time in the NBA and our model only accurately predicts the winner 63% of the time, then what's the point of the model? In this case, the BSS would be "0.0". Thus a positive brier score is good!
When using an average like this as the baseline, then BSS is the same as R-Squared (R²), the same R² from your high school stats class.
We also use BSS as the as the target loss-function for minimizing when training our models.
BSS is a strictly proper scoring rule.
We also provide accuracy since it is the easiest for humans to quickly make sense of.
We do not use it in the callibration of our models, however, since Accuracy is NOT a proper scoring rule.
So how do these metrics stack up against other models?
Well, the industry standard for modeling sports is arguable casino sportsbooks. Not only are their models highly sophisticated, they also have the benefit of operating in a highly efficient market. The two dynamically correct each other, resulting in a high performing prediction.
That being said, their BSS and accuracy are only marginally higher than our Elo model. e.g. 0.14 NBA BSS vs. our 0.12 with 68% accuracy vs. our 66%.
Margins are everything, if you could out perform the S&P by 1% consistently every year, you would have investers clammering to send you capital! So while our Elo model doesn't quite perform at bleeding edge billion dollar hedge fund level, it's pretty dang great for just being semi-sophisticated Python model trained on some basic boxscore data.
Check back later for specific sportsbook metric stats!