Scoring rule

In decision theory, a score function, or scoring rule, measures the accuracy of probabilistic predictions. It is applicable to tasks in which predictions must assign probabilities to a set of mutually exclusive discrete outcomes. The set of possible outcomes can be either binary or categorical in nature, and the probabilities assigned to this set of outcomes must sum to one (where each individual probability is in the range of 0 to 1). A score can be thought of as either a measure of the "calibration" of a set of probabilistic predictions, or as a "cost function" or "loss function".

If a cost is levied in proportion to a proper scoring rule, the minimal expected cost corresponds to reporting the true set of probabilities. Proper scoring rules are used in meteorology, finance, and pattern classification where a forecaster or algorithm will attempt to minimize the average score to yield refined, calibrated probabilities (i.e. accurate probabilities). Various scoring rules have also been used to assess the predictive accuracy of forecast models for association football.

Suppose ${\displaystyle Y:\Omega \rightarrow X}$ and ${\displaystyle G:\Omega \rightarrow Z}$ are two random variables defined on a sample space ${\displaystyle \Omega }$ with ${\displaystyle f_{Y}:X\rightarrow V}$ and ${\displaystyle f_{G}:Z\rightarrow V}$ as their corresponding density (mass) functions, in which ${\displaystyle Y}$ is a forecast target variable and ${\displaystyle G}$ is the random variable generated from a forecast schema. Also, assume that the ${\displaystyle Y=y}$, for ${\displaystyle y\in X}$ is the realized value. A scoring rule is a function such as ${\displaystyle S:Z\times X\rightarrow \mathbb {R} }$ (i.e., ${\displaystyle S(G,y)}$) which calculates the distance between ${\displaystyle G}$ and ${\displaystyle y}$.

...
Wikipedia