Metrics

This section describes all the metrics used within FireBench benchmarks.

1D metrics

Input: Two 1D vectors of size \(N\):

  • \(x_i\): evaluated dataset

  • \(y_i\): reference dataset

Mean

Description: Average value of a 1D vector \(x\).
Range: Same as range of \(x\).
Units: Same as input units.
Formula:

\[ \bar x = \frac{1}{N} \sum_{i=1}^N x_i \]

Bias

Description: Difference between the mean of \(x\) and the mean of \(y\).
Range: Same as range of input values.
Units: Same as input units.
Formula:

\[ B = \bar x - \bar y \]

Root Mean Square Error

Description: Square root of the mean squared difference between (x) and (y), noted RMSE.
Range: \([0, +\infty[\).
Units: Same as input units.
Formula:

\[ RMSE(x, y) = \sqrt{\frac{1}{N} \sum_{i=1}^N (x_i - y_i)^2} \]

Mean Absolute Error

Description: Mean of the absolute difference between (x) and (y), noted MAE.
Range: \([0, +\infty[\).
Units: Same as input units.
Formula:

\[ MAE(x, y) = \frac{1}{N} \sum_{i=1}^N |x_i - y_i | \]

Normalized MSE - power normalization

Description: RMSE normalized by the range of the reference dataset.
Range: \([0, +\infty)\).
Units: Dimensionless.
Formula:

\[ NMSE_p = \frac{RMSE(x, y)}{\max(y) - \min(y)} \]

Normalized MSE – range normalization

Description: Squared RMSE normalized by the product of mean values of the datasets.
Range: \([0, +\infty)\) (undefined if \(\bar x = 0\) or \(\bar y = 0\)).
Units: Dimensionless.
Formula:

\[ NMSE_r = \frac{RMSE(x, y)^2}{\bar x \, \bar y} \]

Binary Confusion Matrix

Input: Two 1D binary vectors (0 or 1) of size \(N\):

  • \(x_i\): evaluated dataset

  • \(y_i\): reference dataset

The following metrics are derived from the Binary confusion matrix generated from both dataset. The Binary confusion matrix is a 2x2 matrix containing:

Reference = 1

Reference = 0

Eval = 1

TP

FP

Eval = 0

FN

TN

Where:

  • TP: True Positive

  • FP: False Positive

  • FN: False Negative

  • TN: True Negative

Accuracy

Description: Fraction of correct predictions among all samples (see accuracy).
Range: \([0, 1]\)
Units: Dimensionless.
Formula:

\[ Accuracy = \frac{TP + TN}{TP + TN + FP + FN} \]

Precision

Description: Fraction of predicted positives that are true positives (see precision).
Range: \([0, 1]\)
Units: Dimensionless.
Formula:

\[ Precision = \frac{TP}{TP + FP}, \]

Recall

Description: Fraction of actual positives correctly identified (see recall). Recall can also be named Sensitivity or True Positive Rate.
Range: \([0, 1]\)
Units: Dimensionless.
Formula:

\[ Recall = \frac{TP}{TP + FN}, \]

Specificity

Description: Fraction of actual negatives correctly identified (see specificity). Recall can also be named True Negative Rate.
Range: \([0, 1]\)
Units: Dimensionless.
Formula:

\[ Specificity = \frac{TN}{TN + FP} \]

Negative Predictive Value

Description: Fraction of predicted negatives that are true negatives (see Negative Predictive Value).
Range: \([0, 1]\)
Units: Dimensionless.
Formula:

\[ Negative Predictive Value = \frac{TN}{TN + FN} \]

F1 Score

Description: Harmonic mean of Precision and Recall (see F1 Score).
Range: \([0, 1]\)
Units: Dimensionless.
Formula:

\[ F1 Score = \frac{2 \times Precision \times Recall}{Precision + Recall} \]