# 2021 Caldor Fire **Version**: 2026.1
**Case ID**: FB001
**FireBench IO std version**: >= 1.0
**Date of last update**: 03/15/2025 ## Contributors - Aurélien Costes, [Wildfire Interdisciplinary Research Center](https://www.wildfirecenter.org/), San Jose State University, [aurelien.costes@sjsu.edu](mailto:aurelien.costes@sjsu.edu), [ORCID](https://orcid.org/0000-0003-4543-5107) - Angel Farguell Caus, [Wildfire Interdisciplinary Research Center](https://www.wildfirecenter.org/), San Jose State University, [angel.farguellcaus@sjsu.edu](mailto:[angel.farguellcaus@sjsu.edu), [ORCID](https://orcid.org/0000-0003-2395-220X) - Adam Kochanski, [Wildfire Interdisciplinary Research Center](https://www.wildfirecenter.org/), San Jose State University, [adam.kochanski@sjsu.edu](mailto:[adam.kochanski@sjsu.edu), [ORCID](https://orcid.org/0000-0001-7820-2831) ## Description This collection of benchmarks uses the public resources about the 2021 Caldor Fire. It contains over 300 benchmarks on various datasets. It contains observation datasets for: - Building damaged (CALFIRE) - Burn severity (MTBS) - Burn severity (RAVG) - Canopy bottom height (LANDFIRE) - Canopy bulk density (LANDFIRE) - Canopy cover loss (RAVG) - Canopy height (LANDFIRE) - Infrared fire perimeters (NIROPS) - Live basal area change (RAVG) - Weather stations (Synoptic) ## Buildings damage ### Dataset The data has been collected using **CAL FIRE Damage Inspection (DINS) Data** (version of 2025/11/05). The original CSV file containing multiple fires has been processed to extract only the buildings damaged by the Caldor Fire. The dataset includes the positions (lat, lon) of buildings within the area of influence of the fire. The state of buildings is one of the following: - 'No Damage', - 'Affected (1-9%)', - 'Minor (10-25%)', - 'Major (26-50%)', - 'Destroyed (>50%)', - 'Inaccessible'. The sha256 of the source file is: *0190a5a51aafafa20270fe046a7ae17a53697b1fb218ff8096a3d8ebbc9ef983*. If the evaluated model does not explicitly represent individual buildings, it should treat all buildings within a cell as sharing the cell value for building damage (deterministic models) or the median of the building damage distribution (probabilistic models). Figure 1 shows the spatial distribution of building damage for the Caldor Fire. ![blockdiagram](../../_static/benchmarks/FB001/Caldor_bd_map.png)

Fig. 1 : Building damage map

Figure 2 shows the distribution of building damage for the Caldor Fire. The following Table shows the number of structures in each damage category. Damage category | Counts [-] ---------------------- | ----------------- No Damage | 3356 Affected (1-9%) | 56 Minor (10-25%) | 18 Major (26-50%) | 7 Destroyed (>50%) | 1005 Inaccessible | 2 Total | 4444 ![blockdiagram](../../_static/benchmarks/FB001/Caldor_bd_distribution.png)

Fig. 2 : Distribution of buildings damage

### Processing of dataset *Performed at obs dataset level* The data from the original CSV file were standardized without modification. The column names from the original csv file were corrected from "* Damage" to "Damage" and "* Incident Name" to "Incident Name" to simplify processing. #### Binary classes of building damage *Performed at benchmark run level* To perform some calculations, the damaged building classes can be aggregated to form binary classes. The `Inaccessible` is ignored. The following aggregation method is used: - `unburnt` binary class contains `No Damage`, `Affected (1-9%)`, and `Minor (10-25%)`, - `burnt` binary class contains `Major (26-50%)`, and `Destroyed (>50%)`. ### Benchmarks See Key Performance Indicator (KPI) and normalization definitions [here](../../metrics/index.md). #### Binary Structure Loss Accuracy **Short IDs**: BD01
**KPI**: Binary Structure Loss Accuracy
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: Binary Structure Loss Accuracy
This benchmark is performed on the binary classes for damaged buildings. #### Binary Structure Loss Precision **Short IDs**: BD02
**KPI**: Binary Structure Loss Precision
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: Binary Structure Loss Precision
This benchmark is performed on the binary classes for damaged buildings. #### Binary Structure Loss Recall **Short IDs**: BD03
**KPI**: Binary Structure Loss Recall
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: Binary Structure Loss Recall
This benchmark is performed on the binary classes for damaged buildings. #### Binary Structure Loss Specificity **Short IDs**: BD04
**KPI**: Binary Structure Loss Specificity
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: Binary Structure Loss Specificity
This benchmark is performed on the binary classes for damaged buildings. #### Binary Structure Loss Negative Predictive Value **Short IDs**: BD05
**KPI**: Binary Structure Loss Negative Predictive Value
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: Binary Structure Loss Negative Predictive Value
This benchmark is performed on the binary classes for damaged buildings. #### Binary Structure Loss F1 Score **Short IDs**: BD06
**KPI**: Binary Structure Loss F1 Score
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: Binary Structure Loss F1 Score
This benchmark is performed on the binary classes for damaged buildings. ## Burn severity from MTBS ### Dataset The data has been collected using [Monitoring Trends in Burning Severity](https://mtbs.gov/) (MTBS). The original zip file contains burn severity, pre/post burn images, and the final fire perimeter. The source of the burn severity used in FireBench is the file `ca3858612053820210815_20210805_20220723_dnbr6.tif`. The source of the final fire perimeter is the kmz file `ca3858612053820210815_20210805_20220723.kmz`. The burn severity categories, described with the corresponding index used in the dataset, are the following: - 'no data': 0 - 'unburnt to low': 1 - 'low': 2 - 'moderate': 3 - 'high': 4 - 'increased greenness': 5 The hashes of the original source files are: - zip file: 171b9604c0654d8612eaabcfcad93d2374762661ab34b4d62718630a13469841 - tif dnbr6: 33db74d3c5798c41ff3a4fc5ee57da9105fdc7a75d7f8af0d053d2f82cfdc0b6 - final perimeter kmz: 4ed7a0ee585f8118b65a29375a3d5ee8a69e85a95ee155205ba5d781289c6e2b Figure 3 shows the MTBS map from the original source. ![blockdiagram](../../_static/benchmarks/FB001/mtbs_map.jpg)

Fig. 3 : Map of burn severity from MTBS. Source: MTBS (`ca3858612053820210815_map.pdf`)

### Processing of dataset *Performed at obs dataset level* The burn severity array is extracted from the original file without any modification. The latitude and longitude array are reconstructed using projection parameters (see `firebench.standardize.mtbs.standardize_mtbs_from_geotiff`). The final perimeter has been processed using QGIS. The original data (kmz file) has been imported and cleaned. Extra perimeters have been removed to conserve only the final fire perimeter. No modification to the polygons has been performed. Then, the multipolygons were exported to kml format and integrated into the dataset HDF5 file. #### Binary classes for high severity *Performed at benchmark run level* To perform the high-severity benchmarks using a binary confusion matrix, we construct a binary field based on the high-severity index. All points will have a burn severity of 4 ('high') and will be assigned the value 1. The other points are assigned a value of 0. This processing is done when the benchmark is performed. ### Benchmarks See Key Performance Indicator (KPI) and normalization definitions [here](../../metrics/index.md). #### Binary High Severity Accuracy **Short IDs**: SV01
**KPI**: Binary High Severity Accuracy
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: Binary High Severity Accuracy
This benchmark is performed on the binary classes for high severity points (Binary High severity processed variable) #### Binary High Severity Precision **Short IDs**: SV02
**KPI**: Binary High Severity Precision
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: Binary High Severity Precision
This benchmark is performed on the binary classes for high severity points (Binary High severity processed variable) #### Binary High Severity Recall **Short IDs**: SV03
**KPI**: Binary High Severity Recall
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: Binary High Severity Recall
This benchmark is performed on the binary classes for high severity points (Binary High severity processed variable) #### Binary High Severity Specificity **Short IDs**: SV04
**KPI**: Binary High Severity Specificity
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: Binary High Severity Specificity
This benchmark is performed on the binary classes for high severity points (Binary High severity processed variable) #### Binary High Severity Negative Predictive Value **Short IDs**: SV05
**KPI**: Binary High Severity Negative Predictive Value
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: Binary High Severity Negative Predictive Value
This benchmark is performed on the binary classes for high severity points (Binary High severity processed variable) #### Binary High Severity F1 Score **Short IDs**: SV06
**KPI**: Binary High Severity F1 Score
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: Binary High Severity F1 Score
This benchmark is performed on the binary classes for high severity points (Binary High severity processed variable) ## Canopy cover loss ### Dataset The data has been collected using [Rapid Assessment of Vegetation Condition after Wildfire](https://burnseverity.cr.usgs.gov/ravg/) (RAVG). The source of the canopy cover loss used in FireBench is the dataset over CONUS for 2021, `ravg_2021_cc5.tif`. The region around the Caldor Fire has been processed and standardized using the following bounding box: - south west: (38.4, -120.8) - north east: (39.0, -119.7) The canopy cover loss categories, described with the corresponding index used in the dataset, are the following: - 'Unmappable': 0 - '0%': 1 - '>0-<25%': 2 - '25-<50%': 3 - '50-<75%': 4 - '75-100%': 5 - 'Outide burn area': 9 In addition, a bounding box has been used to remove the data from another fire (forced to `0`): - south west: (38.6, -119.9) - north east: (38.805, -119.7) Figure 4 shows the processed RAVG dataset available in FireBench. ![blockdiagram](../../_static/benchmarks/FB001/RAVG_CC_final.png)

Fig. 4 : Map of standardized canopy cover loss from RAVG for Caldor Fire.

### Processing of dataset *Performed at obs dataset level* A bounding box has been used to remove the data from another fire (forced to `0`): - south west: (38.6, -119.9) - north east: (38.805, -119.7) #### Masking using LANDFIRE dataset *Performed at benchmark run level* To perform an evaluation of high canopy cover loss, a mask is defined using three LANDFIRE datasets: - Canopy bulk density - Canopy height - Canopy bottom height The variable `masked high binary canopy cover loss` used in various benchmarks is computed only where all LANDFIRE canopy variables (interpolated using the nearest method on the RAVG grid) are strictly greater than 0 (presence of canopy fuel) and is defined as a binary variable: - `1` if RAVG canopy cover loss value is `5`, - `0` if RAVG canopy cover loss value is between `1` and `4`, - `nan` otherwise. Figure 5 shows the processed `masked high binary canopy cover loss` dataset used for related benchmarks. ![blockdiagram](../../_static/benchmarks/FB001/RAVG_CC_masked.png)

Fig. 5 : Map of standardized canopy cover loss from RAVG for Caldor Fire.

### Benchmarks See Key Performance Indicator (KPI) and normalization definitions [here](../../metrics/index.md). #### Masked High Binary Canopy Cover Loss Accuracy **Short IDs**: CC01
**KPI**: Binary High Canopy Cover Loss Accuracy
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: Binary High Canopy Cover Loss Accuracy
This benchmark is performed on the binary classes `masked high binary canopy cover loss`. #### Masked High Binary Canopy Cover Precision **Short IDs**: CC02
**KPI**: Binary High Canopy Cover Loss Precision
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: Binary High Canopy Cover Loss Precision
This benchmark is performed on the binary classes `masked high binary canopy cover loss`. #### Masked High Binary Canopy Cover Recall **Short IDs**: CC03
**KPI**: Binary High Canopy Cover Loss Recall
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: Binary High Canopy Cover Loss Recall
This benchmark is performed on the binary classes `masked high binary canopy cover loss`. #### Masked High Binary Canopy Cover Specificity **Short IDs**: CC04
**KPI**: Binary High Canopy Cover Loss Specificity
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: Binary High Canopy Cover Loss Specificity
This benchmark is performed on the binary classes `masked high binary canopy cover loss`. #### Masked High Binary Canopy Cover Negative Predictive Value **Short IDs**: CC05
**KPI**: Binary High Canopy Cover Loss Negative Predictive Value
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: Binary High Canopy Cover Loss Negative Predictive Value
This benchmark is performed on the binary classes `masked high binary canopy cover loss`. #### Masked High Binary Canopy Cover F1 Score **Short IDs**: CC06
**KPI**: Binary High Canopy Cover Loss F1 Score
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: Binary High Canopy Cover Loss F1 Score
This benchmark is performed on the binary classes `masked high binary canopy cover loss`. ## Infrared fire perimeters ### Dataset The infrared fire perimeters have been gathered from [NIROPS](https://ftp.wildfire.gov/public/incident_specific_data/calif_n/2021_FEDERAL_Incidents/CA-ENF-024030_Caldor/IR/NIROPS/) dataset. Every orginal file has been manually processed to extract only the perimeter. The time stamp of the perimeter has been defined from the imaging report (e.g. [Report for 2021/08/17](https://ftp.wildfire.gov/public/incident_specific_data/calif_n/2021_FEDERAL_Incidents/CA-ENF-024030_Caldor/IR/NIROPS/20210818/20210818_Caldor_IR_Topo_11x17.pdf)) using the `Imagery Date` and `Imagery Time`. The burn area obtained using the KML file and python tools has been verified against the `Interpreted Acreage` when specified in the reports. Each fire perimeter (see Fig. 6) is stored as a group within the HDF5 data file with attributes containing the path of the KML file that contains the fire perimeter dataset. The perimeters have been processed from August 17th (first IR perimeter available) to September 10th, when the burn area is 99% if the final burn area, as shown in Figure 7 (source: [CALFIRE](https://www.fire.ca.gov/incidents/2021/8/14/caldor-fire/)). The final dataset contains 21 perimeters. The following study periods (see Fig. 7) are defined in the following Table: Name | Start time | End time | Duration | Burn area [acre] -----|-------------------|--------------------|---------------|----------------- W1 | Aug 17 20h20 PDT | Sep 10 23h34 PDT | 24d 3h 14min | 166,256 W2 | Aug 19 20h45 PDT | Aug 21 21h15 PDT | 2d 0h 30min | 24,941 W3 | Aug 26 02h30 PDT | Aug 28 20h30 PDT | 2d 18h 0min | 19,992 W4 | Aug 28 20h30 PDT | Sep 3 00h40 PDT | 5d 4h 10min | 56,272 Figure 6 shows the processed fire perimeter as a colored solid contour. The color of the contour indicates the timestamp of the perimeter. ![blockdiagram](../../_static/benchmarks/FB001/Caldor_perimeters.png)

Fig. 6 : Infrared fire perimeters from August 17th to September 10th.

![blockdiagram](../../_static/benchmarks/FB001/Caldor_burnt_area.png)

Fig. 7 : Burn area derived from IR perimeters from August 17th to September 10th. The red dashed line shows the final burn area from CALFIRE. The orange dashed line shows the final burn area from the MTBS final perimeter.

### Benchmarks See Key Performance Indicator (KPI) and normalization definitions [here](../../metrics/index.md). #### Average Jaccard Index over study period **Short IDs**: See Table
**KPI**: Average Jaccard Index
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: See Table
The first perimeter at the start of the period can serve as an initial condition for the fire perimeter. The first perimeter is not used to compute any metric. The area preserving project used is EPSG:5070. The following Table gives the correspondence between the benchmark ID and the study period: ID | Study period | Name in Score Card -----|--------------|------------------- FP01 | W1 | Average Jaccard Index W1 FP02 | W2 | Average Jaccard Index W2 FP03 | W3 | Average Jaccard Index W3 FP04 | W4 | Average Jaccard Index W4 #### Minimum Jaccard Index over study period **Short IDs**: See Table
**KPI**: Minimum Jaccard Index
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: See Table
The first perimeter at the start of the period can serve as an initial condition for the fire perimeter. The first perimeter is not used to compute any metric. The area preserving project used is EPSG:5070. The following Table gives the correspondence between the benchmark ID and the study period: ID | Study period | Name in Score Card -----|--------------|------------------- FP05 | W1 | Minimum Jaccard Index W1 FP06 | W2 | Minimum Jaccard Index W2 FP07 | W3 | Minimum Jaccard Index W3 FP08 | W4 | Minimum Jaccard Index W4 #### Maximum Jaccard Index over study period **Short IDs**: See Table
**KPI**: Maximum Jaccard Index
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: See Table
The first perimeter at the start of the period can serve as an initial condition for the fire perimeter. The first perimeter is not used to compute any metric. The area preserving project used is EPSG:5070. The following Table gives the correspondence between the benchmark ID and the study period: ID | Study period | Name in Score Card -----|--------------|------------------- FP09 | W1 | Minimum Jaccard Index W1 FP10 | W2 | Minimum Jaccard Index W2 FP11 | W3 | Minimum Jaccard Index W3 FP12 | W4 | Minimum Jaccard Index W4 #### Average Dice-Sorensen Index over study period **Short IDs**: See Table
**KPI**: Average Dice-Sorensen Index
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: See Table
The first perimeter at the start of the period can serve as an initial condition for the fire perimeter. The first perimeter is not used to compute any metric. The area preserving project used is EPSG:5070. The following Table gives the correspondence between the benchmark ID and the study period: ID | Study period | Name in Score Card -----|--------------|------------------- FP13 | W1 | Average Dice-Sorensen Index W1 FP14 | W2 | Average Dice-Sorensen Index W2 FP15 | W3 | Average Dice-Sorensen Index W3 FP16 | W4 | Average Dice-Sorensen Index W4 #### Minimum Dice-Sorensen Index over study period **Short IDs**: See Table
**KPI**: Minimum Dice-Sorensen Index
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: See Table
The first perimeter at the start of the period can serve as an initial condition for the fire perimeter. The first perimeter is not used to compute any metric. The area preserving project used is EPSG:5070. The following Table gives the correspondence between the benchmark ID and the study period: ID | Study period | Name in Score Card -----|--------------|------------------- FP17 | W1 | Minimum Dice-Sorensen Index W1 FP18 | W2 | Minimum Dice-Sorensen Index W2 FP19 | W3 | Minimum Dice-Sorensen Index W3 FP20 | W4 | Minimum Dice-Sorensen Index W4 #### Maximum Dice-Sorensen Index over study period **Short IDs**: See Table
**KPI**: Maximum Dice-Sorensen Index
**Normalization**: Linear Bounded Normalization with $a=0$, $b=1$
**Name in Score Card**: See Table
The first perimeter at the start of the period can serve as an initial condition for the fire perimeter. The first perimeter is not used to compute any metric. The area preserving project used is EPSG:5070. The following Table gives the correspondence between the benchmark ID and the study period: ID | Study period | Name in Score Card -----|--------------|------------------- FP21 | W1 | Minimum Dice-Sorensen Index W1 FP22 | W2 | Minimum Dice-Sorensen Index W2 FP23 | W3 | Minimum Dice-Sorensen Index W3 FP24 | W4 | Minimum Dice-Sorensen Index W4 #### Final Burn Area Bias **Short IDs**: See Table
**KPI**: Burn Area Bias
**Normalization**: Symmetric Exponential Open Normalization ($m$ value in Table)
**Name in Score Card**: See Table
The first perimeter, at the start of the period, can be used as initial condition for the fire perimeter. The bias is calculated on the last perimeter of the study period as the difference between the model and the observed burn area. A bias of $m$ acres, representing $B_{50}$% of burn area during the study period, will lead to a score of 50.00. The value of $m$ represents the benchmark difficulty (smaller $m$ means greater difficulty) and must be chosen by the community. The following Table gives the correspondence between the benchmark ID and the study period: ID | Study period | Name in Score Card | $m$ | $B_{50}$ -----|--------------|--------------------|--------|--------- FP25 | W1 | Burn Area Bias W1 | 80,000 | 48% FP26 | W2 | Burn Area Bias W2 | 5,000 | 20% FP27 | W3 | Burn Area Bias W3 | 5,000 | 25% FP28 | W4 | Burn Area Bias W4 | 17,000 | 30% #### Burn Area RMSE **Short IDs**: See Table
**KPI**: Burn Area RMSE
**Normalization**: Symmetric Exponential Open Normalization ($m$ value in Table)
**Name in Score Card**: See Table
The first perimeter, at the start of the period, can be used as initial condition for the fire perimeter. A bias of $m$ acres, representing $B_{50}$% of burn area during the study period, will lead to a score of 50.00. The value of $m$ represents the benchmark difficulty (smaller $m$ means greater difficulty) and must be chosen by the community. The following Table gives the correspondence between the benchmark ID and the study period: ID | Study period | Name in Score Card | $m$ | $B_{50}$ -----|--------------|--------------------|--------|--------- FP29 | W1 | Burn Area RMSE W1 | 80,000 | 48% FP30 | W2 | Burn Area RMSE W2 | 5,000 | 20% FP31 | W3 | Burn Area RMSE W3 | 5,000 | 25% FP32 | W4 | Burn Area RMSE W4 | 17,000 | 30% ## Weather stations ### Dataset Weather stations datasets have been gathered from [Synoptics](https://synopticdata.com). All the stations available in the following bounding box have been processed: - south west: (38.4, -120.8) - north east: (39.0, -119.7) The following variables have been processed (following FireBench namespace): - air_temperature - relative_humidity - solar_radiation - fuel_moisture_content_10h - wind_direction - wind_gust - wind_speed ```{note} If you want to process more variables or require new benchmarks for existing variables, please reach out to the FireBench team to integrate these changes into a future version of the benchmarks. ``` Some stations don't have data for the period W1 and have been excluded from the dataset. The list of excluded stations for missing data in the study period is: 403_PG, 412_PG, 413_PG, F9934. Also, some stations did not meet the data quality criterion and have been excluded from the dataset. The list of excluded stations for data quality reasons is: AV833, BLCC1, C9148, COOPDAGN2, COOPMINN2, FOIC1, FPDC1, G0658, GEOC1, LNLC1, PFHC1, SBKC1, SLPC1, STAN2, UTRC1, WDFC1, XOHC1. Sensor height data has been extracted following the sensor height priority rules defined here. The current version of knowledge about sensor heights for the case weather stations are: - 10 stations with a complete dataset (sensor height found in the source file) - 98 stations with missing metadata - 21 stations skipped - 81 datasets with sensor height metadata - 0 datasets from trusted stations from the FireBench database - 0 datasets from trusted history from the FireBench database - 5 datasets from the FireBench provider default database - 394 datasets using FireBench default metadata Therefore, 81 datasets are considered trusted and will be used in the benchmarks `trusted source only` (TSO). All 399 datasets are used in benchmarks "all sources". ```{note} If you have information about sensor height and want to help increase the number of trusted datasets, please get in touch with the FireBench Team. ``` Weather stations are stored in the HDF5 file using their STID. ### Benchmarks See Key Performance Indicator (KPI) and normalization definitions [here](../../metrics/index.md). #### Air temperature **Short IDs**: See Table
**KPI**: Air temperature MAE/RMSE/Bias
**Normalization**: Symmetric Exponential Open Normalization ($m$ value in Table)
**Name in Score Card**: See Table
Each metric (MAE, RMSE, Bias) is calculated for each station for both model and observational dataset for a specified period. Then we apply summary statistics (*e.g.*, min, mean, Q3) across all available weather stations before applying the normalization. Implementation of metrics are `firebench.metrics.stats.mae`, `firebench.metrics.stats.rmse`, `firebench.metrics.stats.bias`. Datasets are converted into `degC` for comparison. The normalization parameter $m$ sets which KPI value gives a Score of 50. It represents the difficulty of the benchmark. The following Table gives the correspondence between the benchmark ID and the study period: ID | Study period | Summary stats func | Name in Score Card | $m$ | trusted source only ------|--------------|--------------------|-------------------------|---------|-------------------- WX001 | W1 | MAE | Air temp MAE min W1 TSO | 5.0 degC | False WX002 | W1 | MAE | Air temp MAE mean W1 TSO | 5.0 degC | False WX003 | W1 | MAE | Air temp MAE max W1 TSO | 5.0 degC | False WX004 | W1 | MAE | Air temp MAE min W1 | 5.0 degC | True WX005 | W1 | MAE | Air temp MAE mean W1 | 5.0 degC | True WX006 | W1 | MAE | Air temp MAE max W1 | 5.0 degC | True WX007 | W1 | RMSE | Air temp RMSE min W1 TSO | 5.0 degC | False WX008 | W1 | RMSE | Air temp RMSE mean W1 TSO | 5.0 degC | False WX009 | W1 | RMSE | Air temp RMSE max W1 TSO | 5.0 degC | False WX010 | W1 | RMSE | Air temp RMSE min W1 | 5.0 degC | True WX011 | W1 | RMSE | Air temp RMSE mean W1 | 5.0 degC | True WX012 | W1 | RMSE | Air temp RMSE max W1 | 5.0 degC | True WX013 | W1 | Bias | Air temp Bias min W1 TSO | 5.0 degC | False WX014 | W1 | Bias | Air temp Bias mean W1 TSO | 5.0 degC | False WX015 | W1 | Bias | Air temp Bias max W1 TSO | 5.0 degC | False WX016 | W1 | Bias | Air temp Bias min W1 | 5.0 degC | True WX017 | W1 | Bias | Air temp Bias mean W1 | 5.0 degC | True WX018 | W1 | Bias | Air temp Bias max W1 | 5.0 degC | True WX019 | W2 | MAE | Air temp MAE min W2 TSO | 5.0 degC | False WX020 | W2 | MAE | Air temp MAE mean W2 TSO | 5.0 degC | False WX021 | W2 | MAE | Air temp MAE max W2 TSO | 5.0 degC | False WX022 | W2 | MAE | Air temp MAE min W2 | 5.0 degC | True WX023 | W2 | MAE | Air temp MAE mean W2 | 5.0 degC | True WX024 | W2 | MAE | Air temp MAE max W2 | 5.0 degC | True WX025 | W2 | RMSE | Air temp RMSE min W2 TSO | 5.0 degC | False WX026 | W2 | RMSE | Air temp RMSE mean W2 TSO | 5.0 degC | False WX027 | W2 | RMSE | Air temp RMSE max W2 TSO | 5.0 degC | False WX028 | W2 | RMSE | Air temp RMSE min W2 | 5.0 degC | True WX029 | W2 | RMSE | Air temp RMSE mean W2 | 5.0 degC | True WX030 | W2 | RMSE | Air temp RMSE max W2 | 5.0 degC | True WX031 | W2 | Bias | Air temp Bias min W2 TSO | 5.0 degC | False WX032 | W2 | Bias | Air temp Bias mean W2 TSO | 5.0 degC | False WX033 | W2 | Bias | Air temp Bias max W2 TSO | 5.0 degC | False WX034 | W2 | Bias | Air temp Bias min W2 | 5.0 degC | True WX035 | W2 | Bias | Air temp Bias mean W2 | 5.0 degC | True WX036 | W2 | Bias | Air temp Bias max W2 | 5.0 degC | True WX037 | W3 | MAE | Air temp MAE min W3 TSO | 5.0 degC | False WX038 | W3 | MAE | Air temp MAE mean W3 TSO | 5.0 degC | False WX039 | W3 | MAE | Air temp MAE max W3 TSO | 5.0 degC | False WX040 | W3 | MAE | Air temp MAE min W3 | 5.0 degC | True WX041 | W3 | MAE | Air temp MAE mean W3 | 5.0 degC | True WX042 | W3 | MAE | Air temp MAE max W3 | 5.0 degC | True WX043 | W3 | RMSE | Air temp RMSE min W3 TSO | 5.0 degC | False WX044 | W3 | RMSE | Air temp RMSE mean W3 TSO | 5.0 degC | False WX045 | W3 | RMSE | Air temp RMSE max W3 TSO | 5.0 degC | False WX046 | W3 | RMSE | Air temp RMSE min W3 | 5.0 degC | True WX047 | W3 | RMSE | Air temp RMSE mean W3 | 5.0 degC | True WX048 | W3 | RMSE | Air temp RMSE max W3 | 5.0 degC | True WX049 | W3 | Bias | Air temp Bias min W3 TSO | 5.0 degC | False WX050 | W3 | Bias | Air temp Bias mean W3 TSO | 5.0 degC | False WX051 | W3 | Bias | Air temp Bias max W3 TSO | 5.0 degC | False WX052 | W3 | Bias | Air temp Bias min W3 | 5.0 degC | True WX053 | W3 | Bias | Air temp Bias mean W3 | 5.0 degC | True WX054 | W3 | Bias | Air temp Bias max W3 | 5.0 degC | True WX055 | W4 | MAE | Air temp MAE min W4 TSO | 5.0 degC | False WX056 | W4 | MAE | Air temp MAE mean W4 TSO | 5.0 degC | False WX057 | W4 | MAE | Air temp MAE max W4 TSO | 5.0 degC | False WX058 | W4 | MAE | Air temp MAE min W4 | 5.0 degC | True WX059 | W4 | MAE | Air temp MAE mean W4 | 5.0 degC | True WX060 | W4 | MAE | Air temp MAE max W4 | 5.0 degC | True WX061 | W4 | RMSE | Air temp RMSE min W4 TSO | 5.0 degC | False WX062 | W4 | RMSE | Air temp RMSE mean W4 TSO | 5.0 degC | False WX063 | W4 | RMSE | Air temp RMSE max W4 TSO | 5.0 degC | False WX064 | W4 | RMSE | Air temp RMSE min W4 | 5.0 degC | True WX065 | W4 | RMSE | Air temp RMSE mean W4 | 5.0 degC | True WX066 | W4 | RMSE | Air temp RMSE max W4 | 5.0 degC | True WX067 | W4 | Bias | Air temp Bias min W4 TSO | 5.0 degC | False WX068 | W4 | Bias | Air temp Bias mean W4 TSO | 5.0 degC | False WX069 | W4 | Bias | Air temp Bias max W4 TSO | 5.0 degC | False WX070 | W4 | Bias | Air temp Bias min W4 | 5.0 degC | True WX071 | W4 | Bias | Air temp Bias mean W4 | 5.0 degC | True WX072 | W4 | Bias | Air temp Bias max W4 | 5.0 degC | True #### Relative Humidity **Short IDs**: See Table
**KPI**: Relative humidity MAE/RMSE/Bias
**Normalization**: Symmetric Exponential Open Normalization ($m$ value in Table)
**Name in Score Card**: See Table
Each metric (MAE, RMSE, Bias) is calculated for each station for both model and observational dataset for a specified period. Then we apply summary statistics (*e.g.*, min, mean, Q3) across all available weather stations before applying the normalization. Implementation of metrics are `firebench.metrics.stats.mae`, `firebench.metrics.stats.rmse`, `firebench.metrics.stats.bias`. Datasets are converted into `percent` for comparison. The normalization parameter $m$ sets which KPI value gives a Score of 50. It represents the difficulty of the benchmark. The following Table gives the correspondence between the benchmark ID and the study period: ID | Study period | Summary stats func | Name in Score Card | $m$ | trusted source only ------|--------------|--------------------|-------------------------|---------|-------------------- WX073 | W1 | MAE | RH MAE min W1 TSO | 15.0 percent | False WX074 | W1 | MAE | RH MAE mean W1 TSO | 15.0 percent | False WX075 | W1 | MAE | RH MAE max W1 TSO | 15.0 percent | False WX076 | W1 | MAE | RH MAE min W1 | 15.0 percent | True WX077 | W1 | MAE | RH MAE mean W1 | 15.0 percent | True WX078 | W1 | MAE | RH MAE max W1 | 15.0 percent | True WX079 | W1 | RMSE | RH RMSE min W1 TSO | 15.0 percent | False WX080 | W1 | RMSE | RH RMSE mean W1 TSO | 15.0 percent | False WX081 | W1 | RMSE | RH RMSE max W1 TSO | 15.0 percent | False WX082 | W1 | RMSE | RH RMSE min W1 | 15.0 percent | True WX083 | W1 | RMSE | RH RMSE mean W1 | 15.0 percent | True WX084 | W1 | RMSE | RH RMSE max W1 | 15.0 percent | True WX085 | W1 | Bias | RH Bias min W1 TSO | 15.0 percent | False WX086 | W1 | Bias | RH Bias mean W1 TSO | 15.0 percent | False WX087 | W1 | Bias | RH Bias max W1 TSO | 15.0 percent | False WX088 | W1 | Bias | RH Bias min W1 | 15.0 percent | True WX089 | W1 | Bias | RH Bias mean W1 | 15.0 percent | True WX090 | W1 | Bias | RH Bias max W1 | 15.0 percent | True WX091 | W2 | MAE | RH MAE min W2 TSO | 15.0 percent | False WX092 | W2 | MAE | RH MAE mean W2 TSO | 15.0 percent | False WX093 | W2 | MAE | RH MAE max W2 TSO | 15.0 percent | False WX094 | W2 | MAE | RH MAE min W2 | 15.0 percent | True WX095 | W2 | MAE | RH MAE mean W2 | 15.0 percent | True WX096 | W2 | MAE | RH MAE max W2 | 15.0 percent | True WX097 | W2 | RMSE | RH RMSE min W2 TSO | 15.0 percent | False WX098 | W2 | RMSE | RH RMSE mean W2 TSO | 15.0 percent | False WX099 | W2 | RMSE | RH RMSE max W2 TSO | 15.0 percent | False WX100 | W2 | RMSE | RH RMSE min W2 | 15.0 percent | True WX101 | W2 | RMSE | RH RMSE mean W2 | 15.0 percent | True WX102 | W2 | RMSE | RH RMSE max W2 | 15.0 percent | True WX103 | W2 | Bias | RH Bias min W2 TSO | 15.0 percent | False WX104 | W2 | Bias | RH Bias mean W2 TSO | 15.0 percent | False WX105 | W2 | Bias | RH Bias max W2 TSO | 15.0 percent | False WX106 | W2 | Bias | RH Bias min W2 | 15.0 percent | True WX107 | W2 | Bias | RH Bias mean W2 | 15.0 percent | True WX108 | W2 | Bias | RH Bias max W2 | 15.0 percent | True WX109 | W3 | MAE | RH MAE min W3 TSO | 15.0 percent | False WX110 | W3 | MAE | RH MAE mean W3 TSO | 15.0 percent | False WX111 | W3 | MAE | RH MAE max W3 TSO | 15.0 percent | False WX112 | W3 | MAE | RH MAE min W3 | 15.0 percent | True WX113 | W3 | MAE | RH MAE mean W3 | 15.0 percent | True WX114 | W3 | MAE | RH MAE max W3 | 15.0 percent | True WX115 | W3 | RMSE | RH RMSE min W3 TSO | 15.0 percent | False WX116 | W3 | RMSE | RH RMSE mean W3 TSO | 15.0 percent | False WX117 | W3 | RMSE | RH RMSE max W3 TSO | 15.0 percent | False WX118 | W3 | RMSE | RH RMSE min W3 | 15.0 percent | True WX119 | W3 | RMSE | RH RMSE mean W3 | 15.0 percent | True WX120 | W3 | RMSE | RH RMSE max W3 | 15.0 percent | True WX121 | W3 | Bias | RH Bias min W3 TSO | 15.0 percent | False WX122 | W3 | Bias | RH Bias mean W3 TSO | 15.0 percent | False WX123 | W3 | Bias | RH Bias max W3 TSO | 15.0 percent | False WX124 | W3 | Bias | RH Bias min W3 | 15.0 percent | True WX125 | W3 | Bias | RH Bias mean W3 | 15.0 percent | True WX126 | W3 | Bias | RH Bias max W3 | 15.0 percent | True WX127 | W4 | MAE | RH MAE min W4 TSO | 15.0 percent | False WX128 | W4 | MAE | RH MAE mean W4 TSO | 15.0 percent | False WX129 | W4 | MAE | RH MAE max W4 TSO | 15.0 percent | False WX130 | W4 | MAE | RH MAE min W4 | 15.0 percent | True WX131 | W4 | MAE | RH MAE mean W4 | 15.0 percent | True WX132 | W4 | MAE | RH MAE max W4 | 15.0 percent | True WX133 | W4 | RMSE | RH RMSE min W4 TSO | 15.0 percent | False WX134 | W4 | RMSE | RH RMSE mean W4 TSO | 15.0 percent | False WX135 | W4 | RMSE | RH RMSE max W4 TSO | 15.0 percent | False WX136 | W4 | RMSE | RH RMSE min W4 | 15.0 percent | True WX137 | W4 | RMSE | RH RMSE mean W4 | 15.0 percent | True WX138 | W4 | RMSE | RH RMSE max W4 | 15.0 percent | True WX139 | W4 | Bias | RH Bias min W4 TSO | 15.0 percent | False WX140 | W4 | Bias | RH Bias mean W4 TSO | 15.0 percent | False WX141 | W4 | Bias | RH Bias max W4 TSO | 15.0 percent | False WX142 | W4 | Bias | RH Bias min W4 | 15.0 percent | True WX143 | W4 | Bias | RH Bias mean W4 | 15.0 percent | True WX144 | W4 | Bias | RH Bias max W4 | 15.0 percent | True #### Wind Speed **Short IDs**: See Table
**KPI**: Wind Speed MAE/RMSE/Bias
**Normalization**: Symmetric Exponential Open Normalization ($m$ value in Table)
**Name in Score Card**: See Table
Each metric (MAE, RMSE, Bias) is calculated for each station for both model and observational dataset for a specified period. Then we apply summary statistics (*e.g.*, min, mean, Q3) across all available weather stations before applying the normalization. Implementation of metrics are `firebench.metrics.stats.mae`, `firebench.metrics.stats.rmse`, `firebench.metrics.stats.bias`. Datasets are converted into `m/s` for comparison. The normalization parameter $m$ sets which KPI value gives a Score of 50. It represents the difficulty of the benchmark. The following Table gives the correspondence between the benchmark ID and the study period: ID | Study period | Summary stats func | Name in Score Card | $m$ | trusted source only ------|--------------|--------------------|-------------------------|---------|-------------------- WX145 | W1 | MAE | Wind Speed MAE min W1 TSO | 5.0 m/s | False WX146 | W1 | MAE | Wind Speed MAE mean W1 TSO | 5.0 m/s | False WX147 | W1 | MAE | Wind Speed MAE max W1 TSO | 5.0 m/s | False WX148 | W1 | MAE | Wind Speed MAE min W1 | 5.0 m/s | True WX149 | W1 | MAE | Wind Speed MAE mean W1 | 5.0 m/s | True WX150 | W1 | MAE | Wind Speed MAE max W1 | 5.0 m/s | True WX151 | W1 | RMSE | Wind Speed RMSE min W1 TSO | 5.0 m/s | False WX152 | W1 | RMSE | Wind Speed RMSE mean W1 TSO | 5.0 m/s | False WX153 | W1 | RMSE | Wind Speed RMSE max W1 TSO | 5.0 m/s | False WX154 | W1 | RMSE | Wind Speed RMSE min W1 | 5.0 m/s | True WX155 | W1 | RMSE | Wind Speed RMSE mean W1 | 5.0 m/s | True WX156 | W1 | RMSE | Wind Speed RMSE max W1 | 5.0 m/s | True WX157 | W1 | Bias | Wind Speed Bias min W1 TSO | 5.0 m/s | False WX158 | W1 | Bias | Wind Speed Bias mean W1 TSO | 5.0 m/s | False WX159 | W1 | Bias | Wind Speed Bias max W1 TSO | 5.0 m/s | False WX160 | W1 | Bias | Wind Speed Bias min W1 | 5.0 m/s | True WX161 | W1 | Bias | Wind Speed Bias mean W1 | 5.0 m/s | True WX162 | W1 | Bias | Wind Speed Bias max W1 | 5.0 m/s | True WX163 | W2 | MAE | Wind Speed MAE min W2 TSO | 5.0 m/s | False WX164 | W2 | MAE | Wind Speed MAE mean W2 TSO | 5.0 m/s | False WX165 | W2 | MAE | Wind Speed MAE max W2 TSO | 5.0 m/s | False WX166 | W2 | MAE | Wind Speed MAE min W2 | 5.0 m/s | True WX167 | W2 | MAE | Wind Speed MAE mean W2 | 5.0 m/s | True WX168 | W2 | MAE | Wind Speed MAE max W2 | 5.0 m/s | True WX169 | W2 | RMSE | Wind Speed RMSE min W2 TSO | 5.0 m/s | False WX170 | W2 | RMSE | Wind Speed RMSE mean W2 TSO | 5.0 m/s | False WX171 | W2 | RMSE | Wind Speed RMSE max W2 TSO | 5.0 m/s | False WX172 | W2 | RMSE | Wind Speed RMSE min W2 | 5.0 m/s | True WX173 | W2 | RMSE | Wind Speed RMSE mean W2 | 5.0 m/s | True WX174 | W2 | RMSE | Wind Speed RMSE max W2 | 5.0 m/s | True WX175 | W2 | Bias | Wind Speed Bias min W2 TSO | 5.0 m/s | False WX176 | W2 | Bias | Wind Speed Bias mean W2 TSO | 5.0 m/s | False WX177 | W2 | Bias | Wind Speed Bias max W2 TSO | 5.0 m/s | False WX178 | W2 | Bias | Wind Speed Bias min W2 | 5.0 m/s | True WX179 | W2 | Bias | Wind Speed Bias mean W2 | 5.0 m/s | True WX180 | W2 | Bias | Wind Speed Bias max W2 | 5.0 m/s | True WX181 | W3 | MAE | Wind Speed MAE min W3 TSO | 5.0 m/s | False WX182 | W3 | MAE | Wind Speed MAE mean W3 TSO | 5.0 m/s | False WX183 | W3 | MAE | Wind Speed MAE max W3 TSO | 5.0 m/s | False WX184 | W3 | MAE | Wind Speed MAE min W3 | 5.0 m/s | True WX185 | W3 | MAE | Wind Speed MAE mean W3 | 5.0 m/s | True WX186 | W3 | MAE | Wind Speed MAE max W3 | 5.0 m/s | True WX187 | W3 | RMSE | Wind Speed RMSE min W3 TSO | 5.0 m/s | False WX188 | W3 | RMSE | Wind Speed RMSE mean W3 TSO | 5.0 m/s | False WX189 | W3 | RMSE | Wind Speed RMSE max W3 TSO | 5.0 m/s | False WX190 | W3 | RMSE | Wind Speed RMSE min W3 | 5.0 m/s | True WX191 | W3 | RMSE | Wind Speed RMSE mean W3 | 5.0 m/s | True WX192 | W3 | RMSE | Wind Speed RMSE max W3 | 5.0 m/s | True WX193 | W3 | Bias | Wind Speed Bias min W3 TSO | 5.0 m/s | False WX194 | W3 | Bias | Wind Speed Bias mean W3 TSO | 5.0 m/s | False WX195 | W3 | Bias | Wind Speed Bias max W3 TSO | 5.0 m/s | False WX196 | W3 | Bias | Wind Speed Bias min W3 | 5.0 m/s | True WX197 | W3 | Bias | Wind Speed Bias mean W3 | 5.0 m/s | True WX198 | W3 | Bias | Wind Speed Bias max W3 | 5.0 m/s | True WX199 | W4 | MAE | Wind Speed MAE min W4 TSO | 5.0 m/s | False WX200 | W4 | MAE | Wind Speed MAE mean W4 TSO | 5.0 m/s | False WX201 | W4 | MAE | Wind Speed MAE max W4 TSO | 5.0 m/s | False WX202 | W4 | MAE | Wind Speed MAE min W4 | 5.0 m/s | True WX203 | W4 | MAE | Wind Speed MAE mean W4 | 5.0 m/s | True WX204 | W4 | MAE | Wind Speed MAE max W4 | 5.0 m/s | True WX205 | W4 | RMSE | Wind Speed RMSE min W4 TSO | 5.0 m/s | False WX206 | W4 | RMSE | Wind Speed RMSE mean W4 TSO | 5.0 m/s | False WX207 | W4 | RMSE | Wind Speed RMSE max W4 TSO | 5.0 m/s | False WX208 | W4 | RMSE | Wind Speed RMSE min W4 | 5.0 m/s | True WX209 | W4 | RMSE | Wind Speed RMSE mean W4 | 5.0 m/s | True WX210 | W4 | RMSE | Wind Speed RMSE max W4 | 5.0 m/s | True WX211 | W4 | Bias | Wind Speed Bias min W4 TSO | 5.0 m/s | False WX212 | W4 | Bias | Wind Speed Bias mean W4 TSO | 5.0 m/s | False WX213 | W4 | Bias | Wind Speed Bias max W4 TSO | 5.0 m/s | False WX214 | W4 | Bias | Wind Speed Bias min W4 | 5.0 m/s | True WX215 | W4 | Bias | Wind Speed Bias mean W4 | 5.0 m/s | True WX216 | W4 | Bias | Wind Speed Bias max W4 | 5.0 m/s | True #### Wind Direction **Short IDs**: See Table
**KPI**: Wind Direction circular Bias
**Normalization**: Symmetric Exponential Open Normalization ($m$ value in Table)
**Name in Score Card**: See Table
Each metric is calculated for each station for both model and observational dataset for a specified period. Then we apply summary statistics (*e.g.*, min, mean, Q3) across all available weather stations before applying the normalization. Implementation of metrics are `firebench.metrics.stats.circular_bias_deg`. Datasets are converted into `degree` for comparison. The normalization parameter $m$ sets which KPI value gives a Score of 50. It represents the difficulty of the benchmark. The following Table gives the correspondence between the benchmark ID and the study period: ID | Study period | Summary stats func | Name in Score Card | $m$ | trusted source only ------|--------------|--------------------|-------------------------|---------|-------------------- WX217 | W1 | circular bias | Wind Direction circular bias min W1 TSO | 45.0 degree | False WX218 | W1 | circular bias | Wind Direction circular bias mean W1 TSO | 45.0 degree | False WX219 | W1 | circular bias | Wind Direction circular bias max W1 TSO | 45.0 degree | False WX220 | W1 | circular bias | Wind Direction circular bias min W1 | 45.0 degree | True WX221 | W1 | circular bias | Wind Direction circular bias mean W1 | 45.0 degree | True WX222 | W1 | circular bias | Wind Direction circular bias max W1 | 45.0 degree | True WX223 | W2 | circular bias | Wind Direction circular bias min W2 TSO | 45.0 degree | False WX224 | W2 | circular bias | Wind Direction circular bias mean W2 TSO | 45.0 degree | False WX225 | W2 | circular bias | Wind Direction circular bias max W2 TSO | 45.0 degree | False WX226 | W2 | circular bias | Wind Direction circular bias min W2 | 45.0 degree | True WX227 | W2 | circular bias | Wind Direction circular bias mean W2 | 45.0 degree | True WX228 | W2 | circular bias | Wind Direction circular bias max W2 | 45.0 degree | True WX229 | W3 | circular bias | Wind Direction circular bias min W3 TSO | 45.0 degree | False WX230 | W3 | circular bias | Wind Direction circular bias mean W3 TSO | 45.0 degree | False WX231 | W3 | circular bias | Wind Direction circular bias max W3 TSO | 45.0 degree | False WX232 | W3 | circular bias | Wind Direction circular bias min W3 | 45.0 degree | True WX233 | W3 | circular bias | Wind Direction circular bias mean W3 | 45.0 degree | True WX234 | W3 | circular bias | Wind Direction circular bias max W3 | 45.0 degree | True WX235 | W4 | circular bias | Wind Direction circular bias min W4 TSO | 45.0 degree | False WX236 | W4 | circular bias | Wind Direction circular bias mean W4 TSO | 45.0 degree | False WX237 | W4 | circular bias | Wind Direction circular bias max W4 TSO | 45.0 degree | False WX238 | W4 | circular bias | Wind Direction circular bias min W4 | 45.0 degree | True WX239 | W4 | circular bias | Wind Direction circular bias mean W4 | 45.0 degree | True WX240 | W4 | circular bias | Wind Direction circular bias max W4 | 45.0 degree | True #### Fuel Moisture Content 10h **Short IDs**: See Table
**KPI**: FMC 10h MAE/RMSE/Bias
**Normalization**: Symmetric Exponential Open Normalization ($m$ value in Table)
**Name in Score Card**: See Table
Each metric is calculated for each station for both model and observational dataset for a specified period. Then we apply summary statistics (*e.g.*, min, mean, Q3) across all available weather stations before applying the normalization. Implementation of metrics are `firebench.metrics.stats.mae`, `firebench.metrics.stats.rmse`, `firebench.metrics.stats.bias`. Datasets are converted into `percent` for comparison. The normalization parameter $m$ sets which KPI value gives a Score of 50. It represents the difficulty of the benchmark. The following Table gives the correspondence between the benchmark ID and the study period: ID | Study period | Summary stats func | Name in Score Card | $m$ | trusted source only ------|--------------|--------------------|-------------------------|---------|-------------------- WX241 | W1 | MAE | FMC 10h MAE min W1 TSO | 5.0 percent | False WX242 | W1 | MAE | FMC 10h MAE mean W1 TSO | 5.0 percent | False WX243 | W1 | MAE | FMC 10h MAE max W1 TSO | 5.0 percent | False WX244 | W1 | MAE | FMC 10h MAE min W1 | 5.0 percent | True WX245 | W1 | MAE | FMC 10h MAE mean W1 | 5.0 percent | True WX246 | W1 | MAE | FMC 10h MAE max W1 | 5.0 percent | True WX247 | W1 | RMSE | FMC 10h RMSE min W1 TSO | 5.0 percent | False WX248 | W1 | RMSE | FMC 10h RMSE mean W1 TSO | 5.0 percent | False WX249 | W1 | RMSE | FMC 10h RMSE max W1 TSO | 5.0 percent | False WX250 | W1 | RMSE | FMC 10h RMSE min W1 | 5.0 percent | True WX251 | W1 | RMSE | FMC 10h RMSE mean W1 | 5.0 percent | True WX252 | W1 | RMSE | FMC 10h RMSE max W1 | 5.0 percent | True WX253 | W1 | Bias | FMC 10h Bias min W1 TSO | 5.0 percent | False WX254 | W1 | Bias | FMC 10h Bias mean W1 TSO | 5.0 percent | False WX255 | W1 | Bias | FMC 10h Bias max W1 TSO | 5.0 percent | False WX256 | W1 | Bias | FMC 10h Bias min W1 | 5.0 percent | True WX257 | W1 | Bias | FMC 10h Bias mean W1 | 5.0 percent | True WX258 | W1 | Bias | FMC 10h Bias max W1 | 5.0 percent | True WX259 | W2 | MAE | FMC 10h MAE min W2 TSO | 5.0 percent | False WX260 | W2 | MAE | FMC 10h MAE mean W2 TSO | 5.0 percent | False WX261 | W2 | MAE | FMC 10h MAE max W2 TSO | 5.0 percent | False WX262 | W2 | MAE | FMC 10h MAE min W2 | 5.0 percent | True WX263 | W2 | MAE | FMC 10h MAE mean W2 | 5.0 percent | True WX264 | W2 | MAE | FMC 10h MAE max W2 | 5.0 percent | True WX265 | W2 | RMSE | FMC 10h RMSE min W2 TSO | 5.0 percent | False WX266 | W2 | RMSE | FMC 10h RMSE mean W2 TSO | 5.0 percent | False WX267 | W2 | RMSE | FMC 10h RMSE max W2 TSO | 5.0 percent | False WX268 | W2 | RMSE | FMC 10h RMSE min W2 | 5.0 percent | True WX269 | W2 | RMSE | FMC 10h RMSE mean W2 | 5.0 percent | True WX270 | W2 | RMSE | FMC 10h RMSE max W2 | 5.0 percent | True WX271 | W2 | Bias | FMC 10h Bias min W2 TSO | 5.0 percent | False WX272 | W2 | Bias | FMC 10h Bias mean W2 TSO | 5.0 percent | False WX273 | W2 | Bias | FMC 10h Bias max W2 TSO | 5.0 percent | False WX274 | W2 | Bias | FMC 10h Bias min W2 | 5.0 percent | True WX275 | W2 | Bias | FMC 10h Bias mean W2 | 5.0 percent | True WX276 | W2 | Bias | FMC 10h Bias max W2 | 5.0 percent | True WX277 | W3 | MAE | FMC 10h MAE min W3 TSO | 5.0 percent | False WX278 | W3 | MAE | FMC 10h MAE mean W3 TSO | 5.0 percent | False WX279 | W3 | MAE | FMC 10h MAE max W3 TSO | 5.0 percent | False WX280 | W3 | MAE | FMC 10h MAE min W3 | 5.0 percent | True WX281 | W3 | MAE | FMC 10h MAE mean W3 | 5.0 percent | True WX282 | W3 | MAE | FMC 10h MAE max W3 | 5.0 percent | True WX283 | W3 | RMSE | FMC 10h RMSE min W3 TSO | 5.0 percent | False WX284 | W3 | RMSE | FMC 10h RMSE mean W3 TSO | 5.0 percent | False WX285 | W3 | RMSE | FMC 10h RMSE max W3 TSO | 5.0 percent | False WX286 | W3 | RMSE | FMC 10h RMSE min W3 | 5.0 percent | True WX287 | W3 | RMSE | FMC 10h RMSE mean W3 | 5.0 percent | True WX288 | W3 | RMSE | FMC 10h RMSE max W3 | 5.0 percent | True WX289 | W3 | Bias | FMC 10h Bias min W3 TSO | 5.0 percent | False WX290 | W3 | Bias | FMC 10h Bias mean W3 TSO | 5.0 percent | False WX291 | W3 | Bias | FMC 10h Bias max W3 TSO | 5.0 percent | False WX292 | W3 | Bias | FMC 10h Bias min W3 | 5.0 percent | True WX293 | W3 | Bias | FMC 10h Bias mean W3 | 5.0 percent | True WX294 | W3 | Bias | FMC 10h Bias max W3 | 5.0 percent | True WX295 | W4 | MAE | FMC 10h MAE min W4 TSO | 5.0 percent | False WX296 | W4 | MAE | FMC 10h MAE mean W4 TSO | 5.0 percent | False WX297 | W4 | MAE | FMC 10h MAE max W4 TSO | 5.0 percent | False WX298 | W4 | MAE | FMC 10h MAE min W4 | 5.0 percent | True WX299 | W4 | MAE | FMC 10h MAE mean W4 | 5.0 percent | True WX300 | W4 | MAE | FMC 10h MAE max W4 | 5.0 percent | True WX301 | W4 | RMSE | FMC 10h RMSE min W4 TSO | 5.0 percent | False WX302 | W4 | RMSE | FMC 10h RMSE mean W4 TSO | 5.0 percent | False WX303 | W4 | RMSE | FMC 10h RMSE max W4 TSO | 5.0 percent | False WX304 | W4 | RMSE | FMC 10h RMSE min W4 | 5.0 percent | True WX305 | W4 | RMSE | FMC 10h RMSE mean W4 | 5.0 percent | True WX306 | W4 | RMSE | FMC 10h RMSE max W4 | 5.0 percent | True WX307 | W4 | Bias | FMC 10h Bias min W4 TSO | 5.0 percent | False WX308 | W4 | Bias | FMC 10h Bias mean W4 TSO | 5.0 percent | False WX309 | W4 | Bias | FMC 10h Bias max W4 TSO | 5.0 percent | False WX310 | W4 | Bias | FMC 10h Bias min W4 | 5.0 percent | True WX311 | W4 | Bias | FMC 10h Bias mean W4 | 5.0 percent | True WX312 | W4 | Bias | FMC 10h Bias max W4 | 5.0 percent | True ## Requirements The following sections list the datasets' requirements to run the different benchmarks. When the benchmark script runs, each requirement is validated against the HDF5 file provided as input (from the model output/data the user wants to evaluate). If a requirement is met, each corresponding benchmark is run. Each requirement lists the required datasets/groups (as paths) and the mandatory attributes for each dataset/group. The current version of FireBench does not support more complex checks (e.g., array size and dtype). Requirement | Benchmarks ---------------------- | ----------------- R01 | BD01 to BD06 R02 | SV01 to SV06 R03 | FP01, FP05, FP09, FP13, FP17, FP21, FP25, FP29 R04 | FP02, FP06, FP10, FP14, FP18, FP22, FP26, FP30 R05 | FP03, FP07, FP11, FP15, FP19, FP23, FP27, FP31 R06 | FP04, FP08, FP12, FP16, FP20, FP24, FP28, FP32 R07 | CC01 to CC06 R08 | WX001 to WX072 R09 | WX073 to WX144 R10 | WX145 to WX216 R11 | WX217 to WX240 R12 | WX241 to WX312 ### R01 Mandatory group/dataset| Mandatory attributes ---------------------- | -------------------- `/points/building_damaged/building_damage` | units ### R02 Mandatory group/dataset| Mandatory attributes ---------------------- | -------------------- `/spatial_2d/Caldor_MTBS`| crs `/spatial_2d/Caldor_MTBS/fire_burn_severity`| units, _FillValue `/spatial_2d/Caldor_MTBS/position_lat`| units `/spatial_2d/Caldor_MTBS/position_lon`| units ### R03 Mandatory group/dataset| Mandatory attributes ---------------------- | -------------------- `/polygons/Caldor_2021-08-18T20:30-07:00`| rel_path, time `/polygons/Caldor_2021-08-19T20:45-07:00`| rel_path, time `/polygons/Caldor_2021-08-20T20:20-07:00`| rel_path, time `/polygons/Caldor_2021-08-21T21:15-07:00`| rel_path, time `/polygons/Caldor_2021-08-24T22:07-07:00`| rel_path, time `/polygons/Caldor_2021-08-26T03:30-06:00`| rel_path, time `/polygons/Caldor_2021-08-26T22:15-06:00`| rel_path, time `/polygons/Caldor_2021-08-27T00:22-06:00`| rel_path, time `/polygons/Caldor_2021-08-28T21:30-06:00`| rel_path, time `/polygons/Caldor_2021-08-29T22:32-07:00`| rel_path, time `/polygons/Caldor_2021-08-30T21:09-07:00`| rel_path, time `/polygons/Caldor_2021-08-31T21:08-07:00`| rel_path, time `/polygons/Caldor_2021-09-01T21:12-07:00`| rel_path, time `/polygons/Caldor_2021-09-03T00:40-07:00`| rel_path, time `/polygons/Caldor_2021-09-04T23:29-07:00`| rel_path, time `/polygons/Caldor_2021-09-05T23:41-07:00`| rel_path, time `/polygons/Caldor_2021-09-06T23:09-07:00`| rel_path, time `/polygons/Caldor_2021-09-07T22:40-07:00`| rel_path, time `/polygons/Caldor_2021-09-08T22:33-07:00`| rel_path, time `/polygons/Caldor_2021-09-10T23:34-07:00`| rel_path, time Files (KML) at path defined in `rel_path` attributes must exist. ### R04 Mandatory group/dataset| Mandatory attributes ---------------------- | -------------------- `/polygons/Caldor_2021-08-20T20:20-07:00`| rel_path, time `/polygons/Caldor_2021-08-21T21:15-07:00`| rel_path, time Files (KML) at path defined in `rel_path` attributes must exist. ### R05 Mandatory group/dataset| Mandatory attributes ---------------------- | -------------------- `/polygons/Caldor_2021-08-26T22:15-06:00`| rel_path, time `/polygons/Caldor_2021-08-27T00:22-06:00`| rel_path, time `/polygons/Caldor_2021-08-28T21:30-06:00`| rel_path, time Files (KML) at path defined in `rel_path` attributes must exist. ### R06 Mandatory group/dataset| Mandatory attributes ---------------------- | -------------------- `/polygons/Caldor_2021-08-29T22:32-07:00`| rel_path, time `/polygons/Caldor_2021-08-30T21:09-07:00`| rel_path, time `/polygons/Caldor_2021-08-31T21:08-07:00`| rel_path, time `/polygons/Caldor_2021-09-01T21:12-07:00`| rel_path, time `/polygons/Caldor_2021-09-03T00:40-07:00`| rel_path, time Files (KML) at path defined in `rel_path` attributes must exist. ### R07 Mandatory group/dataset| Mandatory attributes ---------------------- | -------------------- `/spatial_2d/ravg_cc`| crs `/spatial_2d/ravg_cc/ravg_canopy_cover_loss`| units, _FillValue `/spatial_2d/ravg_cc/position_lat`| units `/spatial_2d/ravg_cc/position_lon`| units ### R08 Verify that the model and observational datasets contain the same weather station groups with the following datasets: Mandatory group/dataset| Mandatory attributes ---------------------- | -------------------- `/time_series/station_/time`| None `/time_series/station_/air_temperature`| None ### R09 Verify that the model and observational datasets contain the same weather station groups with the following datasets: Mandatory group/dataset| Mandatory attributes ---------------------- | -------------------- `/time_series/station_/time`| None `/time_series/station_/relative_humidity`| None ### R10 Verify that the model and observational datasets contain the same weather station groups with the following datasets: Mandatory group/dataset| Mandatory attributes ---------------------- | -------------------- `/time_series/station_/time`| None `/time_series/station_/wind_speed`| None ### R11 Verify that the model and observational datasets contain the same weather station groups with the following datasets: Mandatory group/dataset| Mandatory attributes ---------------------- | -------------------- `/time_series/station_/time`| None `/time_series/station_/wind_direction`| None ### R12 Verify that the model and observational datasets contain the same weather station groups with the following datasets: Mandatory group/dataset| Mandatory attributes ---------------------- | -------------------- `/time_series/station_/time`| None `/time_series/station_/fuel_moisture_content_10h`| None ## Aggregation Schemes This section describes the weights used to aggregate KPI unit scores. More information about aggregation methods [here](../../metrics/score.md). If the aggregation scheme `0` is specified, then no aggregation is performed. Therefore, group scores and total scores are not computed. ### Group definition All benchmarks have a default weight of 1 in each group. If custom weights are applied, refer to the custom weight Table. Weight precedence: - Default benchmark weight: 1 - Group benchmark overrides: apply to all schemes unless overridden - Scheme benchmark overrides: apply only within that scheme and override everything else Group | Benchmark ID --------------------------- | ------------ Building Damage | BD01 to BD06 Burn Severity | SV01 to SV06 Fire Perimeter W1 | FP01, FP05, FP09, FP13, FP17, FP21, FP25, FP29 Fire Perimeter W2 | FP02, FP06, FP10, FP14, FP18, FP22, FP26, FP30 Fire Perimeter W3 | FP03, FP07, FP11, FP15, FP19, FP23, FP27, FP31 Fire Perimeter W4 | FP04, FP08, FP12, FP16, FP20, FP24, FP28, FP32 Canopy Cover Loss | CC01 to CC06 Air temperature W1 | WX001 to WX018 Air temperature W2 | WX019 to WX036 Air temperature W3 | WX037 to WX054 Air temperature W4 | WX055 to WX072 Relative humidity 10h W1 | WX073 to WX090 Relative humidity 10h W2 | WX091 to WX108 Relative humidity 10h W3 | WX109 to WX126 Relative humidity 10h W4 | WX127 to WX144 Wind speed W1 | WX145 to WX162 Wind speed W2 | WX163 to WX180 Wind speed W3 | WX181 to WX198 Wind speed W4 | WX199 to WX216 Wind direction W1 | WX217 to WX222 Wind direction W2 | WX223 to WX228 Wind direction W3 | WX229 to WX234 Wind direction W4 | WX235 to WX240 Fuel Moisture 10h W1 | WX241 to WX258 Fuel Moisture 10h W2 | WX259 to WX276 Fuel Moisture 10h W3 | WX277 to WX294 Fuel Moisture 10h W4 | WX295 to WX312 ### Scheme A Scheme A contains all the groups with default weights. It can be used to evaluate complete model performance with balanced weighting. ### Scheme B Scheme B contains only the building damage group. It is used to evaluate the model only on building damage benchmarks. Group | Group Weight ---------------------- | ------------ Building Damage | 1 ### Scheme CC Scheme CC contains only the canopy cover loss group. It is used to evaluate crown fire models. Group | Group Weight ---------------------- | ------------ Canopy Cover Loss | 1 ### Scheme FP Scheme FP contains only the fire perimeter groups. It is used to evaluate the model only on fire perimeter benchmarks for all of the study periods. Group | Group Weight ---------------------- | ------------ Fire Perimeter W1 | 1 Fire Perimeter W2 | 1 Fire Perimeter W3 | 1 Fire Perimeter W4 | 1 ### Scheme short_all Scheme short_all contains all the groups except the groups relative to W1 study period. Therefore, the index i is in [2, 4]. Group | Group Weight ---------------------- | ------------ Air Temp Wi | 1 Building Damage | 1 Burn Severity | 1 Canopy Cover Loss | 1 Fire Perimeter Wi | 1 FMC 10h Wi | 1 RH Wi | 1 Wind Direction Wi | 1 Wind Speed Wi | 1 ### Scheme S Scheme S contains only the burn severity group. It is used to evaluate the model only on building severity from MTBS benchmarks. Group | Group Weight ---------------------- | ------------ Burn Severity | 1 ### Scheme WXi Schemes WXi, for i in [1, 4], contains all the group related to weather stations for a specific study period (W1 to W4) Group | Group Weight ---------------------- | ------------ Air Temp Wi | 1 FMC 10h Wi | 1 RH Wi | 1 Wind Direction Wi | 1 Wind Speed Wi | 1 ### Scheme WX_short Scheme short_all contains all the groups except the groups relative to W1 study period and fire perimeter groups. Therefore, the index i is in [2, 4]. Group | Group Weight ---------------------- | ------------ Air Temp Wi | 1 Building Damage | 1 Burn Severity | 1 Canopy Cover Loss | 1 FMC 10h Wi | 1 RH Wi | 1 Wind Direction Wi | 1 Wind Speed Wi | 1 ## Notes - **Benchmark identifiers** consist of a *case ID* and a *short ID*, for example `FB001-BD01`. Throughout the documentation, the *short ID* alone (e.g. `BD01`) is used when the benchmark case is unambiguous, in order to improve readability. The *full identifier* (`FB001-BD01`) is used whenever the case context must be explicit, such as when comparing benchmarks across different cases. - Each file hash has been performed using `firebench.standardize.calculate_sha256`. - Collection of forecasts or reanalysis is authorized for the benchmark period (e.g., for fire perimeters) but has to be detailed in the model report attached to the Report sent back to the FireBench team for collection and validation of results. ## Acknowledgment - We gratefully acknowledge [Synoptic](https://synopticdata.com) for granting permission to redistribute selected weather-station data as part of the FireBench benchmarking framework. - I would like to thank my colleague Muthu K. Selvaraj (WPI) for his help in this project.