Research > Metrics > Extratropical Modes of Variability
Extratropical Modes of Variability
RMSE Portrait Plots
(Click thumbnail image for the interactive version for CMIP6 historical)
The RMSE Portrait Plots for the modes of variability are akin to Fig. 1a from Lee et al. (2021). For CMIP5 and CMIP6 they show the RMSE relative to the median RMSE of all respective simulations for each mode and season. The RMSE contains errors that arise both in terms of pattern error and amplitude error. The mode/season results for all models are organized by rows, whereas the results from individual realizations across all modes and seasons are isolated by columns. In each cell the normalized results are shown, with for example a result of 0.50 indicating the model RMSE is 50% greater than the median RMSE, whereas a result of −0.25 reflects an error that is 25% smaller than the median of all model results for that particular mode and season. Reference datasets used for this plot are 20CR for the sea-level pressure based modes and HadISSTv1.1 for the PDO and NPGO. Results with alternate reference datasets (ERA-20C for sea-level pressure based modes, HadISSTv2.1 for the PDO) will be added to the interactive plots in the future.
For the RMSE, the advantage of the Portrait Plot is that:
- It allows us to show on a single plot the model performance irrespective of the units of the variable under consideration and/or where the range of the absolute RMSE is different between modes/seasons
- It preserves the relative model performance with respect to the observations
- It gives an indication of how well a given model is performing in a multi-model context
The Portrait Plots are interactive. Clicking on any cell triangle will give its associated value, the observed EOF’s using our primary and secondary observed datasets, EOF’s 1-3 of the model, the model CBF EOF, and the PC time series of the EOF’s and the CBF EOF. Clicking on a mode along the y-axis will give the associated Taylor Plot of all models for that mode/season.
Summary statistics in RMSE Portrait Plots
General Conclusions
There are several general conclusions that we can draw from the RMSE Portrait Plots. Firstly, though the selection of a reference dataset is important, it does not radically impact the general conclusions about a particular model’s agreement with the observed EOF spatial pattern. Secondly, with some binning of the results (the color bar uses 10% increments), we see that in many cases the results for a given model’s relative performance is not very sensitive to which realization is examined. Our results are thus fairly robust to internal variability and selection of reference data. A more detailed discussion of the RMSE results are given in Lee et al. (2019) and Lee et al. (2021).
—
Amplitude Portrait Plots
(Click thumbnail image for the interactive version for CMIP6 historical)
The amplitude Portrait Plots for the modes of variability are akin to Fig. 4a from Lee et al. (2021). It gives the standard deviation of the model CBF PC time series divided by the standard deviation of the observed PC time series such that values close to one indicate consistency between the models and observations.
The Portrait Plots are interactive. Clicking on any cell triangle will give its associated value, the observed EOF’s using our primary and secondary observed datasets, EOF’s 1-3 of the model, the model CBF EOF, and the PC time series of the EOF’s and the CBF EOF. Clicking on a mode along the y-axis will give the associated Taylor Plot of all models for that mode/season.
Summary statistics in Amplitude Portrait Plots
General Conclusions
It is notable that for the majority of amplitude cells the model-to-observed reference ratios are within 0.9–1.1. There is substantial overlap between the RMSE Portrait Plots and the amplitude Portrait Plots. In particular, the amplitude outliers reinforce other Lee et al. (2019) evidence that temporal amplitude errors dominate the RMSE in many cases. Additionally, the results indicate a systematic error in the season immediately following the dominant season during which nearly 80% or more of the models overestimate the temporal variability. The post-dominant season overestimate suggests that the dominant season (winter) mode is too persistent compared to observations, though this hypothesis requires further investigation.
Reference
Lee, J., K. Sperber, P. Gleckler, K. Taylor, and C. Bonfils, 2021: Benchmarking performance changes in the simulation of extratropical modes of variability across CMIP generations. Journal of Climate, 34, 6945–6969, doi: 10.1175/JCLI-D-20-0832.1
Lee, J., K. Sperber, P. Gleckler, C. Bonfils, and K. Taylor, 2019: Quantifying the agreement between observed and simulated extratropical modes of interannual variability. Climate Dynamics, 52: 4057-4089, doi: 10.1007/s00382-018-4355-4