PCMDI Simulation Summaries: CMIP mean state and variability (v1.7.0)
The PCMDI Metrics Package (PMP) is a capability that is used to produce a diverse suite of “quick-look” objective summaries of Earth System Model (ESM) agreement with observations. The PMP is routinely applied to multiple generations of CMIP, including the most recent results from CMIP6 as they become available. These results are regularly updated as additional simulations become available, new analysis are included, and as presentation improvements and corrections are made.
Results are also accessible from the Coordinated Model Evaluation Capabilities (CMEC) website.
Mean Climate (results)
- Using well-established statistics, we provide large-scale seasonal and mean state climatology comparisons between CMIP simulations and observationally-based data. These include traditional measures (e.g. bias, pattern correlation and root-mean-square error) for global, hemispheric, tropical, extra-tropical, and other selected domains using satellite data and atmospheric reanalysis as references. These statistics are routinely computed as part of model evaluation. We use summary diagrams developed by PCMDI scientists (Taylor 2001; Gleckler et al. 2008) to objectively compare the consistency between the observed and simulated climate.
Back to List
Benchmarking Simulated Precipitation (results)
- These results were inspired by the outcomes of a July 2019 DOE workshop. Several teams were established at this workshop with one group tasked to incorporate an initial set of benchmarks into a common analysis framework and applying it to CMIP6 and earlier generations of Earth System Models (Pendergrass, et al., 2019). The results presented here illustrate the progress of this benchmarking effort. In parallel, a second group continues to develop exploratory metrics. Ultimately, this effort aims to provide a guide to modelers as they strive to improve simulated precipitation.
Back to List
El Niño-Southern Oscillation (results)
- El Niño-Southern Oscillation (ENSO) is the dominant mode of interannual variability in the tropical Pacific and has far reaching impacts on global climate. It is there therefore key to ensure its correct simulation in state-of-the-art Earth System Models. Community-wide synthesis of metrics to evaluate the performance, teleconnections and processes of ENSO in coupled GCMs is proposed by the ENSO working group of the International CLIVAR Pacific panel. The corresponding objective comparisons of simulations against observations shown here result from a collaboration between scientists at Institut Pierre Simon Laplace (IPSL) and PCMDI. This effort strives to improve and expand upon the ENSO model performance tests proposed by Bellenger et al. (2014) for CMIP5. The metrics are demonstrated through application to the CMIP archive following works of Planton et al. (2020, BAMS, under review).
Back to List
- Based on the work of Lee et al. (2019, 2021), we present skill metrics for the Northern Annular Model (NAM), the North Atlantic Oscillation (NAO), the Southern Annular Mode (SAM), the Pacific North American pattern (PNA), the North Pacific Oscillation (NPO), the Pacific Decadal Oscillation (PDO), and the North Pacific Gyre Oscillation (NPGO). For NAM, NAO, SAM, PNA, and NPO the results are based on sea-level pressure, while the results for PDO and NPGO are based on sea surface temperature. Our approach distinguishes itself from other studies that analyze modes of variability in that we use the Common Basis Function approach (CBF), in which model anomalies are projected onto the observed modes of variability. Using the Historical simulations, the skill of the spatial patterns is given by the Root-Mean-Squared-Error (RMSE), and the Amplitude gives the standard deviation of the Principal Component time series. The skill metrics are calculated with respect to a primary and secondary sets of observations denoted by the triangles in each cell of the Portrait Plots.
Back to List
Madden-Julian Oscillation (results)
- Based on the work of Ahn et al. (2017), we present skill metrics that indicate how well models simulate eastward propagation of the MJO. We apply frequency-wavenumber decomposition to precipitation from observations (GPCP-based; 1997-2010) and the CMIP5 and CMIP6 Historical simulations (1985-2004).
Back to List
Monsoon Characteristics (example) (results)
- Based on the work of Sperber and Annamalai (2014), we present skill metrics that indicate how well models simulate the onset, decay, and duration of monsoon based on the analysis of climatological pentads of precipitation. Using Historical simulations, the results are based on area-averaged data for All-India Rainfall (AIR), Sahel, Gulf of Guinea (GoG), North American Monsoon (NAM), South American Monsoon (SAM), and Northern Australia (AUS).
Back to List
Update History
- v1.7.0 (2024-11-06): Prototype sea-ice and QBO results added
- v1.6.2 (2023-11-21): ENSO and MJO interactive plots updated (minor fix)
- v1.6.1 (2023-10-09): MJO interactive bar chart updated.
- v1.6.0 (2023-05-19): Precipitation distribution metrics newly added and modes of variability updated.
- v1.5.1 (2022-11-04): Mean climate interactive portrait plot updated and precipitation variability across timescale portrait plot newly added.
- v1.5.0 (2020-10-08): Precipitation benchmarking newly added and Mean climate parallel coordinate and portrait plots updated.
- v1.4.1 (2020-07-20): MJO metrics with recent CMIP6 results
- v1.4.0 (2020-07-10): ENSO Metrics updated with Interactive Portrait Plot with recent CMIP6 results
- v1.3.2 (2020-06-19): Mean climate summaries updated with recent CMIP6 results with OBS info updated using PCMDIobs2
- v1.3.1 (2019-10-07): Mean climate summaries updated with recent CMIP6 results
- v1.3.0 (2019-09-06): ENSO metrics added
- v1.2.0 (2019-08-29): Mean climate metrics added
- v1.1.0 (2019-07-18): MJO metrics added
- v1.0.0 (2019-06-20): Initial public release
- v1.0.0-beta (2019-06-18): Monsoon precipitation onset, decay, and duration (CMIP5) added
- v1.0.0-alpha (2019-05-31): Test release: Extratropical Modes of Variability (CMIP5 and CMIP6)
Back to List
References
Latest:
- Lee, J., Gleckler, P. J., Ahn, M.-S., Ordonez, A., Ullrich, P. A., Sperber, K. R., Taylor, K. E., Planton, Y. Y., Guilyardi, E., Durack, P., Bonfils, C., Zelinka, M. D., Chao, L.-W., Dong, B., Doutriaux, C., Zhang, C., Vo, T., Boutte, J., Wehner, M. F., Pendergrass, A. G., Kim, D., Xue, Z., Wittenberg, A. T., and Krasting, J.: Systematic and objective evaluation of Earth system models: PCMDI Metrics Package (PMP) version 3, Geosci. Model Dev., 17, 3919–3948, https://doi.org/10.5194/gmd-17-3919-2024, 2024.
Earlier versions:
-
Gleckler, P. J., Doutriaux, C., Durack, P. J., Taylor, K. E., Zhang, Y., Williams, D. N., Mason, E., and Servonnat, J.: A more powerful reality test for climate models, Eos T. Am. Geophys. Un., 97, https://doi.org/10.1029/2016eo051663, 2016.
-
Gleckler, P. J., Taylor, K. E., and Doutriaux, C.: Performance metrics for climate models, J. Geophys. Res., 113, D06104, https://doi.org/10.1029/2007jd008972, 2008.
Questions or comments about the PCMDI Simulation Summaries should be sent to the PMP team.
The work of PCMDI and results presented on this web page are supported by the Regional and Global Model Analysis (RGMA) program area under the Earth and Environmental System Modeling (EESM) program within the Earth and Environmental Systems Sciences Division (EESSD) of the United States Department of Energy’s (DOE) Office of Science (OSTI). The work of PCMDI is performed under the auspices of the U.S. DOE by Lawrence Livermore National Laboratory (LLNL) under contract DE-AC52-07NA27344. LLNL-WEB-812310.