Credibility via Coupling

Institutions and Infrastructures in Climate Model Intercomparisons

  • Matthew S. Mayernik National Center for Atmospheric Research


This study investigates Model Intercomparison Projects (MIPs) as one example of a coordinated approach to establishing scientific credibility. MIPs originated within climate science as a method to evaluate and compare disparate climate models, but MIPs or MIP-like projects are now spreading to many scientific fields. Within climate science, MIPs have advanced knowledge of: a) the climate phenomena being modeled, and b) the building of climate models themselves. MIPs thus build scientific confidence in the climate modeling enterprise writ large, reducing questions of the credibility or reproducibility of any single model. This paper will discuss how MIPs organize people, models, and data through institution and infrastructure coupling (IIC). IIC involves establishing mechanisms and technologies for collecting, distributing, and comparing data and models (infrastructural work), alongside corresponding governance structures, rules of participation, and collaboration mechanisms that enable partners around the world to work together effectively (institutional work). Coupling these efforts involves developing formal and informal ways to standardize data and metadata, create common vocabularies, provide uniform tools and methods for evaluating resulting data, and build community around shared research topics.


Ankeny, Rachel A., and Sabina Leonelli. 2011. “What’s so Special about Model Organisms?” Studies in History and Philosophy of Science Part A 42 (2): 313–23.

Ankeny, Rachel A., and Sabina Leonelli. 2016. “Repertoires: A Post-Kuhnian Perspective on Scientific Change and Collaborative Research.” Studies in History and Philosophy of Science Part A 60 (December): 18–28.

Aula, Ville. 2019. “Institutions, Infrastructures, and Data Friction – Reforming Secondary Use of Health Data in Finland.” Big Data & Society 6 (2): 205395171987598.

Balaji, V., Karl E. Taylor, Martin Juckes, Bryan N. Lawrence, Paul J. Durack, Michael Lautenschlager, Chris Blanton, et al. 2018. “Requirements for a Global Data Infrastructure in Support of CMIP6.” Geoscientific Model Development 11 (9): 3659–80.

Bik, Elisabeth M., Ferric C. Fang, Amy L. Kullas, Roger J. Davis, and Arturo Casadevall. 2018. “Analysis and Correction of Inappropriate Image Duplication: The Molecular and Cellular Biology Experience.” Molecular and Cellular Biology 38 (20).

Borgman, Christine L. 2012. “The Conundrum of Sharing Research Data.” Journal of the American Society for Information Science and Technology 63 (6): 1059–78.

Braun, Kathrin, and Cordula Kropp. 2010. “Beyond Speaking Truth? Institutional Responses to Uncertainty in Scientific Governance.” Science, Technology, & Human Values 35 (6): 771–82.

Bush, Rosemary T. 2018. Perspectives on Reproducibility and Replication of Results in Climate Science. Paper commissioned for the National Academies of Sciences, Engineering, and Medicine Committee on Reproducibility and Replicability in Science.

Callaghan, Sarah A., M. Elkington, R. Ford, M. Kolaninski, M. Lautenschlager, B. Lawrence, L. Steenman-Clark, et al. 2010. “The METAFOR Project: Preserving data through metadata standards for climate models and simulations.” In Proceedings of the 1st International Digital Preservation Interoperability Framework Symposium on - INTL-DPIF ’10. ACM Press, 1-8.

Cess, R. D., Potter, G. L., Blanchet, J. P., Boer, G. J., Ghan, S. J., Kiehl, J. T., … Yagai, I. 1989. “Interpretation of Cloud-Climate Feedback as Produced by 14 Atmospheric General Circulation Models.” Science 245 (4917): 513–16.

Christopher, Jana. 2018. “Systematic Fabrication of Scientific Images Revealed.” FEBS Letters 592 (18): 3027–29.

Cinquini, L., Crichton, D., Mattmann, C., Harney, J., Shipman, G., Wang, F., … Schweitzer, R. 2014. “The Earth System Grid Federation: An Open Infrastructure for Access to Distributed Geospatial Data.” Future Generation Computer Systems 36 (July): 400–417.

Collins, Harry. 1985. Changing Order: Replication and Induction in Scientific Practice; University of Chicago Press: Chicago, IL.

Collins, Harry. 2013. Gravity’s Ghost and Big Dog: Scientific Discovery and Social Analysis in the Twenty-First Century. Chicago, IL: University of Chicago Press.

Darch, Peter T. 2018. “Limits to the Pursuit of Reproducibility: Emergent Data-Scarce Domains of Science.” In Transforming Digital Worlds, 164–74. Springer International Publishing.

Deser, Clara, F. Lehner, K. B. Rodgers, T. Ault, T. L. Delworth, P. N. DiNezio, A. Fiore, et al. 2020. “Insights from Earth System Model Initial-Condition Large Ensembles and Future Prospects.” Nature Climate Change 10 (4): 277–86.

Easterbrook, Steve M. 2014. “Open Code for Open Science?” Nature Geoscience 7 (11): 779–81.

Edwards, Paul N. 2010. A Vast Machine: Computer Models, Climate Data, and the Politics of Global Warming. Cambridge, MA: MIT Press.

Edwards, Paul N. 2016. “Knowledge Infrastructures for the Anthropocene.” The Anthropocene Review 4 (1): 34–43.

Edwards, Paul N. 2019a. “Infrastructuration: On Habits, Norms and Routines as Elements of Infrastructure*.” In M. Kornberger et al (Eds.), Thinking Infrastructures, 355–66. Emerald Publishing Limited.

Edwards, Paul N. 2019b. “Knowledge infrastructures under siege: Climate data as memory, truce, and target.” In D. Bigo, E. Isin, E. Ruppert (Eds.), Data Politics: Worlds, Subjects, Rights, 21-42. New York: Routledge.

Edwards, Paul N., Jackson, S.J., Bowker, G.C., & Knobel, C.P. 2007. Understanding Infrastructure: Dynamics, Tensions, and Design. Ann Arbor, MI: University of Michigan.

Fanelli, Daniele. 2018. “Opinion: Is Science Really Facing a Reproducibility Crisis, and Do We Need It To?” Proceedings of the National Academy of Sciences 115 (11): 2628–31.

Feinberg, Melanie, Will Sutherland, Sarah Beth Nelson, Mohammad Hossein Jarrahi, and Arcot Rajasekar. 2020. “The New Reality of Reproducibility: The Role of Data Work in Scientific Research.” Proceedings of the ACM on Human-Computer Interaction 4 (CSCW1): 1–22.

Galison, Peter. 1987. How Experiments End. Chicago, IL: University of Chicago Press.

Gates, W. Lawrence. (Ed.) 1979. Report of the JOC Study Conference on Climate Models: Performance, Intercomparison and Sensitivity Studies, Volume I. Global Atmospheric Research Programme (GARP) Publications Series No. 22.

Gates, W. Lawrence. 1992. “AMIP: The Atmospheric Model Intercomparison Project.” Bulletin of the American Meteorological Society 73 (12): 1962–70.<1962:atamip>;2

Gates, W.L. 1995. “An overview of AMIP and preliminary results.” In Proceedings of the First International AMIP Scientific Conference. World Climate Research Programme, WCRP-92, WMO/TD-No. 732, pp. 1-8.

Gates, W. Lawrence, James S. Boyle, Curt Covey, Clyde G. Dease, Charles M. Doutriaux, Robert S. Drach, Michael Fiorino, et al. 1999. “An Overview of the Results of the Atmospheric Model Intercomparison Project (AMIP I).” Bulletin of the American Meteorological Society 80 (1): 29–55.<0029:aootro>;2

Guilyardi, Eric, Balaji, V., Callaghan, S., DeLuca, C., Devine, G., Denvil, S., Ford, R., et al. 2011. “The CMIP5 model and simulation documentation: a new standard for climate modelling metadata.” CLIVAR Exchanges 56: 16 (2). pp. 42-46.

Harman, Donna K. 2005. “The TREC test collections.” In E.M. Voorhees & D.K. Harman (Eds), TREC: Experiment and Evaluation in Information Retrieval, 21-52. Cambridge, MA: MIT Press.

Harman, Donna K., and Ellen M. Voorhees. 2007. “TREC: An Overview.” Annual Review of Information Science and Technology 40 (1): 113–55.

Harter, Stephen P. 1996. “Variations in Relevance Assessments and the Measurement of Retrieval Effectiveness.” Journal of the American Society for Information Science 47 (1): 37–49.<37::aid-asi4>;2-3

Hoeppe, Götz. 2014. “Working Data Together: The Accountability and Reflexivity of Digital Astronomical Practice.” Social Studies of Science 44 (2): 243–70.

Howe, Cymene, Jessica Lockrem, Hannah Appel, Edward Hackett, Dominic Boyer, Randal Hall, Matthew Schneider-Mayerson, et al. 2015. “Paradoxical Infrastructures.” Science, Technology, & Human Values 41 (3): 547–65.

Jasanoff, Sheila. 2004. Ordering knowledge, ordering society. In S. Jasanoff (Ed), States of Knowledge: The Co-Production of Science and Social Order, 13-45. New York: Routledge.

Jebeile, Julie, and Michel Crucifix. 2020. “Multi-Model Ensembles in Climate Science: Mathematical Structures and Expert Judgements.” Studies in History and Philosophy of Science Part A 83 (October): 44–52.

Jefferson, Gail. 1985. “On the Interactional Unpackaging of a ‘Gloss.’” Language in Society 14 (4): 435–66.

Jensen, Casper Bruun, and Atsuro Morita. 2015. “Infrastructures as Ontological Experiments.” Engaging Science, Technology, and Society 1: 81-87.

Jepperson, R.L. 1991. “Institutions, institutional effects, and institutionalism.” In W.W. Powell & P.J. DiMaggio (Eds.), The New Institutionalism in Organizational Analysis, 143-163. Chicago, IL: University of Chicago Press.

Jewett, Tom, and Rob Kling. 1991. “The Dynamics of Computerization in a Social Science Research Team: A Case Study of Infrastructure, Strategies, and Skills.” Social Science Computer Review 9 (2): 246–75.

Kaiser, Jocelyn. 2003. “Ecological Networks: Panel Suggests a Different Shade of NEON.” Science 301 (5641): 1828a–11828.

Kennefick, Daniel. 2007. Traveling at the Speed of Thought: Einstein and the Quest for Gravitational Waves. Princeton, NJ: Princeton University Press.

Latour, Bruno. 2013. An Inquiry into Modes of Existence: An Anthropology of the Moderns. Trans. Catherine Porter. Cambridge, MA: Harvard University Press.

Lee, Charlotte P., and Kjeld Schmidt. 2018. “A Bridge Too Far? Critical remarks on the concept of ‘infrastructure’ in CSCW and IS.” In V. Wulf, et al.(eds): Socio-Informatics: A Practice-based Perspective on the Design and Use of IT Artifacts. Oxford: Oxford University Press, pp. 177-217. Oxford: Oxford University Press.

Leonelli, Sabina. 2018. “Rethinking Reproducibility as a Criterion for Research Quality.” In Including a Symposium on Mary Morgan: Curiosity, Imagination, and Surprise, 129–46. Emerald Publishing Limited.

Leonelli, Sabina. 2019. “Scientific agency and social scaffolding in contemporary data-intensive biology.” In: Wimsatt, W. and Love, A. (Eds) Beyond the Meme: Articulating Dynamic Structures in Cultural Evolution, 42-63. Minneapolis: University of Minnesota Press.

Loescher, H.W., Kelly, E.F., and Lea, R. 2017. “National Ecological Observatory Network: Beginnings, programmatic and scientific challenges, and ecological forecasting.” In A. Chabbi & H.W. Loescher (Eds), Terrestrial Ecosystem Research Infrastructures, 27-51. Boca Raton, FL: CRC Press.

Martin, P., Chen, Y., Hardisty, A., Jeffery, K., and Zhao, Z. 2017. “Computational challenges in global environmental research infrastructures.” In A. Chabbi & H.W. Loescher (Eds), Terrestrial Ecosystem Research Infrastructures, 305-340. Boca Raton, FL: CRC Press.

Mayernik, Matthew S. 2016. “Research Data and Metadata Curation as Institutional Issues.” Journal of the Association for Information Science and Technology 67 (4): 973–93.

Mayernik, Matthew S. 2019. “Metadata Accounts: Achieving Data and Evidence in Scientific Research.” Social Studies of Science 49 (5): 732–57.

Meehl, Gerald A., George J. Boer, Curt Covey, Mojib Latif, and Ronald J. Stouffer. 1997. “Intercomparison Makes for a Better Climate Model.” Eos, Transactions American Geophysical Union 78 (41): 445.

Meehl, Gerald A., Curt Covey, Thomas Delworth, Mojib Latif, Bryant McAvaney, John F. B. Mitchell, Ronald J. Stouffer, and Karl E. Taylor. 2007. “THE WCRP CMIP3 Multimodel Dataset: A New Era in Climate Change Research.” Bulletin of the American Meteorological Society 88 (9): 1383–94.

Mervis, J. 2015. “NSF Looks for New Contractor to Finish Troubled Observatory.” Science 350 (6267): 1454.

Mervis, Jeffrey. 2019. “Shake-up Threatens Novel U.S. Ecology Facility.” Science 363 (6424): 211–12.

Miller, Clark. 2001. “Hybrid Management: Boundary Organizations, Science Policy, and Environmental Governance in the Climate Regime.” Science, Technology, & Human Values 26 (4): 478–500.

Miller, Seumas. 2011. “Social Institutions.” In Stanford Encyclopedia of Philosophy. Stanford, CA: Stanford University.

Miyakawa, Tsuyoshi. 2020. “No Raw Data, No Science: Another Possible Source of the Reproducibility Crisis.” Molecular Brain 13 (1).

Moylan, Elizabeth C, and Maria K Kowalczuk. 2016. “Why Articles Are Retracted: A Retrospective Cross-Sectional Study of Retraction Notices at BioMed Central.” BMJ Open 6 (11): e012047.

[NASEM] National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press.

North, Douglas C. 1990. Institutions, Institutional Change and Economic Performance. New York: Cambridge University Press.

Paine, Drew, and Charlotte P. Lee. 2020. “Coordinative Entities: Forms of Organizing in Data Intensive Science.” Computer Supported Cooperative Work (CSCW) 29 (3): 335–80.

Penders, B., Holbrook, J.B, & de Rijcke, S. 2019. “Rinse and Repeat: Understanding the Value of Replication across Different Ways of Knowing.” Publications 7 (3): 52.

Potter, Gerald L., and W. Lawrence Gates. 1984. “A Preliminary Intercomparison of the Seasonal Response of Two Atmospheric Climate Models.” Monthly Weather Review 112 (5): 909–17.<0909:apiots>;2

Randall, David A., et al. 2007. “Climate models and their evaluation.” In: Climate Change 2007: The Physical Science Basis. Contribution of Working Group I to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change. New York: Cambridge University Press.

Ribes, David and Steven J. Jackson. 2013. “Data bite man: The work of sustaining a long-term study.” In L. Gitelman (Ed.), “Raw Data" is an Oxymoron, 147-166. Cambridge, MA: MIT Press.

Rood, Richard B. 2019. “Validation of Climate Models: An Essential Practice.” In Simulation Foundations, Methods and Applications, 737–62. Springer International Publishing.

Schmidt, Gavin A., and Steven Sherwood. 2014. “A Practical Philosophy of Complex Climate Modelling.” European Journal for Philosophy of Science 5 (2): 149–69.

Shapin, Steven. 1995. “Cordelia's love: Credibility and the Social Studies of Science.” Perspectives on Science 3 (3): 255-275.

Shapin, Steven and Simon Schaffer. 1985. Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life. Princeton, NJ: Princeton University Press.

Somerville, Richard C.J. 2011. “The co-evolution of climate models and the Intergovernmental Panel on Climate Change.” In Donner, L., Schubert, W., & Somerville, R. (Eds.), The Development of Atmospheric General Circulation Models: Complexity, Synthesis, and Computation, 225-252. New York: Cambridge University Press

Star, Susan Leigh, and Karen Ruhleder. 1996. “Steps Toward an Ecology of Infrastructure: Design and Access for Large Information Spaces.” Information Systems Research 7 (1): 111–34.

Stockhause, Martina, H. Höck, F. Toussaint, and M. Lautenschlager. 2012. “Quality Assessment Concept of the World Data Center for Climate and Its Application to CMIP5 Data.” Geoscientific Model Development 5 (4): 1023–32.

Stodden, Victoria, and Sheila Miguez. 2014. “Best Practices for Computational Science: Software Infrastructure and Environments for Reproducible and Extensible Research.” Journal of Open Research Software 2 (1).

Sundberg, Mikaela. 2010. “The Dynamics of Coordinated Comparisons: How Simulationists in Astrophysics, Oceanography and Meteorology Create Standards for Results.” Social Studies of Science 41 (1): 107–25.

Taylor, Karl. 2013. CMIP5 Standard Output. Livermore, CA: Program for Climate Model Diagnosis & Intercomparison (PCMDI).

Taylor, Karl. E., Doutriaux, C, and Peterschmitt, J.Y. 2006. Climate Model Output Rewriter (CMOR). Livermore, CA: Program for Climate Model Diagnosis & Intercomparison (PCMDI).

Voorhees, E.M. & Harman, D.K. 2005. “The Text REtrieval Conference.” In E.M. Voorhees & D.K. Harman (Eds), TREC: Experiment and Evaluation in Information Retrieval, 3-19. Cambridge, MA: MIT Press.

Walsh, Virginia M. 2004. Global Institutions and Social Knowledge: Generating Research at the Scripps Institution and the Inter-American Tropical Tuna Commission, 1900s-1990s. Cambridge, MA: MIT Press.

[WCRP] World Climate Research Program. 2014. Application for CMIP6-Endorsed MIPs. World Climate Research Programme.

[WCRP] World Climate Research Program. 2019. Report of the 22nd Session of the Working Group on Coupled Modeling: 25th and 29th March 2019, Barcelona, Spain. WCRP Publication No.14/2019. World Climate Research Program.

[WIP] WGCM Infrastructure Panel. 2014. Terms of Reference for the WGCM Infrastructure Panel (WIP). World Climate Research Programme.

Williams, Dean N. 2012. Earth System Grid Federation (ESGF): Future and Governance. World Climate Research Program.

Winsberg, Eric. 2018. Philosophy and Climate Science. New York: Cambridge University Press.

Woolgar, Steve. 1981. “Critique and Criticism: Two Readings of Ethnomethodology.” Social Studies of Science 11 (4): 504–14.

Wylie, Alison. 2016. “How Archaeological Evidence Bites Back.” Science, Technology, & Human Values 42 (2): 203–25.

Yan, An, Caihong Huang, Jian‐Sin Lee, and Carole L. Palmer. 2020. “Cross‐disciplinary Data Practices in Earth System Science: Aligning Services with Reuse and Reproducibility Priorities.” Proceedings of the Association for Information Science and Technology 57 (1): e218.

Young, Oran R., Paul Arthur Berkman, and Alexander N. Vylegzhanin. 2020. “Informed Decisionmaking for the Sustainability of Ecopolitical Regions.” In Informed Decisionmaking for Sustainability, 341–53.

21 Dec 2021
Research Articles