Credibility via Coupling

Institutions and Infrastructures in Climate Model Intercomparisons

  • Matthew S. Mayernik National Center for Atmospheric Research

Abstract

This study investigates Model Intercomparison Projects (MIPs) as one example of a coordinated approach to establishing scientific credibility. MIPs originated within climate science as a method to evaluate and compare disparate climate models, but MIPs or MIP-like projects are now spreading to many scientific fields. Within climate science, MIPs have advanced knowledge of: a) the climate phenomena being modeled, and b) the building of climate models themselves. MIPs thus build scientific confidence in the climate modeling enterprise writ large, reducing questions of the credibility or reproducibility of any single model. This paper will discuss how MIPs organize people, models, and data through institution and infrastructure coupling (IIC). IIC involves establishing mechanisms and technologies for collecting, distributing, and comparing data and models (infrastructural work), alongside corresponding governance structures, rules of participation, and collaboration mechanisms that enable partners around the world to work together effectively (institutional work). Coupling these efforts involves developing formal and informal ways to standardize data and metadata, create common vocabularies, provide uniform tools and methods for evaluating resulting data, and build community around shared research topics.

References

Ankeny, Rachel A., and Sabina Leonelli. 2011. “What’s so Special about Model Organisms?” Studies in History and Philosophy of Science Part A 42 (2): 313–23. https://doi.org/10.1016/j.shpsa.2010.11.039

Ankeny, Rachel A., and Sabina Leonelli. 2016. “Repertoires: A Post-Kuhnian Perspective on Scientific Change and Collaborative Research.” Studies in History and Philosophy of Science Part A 60 (December): 18–28. https://doi.org/10.1016/j.shpsa.2016.08.003

Aula, Ville. 2019. “Institutions, Infrastructures, and Data Friction – Reforming Secondary Use of Health Data in Finland.” Big Data & Society 6 (2): 205395171987598. https://doi.org/10.1177/2053951719875980

Balaji, V., Karl E. Taylor, Martin Juckes, Bryan N. Lawrence, Paul J. Durack, Michael Lautenschlager, Chris Blanton, et al. 2018. “Requirements for a Global Data Infrastructure in Support of CMIP6.” Geoscientific Model Development 11 (9): 3659–80. https://doi.org/10.5194/gmd-11-3659-2018

Bik, Elisabeth M., Ferric C. Fang, Amy L. Kullas, Roger J. Davis, and Arturo Casadevall. 2018. “Analysis and Correction of Inappropriate Image Duplication: The Molecular and Cellular Biology Experience.” Molecular and Cellular Biology 38 (20). https://doi.org/10.1128/mcb.00309-18

Borgman, Christine L. 2012. “The Conundrum of Sharing Research Data.” Journal of the American Society for Information Science and Technology 63 (6): 1059–78. https://doi.org/10.1002/asi.22634

Braun, Kathrin, and Cordula Kropp. 2010. “Beyond Speaking Truth? Institutional Responses to Uncertainty in Scientific Governance.” Science, Technology, & Human Values 35 (6): 771–82. https://doi.org/10.1177/0162243909357916

Bush, Rosemary T. 2018. Perspectives on Reproducibility and Replication of Results in Climate Science. Paper commissioned for the National Academies of Sciences, Engineering, and Medicine Committee on Reproducibility and Replicability in Science. https://www.nap.edu/resource/25303/Reproducibility%20and%20Replication%20in%20Climate%20Science.pdf

Callaghan, Sarah A., M. Elkington, R. Ford, M. Kolaninski, M. Lautenschlager, B. Lawrence, L. Steenman-Clark, et al. 2010. “The METAFOR Project: Preserving data through metadata standards for climate models and simulations.” In Proceedings of the 1st International Digital Preservation Interoperability Framework Symposium on - INTL-DPIF ’10. ACM Press, 1-8. https://doi.org/10.1145/2039263.2039269

Cess, R. D., Potter, G. L., Blanchet, J. P., Boer, G. J., Ghan, S. J., Kiehl, J. T., … Yagai, I. 1989. “Interpretation of Cloud-Climate Feedback as Produced by 14 Atmospheric General Circulation Models.” Science 245 (4917): 513–16. https://doi.org/10.1126/science.245.4917.513

Christopher, Jana. 2018. “Systematic Fabrication of Scientific Images Revealed.” FEBS Letters 592 (18): 3027–29. https://doi.org/10.1002/1873-3468.13201

Cinquini, L., Crichton, D., Mattmann, C., Harney, J., Shipman, G., Wang, F., … Schweitzer, R. 2014. “The Earth System Grid Federation: An Open Infrastructure for Access to Distributed Geospatial Data.” Future Generation Computer Systems 36 (July): 400–417. https://doi.org/10.1016/j.future.2013.07.002

Collins, Harry. 1985. Changing Order: Replication and Induction in Scientific Practice; University of Chicago Press: Chicago, IL.

Collins, Harry. 2013. Gravity’s Ghost and Big Dog: Scientific Discovery and Social Analysis in the Twenty-First Century. Chicago, IL: University of Chicago Press.

Darch, Peter T. 2018. “Limits to the Pursuit of Reproducibility: Emergent Data-Scarce Domains of Science.” In Transforming Digital Worlds, 164–74. Springer International Publishing. https://doi.org/10.1007/978-3-319-78105-1_21

Deser, Clara, F. Lehner, K. B. Rodgers, T. Ault, T. L. Delworth, P. N. DiNezio, A. Fiore, et al. 2020. “Insights from Earth System Model Initial-Condition Large Ensembles and Future Prospects.” Nature Climate Change 10 (4): 277–86. https://doi.org/10.1038/s41558-020-0731-2

Easterbrook, Steve M. 2014. “Open Code for Open Science?” Nature Geoscience 7 (11): 779–81. https://doi.org/10.1038/ngeo2283

Edwards, Paul N. 2010. A Vast Machine: Computer Models, Climate Data, and the Politics of Global Warming. Cambridge, MA: MIT Press.

Edwards, Paul N. 2016. “Knowledge Infrastructures for the Anthropocene.” The Anthropocene Review 4 (1): 34–43. https://doi.org/10.1177/2053019616679854

Edwards, Paul N. 2019a. “Infrastructuration: On Habits, Norms and Routines as Elements of Infrastructure*.” In M. Kornberger et al (Eds.), Thinking Infrastructures, 355–66. Emerald Publishing Limited. https://doi.org/10.1108/s0733-558x20190000062022.

Edwards, Paul N. 2019b. “Knowledge infrastructures under siege: Climate data as memory, truce, and target.” In D. Bigo, E. Isin, E. Ruppert (Eds.), Data Politics: Worlds, Subjects, Rights, 21-42. New York: Routledge. https://doi.org/10.4324/9781315167305-2

Edwards, Paul N., Jackson, S.J., Bowker, G.C., & Knobel, C.P. 2007. Understanding Infrastructure: Dynamics, Tensions, and Design. Ann Arbor, MI: University of Michigan. http://deepblue.lib.umich.edu/handle/2027.42/49353

Fanelli, Daniele. 2018. “Opinion: Is Science Really Facing a Reproducibility Crisis, and Do We Need It To?” Proceedings of the National Academy of Sciences 115 (11): 2628–31. https://doi.org/10.1073/pnas.1708272114

Feinberg, Melanie, Will Sutherland, Sarah Beth Nelson, Mohammad Hossein Jarrahi, and Arcot Rajasekar. 2020. “The New Reality of Reproducibility: The Role of Data Work in Scientific Research.” Proceedings of the ACM on Human-Computer Interaction 4 (CSCW1): 1–22. https://doi.org/10.1145/3392840

Galison, Peter. 1987. How Experiments End. Chicago, IL: University of Chicago Press.

Gates, W. Lawrence. (Ed.) 1979. Report of the JOC Study Conference on Climate Models: Performance, Intercomparison and Sensitivity Studies, Volume I. Global Atmospheric Research Programme (GARP) Publications Series No. 22.

Gates, W. Lawrence. 1992. “AMIP: The Atmospheric Model Intercomparison Project.” Bulletin of the American Meteorological Society 73 (12): 1962–70. https://doi.org/10.1175/1520-0477(1992)073<1962:atamip>2.0.co;2

Gates, W.L. 1995. “An overview of AMIP and preliminary results.” In Proceedings of the First International AMIP Scientific Conference. World Climate Research Programme, WCRP-92, WMO/TD-No. 732, pp. 1-8.

Gates, W. Lawrence, James S. Boyle, Curt Covey, Clyde G. Dease, Charles M. Doutriaux, Robert S. Drach, Michael Fiorino, et al. 1999. “An Overview of the Results of the Atmospheric Model Intercomparison Project (AMIP I).” Bulletin of the American Meteorological Society 80 (1): 29–55. https://doi.org/10.1175/1520-0477(1999)080<0029:aootro>2.0.co;2

Guilyardi, Eric, Balaji, V., Callaghan, S., DeLuca, C., Devine, G., Denvil, S., Ford, R., et al. 2011. “The CMIP5 model and simulation documentation: a new standard for climate modelling metadata.” CLIVAR Exchanges 56: 16 (2). pp. 42-46. http://centaur.reading.ac.uk/25733/

Harman, Donna K. 2005. “The TREC test collections.” In E.M. Voorhees & D.K. Harman (Eds), TREC: Experiment and Evaluation in Information Retrieval, 21-52. Cambridge, MA: MIT Press.

Harman, Donna K., and Ellen M. Voorhees. 2007. “TREC: An Overview.” Annual Review of Information Science and Technology 40 (1): 113–55. https://doi.org/10.1002/aris.1440400111

Harter, Stephen P. 1996. “Variations in Relevance Assessments and the Measurement of Retrieval Effectiveness.” Journal of the American Society for Information Science 47 (1): 37–49. https://doi.org/10.1002/(sici)1097-4571(199601)47:1<37::aid-asi4>3.0.co;2-3

Hoeppe, Götz. 2014. “Working Data Together: The Accountability and Reflexivity of Digital Astronomical Practice.” Social Studies of Science 44 (2): 243–70. https://doi.org/10.1177/0306312713509705

Howe, Cymene, Jessica Lockrem, Hannah Appel, Edward Hackett, Dominic Boyer, Randal Hall, Matthew Schneider-Mayerson, et al. 2015. “Paradoxical Infrastructures.” Science, Technology, & Human Values 41 (3): 547–65. https://doi.org/10.1177/0162243915620017

Jasanoff, Sheila. 2004. Ordering knowledge, ordering society. In S. Jasanoff (Ed), States of Knowledge: The Co-Production of Science and Social Order, 13-45. New York: Routledge.

Jebeile, Julie, and Michel Crucifix. 2020. “Multi-Model Ensembles in Climate Science: Mathematical Structures and Expert Judgements.” Studies in History and Philosophy of Science Part A 83 (October): 44–52. https://doi.org/10.1016/j.shpsa.2020.03.001

Jefferson, Gail. 1985. “On the Interactional Unpackaging of a ‘Gloss.’” Language in Society 14 (4): 435–66. https://doi.org/10.1017/s0047404500011465

Jensen, Casper Bruun, and Atsuro Morita. 2015. “Infrastructures as Ontological Experiments.” Engaging Science, Technology, and Society 1: 81-87. https://doi.org/10.17351/ests2015.21

Jepperson, R.L. 1991. “Institutions, institutional effects, and institutionalism.” In W.W. Powell & P.J. DiMaggio (Eds.), The New Institutionalism in Organizational Analysis, 143-163. Chicago, IL: University of Chicago Press.

Jewett, Tom, and Rob Kling. 1991. “The Dynamics of Computerization in a Social Science Research Team: A Case Study of Infrastructure, Strategies, and Skills.” Social Science Computer Review 9 (2): 246–75. https://doi.org/10.1177/089443939100900205

Kaiser, Jocelyn. 2003. “Ecological Networks: Panel Suggests a Different Shade of NEON.” Science 301 (5641): 1828a–11828. https://doi.org/10.1126/science.301.5641.1828a

Kennefick, Daniel. 2007. Traveling at the Speed of Thought: Einstein and the Quest for Gravitational Waves. Princeton, NJ: Princeton University Press.

Latour, Bruno. 2013. An Inquiry into Modes of Existence: An Anthropology of the Moderns. Trans. Catherine Porter. Cambridge, MA: Harvard University Press.

Lee, Charlotte P., and Kjeld Schmidt. 2018. “A Bridge Too Far? Critical remarks on the concept of ‘infrastructure’ in CSCW and IS.” In V. Wulf, et al.(eds): Socio-Informatics: A Practice-based Perspective on the Design and Use of IT Artifacts. Oxford: Oxford University Press, pp. 177-217. Oxford: Oxford University Press. https://doi.org/10.1093/oso/9780198733249.003.0006

Leonelli, Sabina. 2018. “Rethinking Reproducibility as a Criterion for Research Quality.” In Including a Symposium on Mary Morgan: Curiosity, Imagination, and Surprise, 129–46. Emerald Publishing Limited. https://doi.org/10.1108/s0743-41542018000036b009

Leonelli, Sabina. 2019. “Scientific agency and social scaffolding in contemporary data-intensive biology.” In: Wimsatt, W. and Love, A. (Eds) Beyond the Meme: Articulating Dynamic Structures in Cultural Evolution, 42-63. Minneapolis: University of Minnesota Press.

Loescher, H.W., Kelly, E.F., and Lea, R. 2017. “National Ecological Observatory Network: Beginnings, programmatic and scientific challenges, and ecological forecasting.” In A. Chabbi & H.W. Loescher (Eds), Terrestrial Ecosystem Research Infrastructures, 27-51. Boca Raton, FL: CRC Press.

Martin, P., Chen, Y., Hardisty, A., Jeffery, K., and Zhao, Z. 2017. “Computational challenges in global environmental research infrastructures.” In A. Chabbi & H.W. Loescher (Eds), Terrestrial Ecosystem Research Infrastructures, 305-340. Boca Raton, FL: CRC Press.

Mayernik, Matthew S. 2016. “Research Data and Metadata Curation as Institutional Issues.” Journal of the Association for Information Science and Technology 67 (4): 973–93. https://doi.org/10.1002/asi.23425

Mayernik, Matthew S. 2019. “Metadata Accounts: Achieving Data and Evidence in Scientific Research.” Social Studies of Science 49 (5): 732–57. https://doi.org/10.1177/0306312719863494

Meehl, Gerald A., George J. Boer, Curt Covey, Mojib Latif, and Ronald J. Stouffer. 1997. “Intercomparison Makes for a Better Climate Model.” Eos, Transactions American Geophysical Union 78 (41): 445. https://doi.org/10.1029/97eo00276

Meehl, Gerald A., Curt Covey, Thomas Delworth, Mojib Latif, Bryant McAvaney, John F. B. Mitchell, Ronald J. Stouffer, and Karl E. Taylor. 2007. “THE WCRP CMIP3 Multimodel Dataset: A New Era in Climate Change Research.” Bulletin of the American Meteorological Society 88 (9): 1383–94. https://doi.org/10.1175/bams-88-9-1383

Mervis, J. 2015. “NSF Looks for New Contractor to Finish Troubled Observatory.” Science 350 (6267): 1454. https://doi.org/10.1126/science.350.6267.1454

Mervis, Jeffrey. 2019. “Shake-up Threatens Novel U.S. Ecology Facility.” Science 363 (6424): 211–12. https://doi.org/10.1126/science.363.6424.211

Miller, Clark. 2001. “Hybrid Management: Boundary Organizations, Science Policy, and Environmental Governance in the Climate Regime.” Science, Technology, & Human Values 26 (4): 478–500. https://doi.org/10.1177/016224390102600405

Miller, Seumas. 2011. “Social Institutions.” In Stanford Encyclopedia of Philosophy. Stanford, CA: Stanford University. http://plato.stanford.edu/entries/social-institutions/

Miyakawa, Tsuyoshi. 2020. “No Raw Data, No Science: Another Possible Source of the Reproducibility Crisis.” Molecular Brain 13 (1). https://doi.org/10.1186/s13041-020-0552-2

Moylan, Elizabeth C, and Maria K Kowalczuk. 2016. “Why Articles Are Retracted: A Retrospective Cross-Sectional Study of Retraction Notices at BioMed Central.” BMJ Open 6 (11): e012047. https://doi.org/10.1136/bmjopen-2016-012047

[NASEM] National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science. Washington, DC: The National Academies Press. https://doi.org/10.17226/25303

North, Douglas C. 1990. Institutions, Institutional Change and Economic Performance. New York: Cambridge University Press.

Paine, Drew, and Charlotte P. Lee. 2020. “Coordinative Entities: Forms of Organizing in Data Intensive Science.” Computer Supported Cooperative Work (CSCW) 29 (3): 335–80. https://doi.org/10.1007/s10606-020-09372-2.

Penders, B., Holbrook, J.B, & de Rijcke, S. 2019. “Rinse and Repeat: Understanding the Value of Replication across Different Ways of Knowing.” Publications 7 (3): 52. https://doi.org/10.3390/publications7030052

Potter, Gerald L., and W. Lawrence Gates. 1984. “A Preliminary Intercomparison of the Seasonal Response of Two Atmospheric Climate Models.” Monthly Weather Review 112 (5): 909–17. https://doi.org/10.1175/1520-0493(1984)112<0909:apiots>2.0.co;2

Randall, David A., et al. 2007. “Climate models and their evaluation.” In: Climate Change 2007: The Physical Science Basis. Contribution of Working Group I to the Fourth Assessment Report of the Intergovernmental Panel on Climate Change. New York: Cambridge University Press.

Ribes, David and Steven J. Jackson. 2013. “Data bite man: The work of sustaining a long-term study.” In L. Gitelman (Ed.), “Raw Data" is an Oxymoron, 147-166. Cambridge, MA: MIT Press.

Rood, Richard B. 2019. “Validation of Climate Models: An Essential Practice.” In Simulation Foundations, Methods and Applications, 737–62. Springer International Publishing. https://doi.org/10.1007/978-3-319-70766-2_30

Schmidt, Gavin A., and Steven Sherwood. 2014. “A Practical Philosophy of Complex Climate Modelling.” European Journal for Philosophy of Science 5 (2): 149–69. https://doi.org/10.1007/s13194-014-0102-9

Shapin, Steven. 1995. “Cordelia's love: Credibility and the Social Studies of Science.” Perspectives on Science 3 (3): 255-275. http://nrs.harvard.edu/urn-3:HUL.InstRepos:3293019

Shapin, Steven and Simon Schaffer. 1985. Leviathan and the Air-Pump: Hobbes, Boyle, and the Experimental Life. Princeton, NJ: Princeton University Press.

Somerville, Richard C.J. 2011. “The co-evolution of climate models and the Intergovernmental Panel on Climate Change.” In Donner, L., Schubert, W., & Somerville, R. (Eds.), The Development of Atmospheric General Circulation Models: Complexity, Synthesis, and Computation, 225-252. New York: Cambridge University Press

Star, Susan Leigh, and Karen Ruhleder. 1996. “Steps Toward an Ecology of Infrastructure: Design and Access for Large Information Spaces.” Information Systems Research 7 (1): 111–34. https://doi.org/10.1287/isre.7.1.111

Stockhause, Martina, H. Höck, F. Toussaint, and M. Lautenschlager. 2012. “Quality Assessment Concept of the World Data Center for Climate and Its Application to CMIP5 Data.” Geoscientific Model Development 5 (4): 1023–32. https://doi.org/10.5194/gmd-5-1023-2012

Stodden, Victoria, and Sheila Miguez. 2014. “Best Practices for Computational Science: Software Infrastructure and Environments for Reproducible and Extensible Research.” Journal of Open Research Software 2 (1). https://doi.org/10.5334/jors.ay

Sundberg, Mikaela. 2010. “The Dynamics of Coordinated Comparisons: How Simulationists in Astrophysics, Oceanography and Meteorology Create Standards for Results.” Social Studies of Science 41 (1): 107–25. https://doi.org/10.1177/0306312710385743

Taylor, Karl. 2013. CMIP5 Standard Output. Livermore, CA: Program for Climate Model Diagnosis & Intercomparison (PCMDI). https://pcmdi.llnl.gov/mips/cmip5/docs/standard_output.pdf?id=4

Taylor, Karl. E., Doutriaux, C, and Peterschmitt, J.Y. 2006. Climate Model Output Rewriter (CMOR). Livermore, CA: Program for Climate Model Diagnosis & Intercomparison (PCMDI). https://pcmdi.github.io/cmor-site/media/pdf/cmor_users_guide.pdf

Voorhees, E.M. & Harman, D.K. 2005. “The Text REtrieval Conference.” In E.M. Voorhees & D.K. Harman (Eds), TREC: Experiment and Evaluation in Information Retrieval, 3-19. Cambridge, MA: MIT Press.

Walsh, Virginia M. 2004. Global Institutions and Social Knowledge: Generating Research at the Scripps Institution and the Inter-American Tropical Tuna Commission, 1900s-1990s. Cambridge, MA: MIT Press.

[WCRP] World Climate Research Program. 2014. Application for CMIP6-Endorsed MIPs. World Climate Research Programme. https://www.wcrp-climate.org/modelling-wgcm-mip-catalogue/modelling-wgcm-cmip6-endorsed-mips

[WCRP] World Climate Research Program. 2019. Report of the 22nd Session of the Working Group on Coupled Modeling: 25th and 29th March 2019, Barcelona, Spain. WCRP Publication No.14/2019. World Climate Research Program. https://www.wcrp-climate.org/WCRP-publications/2019/WCRP-Report-No14-2019-WGCM22.pdf

[WIP] WGCM Infrastructure Panel. 2014. Terms of Reference for the WGCM Infrastructure Panel (WIP). World Climate Research Programme. https://www.earthsystemcog.org/site_media/projects/wip/WIP_Terms_of_Reference.pdf

Williams, Dean N. 2012. Earth System Grid Federation (ESGF): Future and Governance. World Climate Research Program. https://cordex2013.wcrp-climate.org/WDAC/documents/ESGF_FutureAndGovernance.pdf

Winsberg, Eric. 2018. Philosophy and Climate Science. New York: Cambridge University Press.

Woolgar, Steve. 1981. “Critique and Criticism: Two Readings of Ethnomethodology.” Social Studies of Science 11 (4): 504–14. https://doi.org/10.1177/030631278101100406

Wylie, Alison. 2016. “How Archaeological Evidence Bites Back.” Science, Technology, & Human Values 42 (2): 203–25. https://doi.org/10.1177/0162243916671200

Yan, An, Caihong Huang, Jian‐Sin Lee, and Carole L. Palmer. 2020. “Cross‐disciplinary Data Practices in Earth System Science: Aligning Services with Reuse and Reproducibility Priorities.” Proceedings of the Association for Information Science and Technology 57 (1): e218. https://doi.org/10.1002/pra2.218

Young, Oran R., Paul Arthur Berkman, and Alexander N. Vylegzhanin. 2020. “Informed Decisionmaking for the Sustainability of Ecopolitical Regions.” In Informed Decisionmaking for Sustainability, 341–53.

Published
21 Dec 2021
Section
Research Articles