Research Reproducibility Activities in Health Sciences Libraries

Introduction

Over the past decade, there has been increased emphasis on research reproducibility (RR) in the health sciences. This is part of a broad trend toward more open research, which has been fortified by policy statements from the NIH and other U.S. government agencies (National Institutes of Health 2015; National Science Foundation 2014; Collins and Tabak 2014).

In response, health sciences librarians have developed services to help researchers adapt to the more open research environment. The authors of this study, for example, are increasingly providing research consultations and instruction sessions on RR-related concepts to local health sciences researchers. This is a relatively recent development and served in large part as motivation for the present study. We aimed to understand how medical and health sciences librarians engage with RR at their institutions. Specifically, we wanted to know:

• What services and activities are medical and health sciences librarians developing to engage with researchers on RR topics?

• What will be the impact of RR on medical and health sciences libraries in the near future?

Our understanding of RR was informed by the National Academies of Sciences, Engineering, and Medicine's (NASEM's) definition, which is "obtaining consistent results using the same input data; computational steps, methods, and code; and conditions of analysis." This definition underlies our methodology and analysis. We did not explore 'replicability' in much depth, but since it is frequently interlinked with RR, it periodically surfaced during our study. For replicability, we relied on NASEM once again: "Replicability means obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data" (National Academies of Sciences, Engineering, and Medicine 2019).

Literature Review

In a recent scoping review, Giustini et al found 54 studies that describe health sciences library engagement in one or more open science domains (Guistini et al. 2021). More than half of the included studies were published in or after 2018. Moreover, RR was a core principle in the study's definition of open science and explicitly referenced in at least three of the included studies. In other words, RR was both an assumed part of other services and a featured service.

This reflects the varied nature of RR. RR covers a range of methods and processes, often prefaced with words like open and data , and specific resources that help make those methods and processes possible (such as e-notebooks, Open Refine, and GitHub). In libraries, we see RR frequently associated with data management (Flores et al. 2015; Yoon and Schultz 2017; Kim and Syn 2021; Tenopir et al. 2015; Vitale 2016). In Murray et al, survey respondents placed RR within the context of research data services (Murray et al. 2019). Steeves highlighted RR in a specialized librarian position and placed it within the context of data management (Steeves 2017).

Even when RR is at the forefront, it is linked directly to other open and data services. Sayre and Riegelman wrote about the reproducibility crisis in academic libraries, and defined a service model, with recommendations, for librarians to support RR (Sayre and Riegelman 2018; Sayre and Riegelman 2019). The type of support was defined by concepts like data management, data sharing, and data repositories, which are all frequently referenced alongside RR. Rethlefsen describes a research reproducibility conference (Rethlefsen, Lackey, and Zhao 2018) that provides a formal venue to discuss the interrelated network of open and data concepts we see in Guistini et al. and Murray et al. Similarly, LaPolla et al. describe educational efforts at five institutions in which librarians are heavily involved in the teaching of RR practices, and these practices tend to be defined by those recurring concepts (LaPolla et al. 2022).

Methods

We conducted semi-structured interviews with medical and health sciences librarians from different institutions across the United States who were engaged in RR work. The interviews were conducted over Zoom between 26 April 2021 and 14 July 2021, and each lasted about an hour. One author (MM) led the interview while the other author (SS) took detailed notes. All documents related to the study, including the interview guide, interview questions, coding schema, and verified quotes are available at DOI 10.17605/OSF.IO/UAFH8.

Population and recruitment: We identified health sciences and medical librarians currently engaged with RR in the United States through a review of recent published literature, presentations, webinars, library websites, and online research guides. Participants were invited directly through email, and non-respondents were sent no more than two follow-up emails. We protected participant identity throughout the study since we asked questions about workplaces that could raise sensitive situations or associations. Participants did not receive any compensation.

Instrument: We used a literature review, internal discussion, discussion with colleagues, and one pilot interview to create and refine a list of questions which were grouped into six discussion topics. The six discussion topics were:

• About research reproducibility

• Research reproducibility services and work at your institution

• Institutional context and campus partnerships

• Assessment of research reproducibility services (if any)

• Vision of research reproducibility work in libraries

• Recommendations to other librarians who are starting on this path

Upon a participant’s agreement to participate, we shared these topics via email, along with other interview logistics.

Data collection: At the start of each interview, we reviewed information about the study and confirmed the participant’s willingness to be interviewed and recorded, with the understanding that the recordings would be used to verify notes and quotes before being deleted. We conducted semi-structured interviews with the participants to gather information about their RR activities. We adhered to our predetermined topics and asked follow-up questions as needed. We took detailed notes during the interview to capture participant responses. Prior to analyzing the data, we refined the notes by spelling out abbreviations, filling in unclear details by consulting the recordings, and removing mention of specific institution names and colleagues.

Data analysis: We used template analysis to analyze our data. The initial template, or set of codes, was informed by the discussion topics used to conduct the interviews. Both authors reviewed the notes independently, noted any additional codes from emerging themes, then came to agreement on the final set of codes. Both authors then used NVivo (release 1.5 on Mac) to code the interview notes documents and aggregate relevant information from each interview. The coded outputs from each author were combined to provide a full picture of the information gathered for each code. Following data analysis, we pulled key excerpts from the notes for each theme, and revisited the recordings to transcribe the excerpts into verified quotes. Each participant’s verified quotes were shared back with them, and participants confirmed that the quotes could be anonymously included in the manuscript.

Institutional Review Board: The University of Michigan Health Sciences and Behavioral Sciences Institutional Review Board determined that the study was exempt from on-going oversight (HUM00196119).

Results

We contacted 15 librarians. Eleven agreed to be interviewed. All 11 were directly involved in RR, either through leadership responsibilities or engagement with researchers. Of these 11, seven had leadership titles, such as associate dean, director, or manager. Regarding the 10 institutions with publicly available NIH extramural award funding (National Institutes of Health n.d.), the average amount was over $313 million for fiscal year 2021, and the median was just over $221 million. To protect the identities of our participants, we chose not to report specific job titles or institutional affiliations beyond this summary.

We were unable to interview four of our initial contacts due to: unavailability (1), non-responsiveness after initial contact (1), job change (1), and referral to a colleague (1). One participant requested that their transcribed quotes be excluded from publication. Their responses are included in the aggregated results.

The sections below summarize the themes identified in our data. We arrived at these themes after careful analysis of our interview data. Each section includes select participant quotes that both substantiate the summary and give voice to our study participants.

Definition of research reproducibility

We asked each participant to define RR at the start of the interview. All participants defined it in their own way, but ultimately agreed that RR refers to a researcher's ability to conduct the same research by using available data and documentation. Every participant acknowledged that there is a difference between reproducibility and replicability, but about half indicated that the difference didn’t matter in their work. We didn't dictate what type of work constituted RR, but instead framed questions around RR and let the participants interpret what that meant in their context.

"I think it's not necessarily about prioritizing research reproducibility so much as it is aligning what we already do with reproducibility, which is really something that our institutions need to be thinking about and supporting."

Framing of RR within the library service structure

While the participants were clearly doing RR work, they seldom used RR terminology to describe or promote that work. In their experience, researchers were not approaching the library with questions framed as RR questions. Rather, the librarian would integrate RR concepts into conversations as the opportunity arose. This was also the case with librarian-led instruction sessions. These sessions might include RR content but wouldn't necessarily be marketed as such. Yet there were contexts in which RR terminology was prominent, such as in instruction sessions for graduate students and in discussions with researchers whose interest in RR aligned with funder statements or requirements.

"It would be very rare that somebody would frame their own question as a reproducibility question."

"I think the conversation about reproducibility with people who are currently looking for or seeking NIH funding is a really different conversation about reproducibility then a broader foundation of reproducibility, like when we're working with early graduate students and other groups."

"For the workshops that I teach, I generally don't mention research reproducibility in those terms. I focus on more specific practices for researchers. So things like using version control for your projects."

Drivers of librarian involvement in RR initiatives

Motivations to pursue RR were driven by both internal and external factors. Every participant agreed that libraries were suitable hubs for supporting and advocating for RR, and that librarians have unique expertise to contribute to RR efforts. In fact, it was noted that librarians have a long history of sharing, organizing, managing, and curating knowledge, which connects key components of RR with core library values.

"I think we really have a unique skill set that very much complements what researchers are doing. Librarians and information professionals in general are trained in working with information, and data is a type of information."

"I think of the library as a leader or librarians as leaders in our institutions to be able to help facilitate change and drive forward change and hopefully improve clinical practice and research and education. And I think this is one of the ways that we can do that - is by aligning some of our efforts with some of these national and international…trends."

"I think libraries in general can be leaders in promoting better research culture, which includes reproducibility and research integrity and open science."

Most participants mentioned institutional priorities as a motivation for their own RR planning. If the institution recognized RR as important, then the library had reason to prioritize it. The challenge, then, was to establish appropriate services and make optimal use of limited resources. But we also heard the opposite. Some participants described an institutional context in which the library was leading the push for RR at the institution and helping define RR as an institutional priority.

"Also, 'we don't want to be on the front page of the New York Times because of research misconduct.' I think that's one of the drivers - as mitigating risk from a school perspective."

"[Research reproducibility is] an area we can invest in. And there are so many areas to invest in. So, really, then it ends up being - how important is it on campus. Is it a named issue on this campus?"

"I don't think that libraries can honestly be completely objective in terms of doing their own thing without considering the academic and university ecosystem that they're a part of. We have to think about the goals of the libraries in alignment with the goals of the broader institution’s and social systems that we’re embedded in."

"What is it that people are asking for and interested in? And then how can you build your other services that you think are really important off of that?"

"At [redacted], it had to be a little bit more of a grassroots effort. And they had all sorts of grassroots efforts going on there as well, but they just…this institution was so massive it just wasn't really getting anywhere as fast."

The National Institutes of Health (NIH) was noted as an important external entity for driving interest in RR. If NIH emphasizes RR in its policies, then researchers with NIH funding are more likely to take interest.

"[I]s this something that the NIH (because almost all of our folks are NIH funded), is this something that they're pushing?"

RR services in health sciences libraries

All participants worked at libraries with some type of RR services. Rarely were these services described with RR terminology. The structure of responsibility for these services varied across the institutions, but most services were handled by designated experts or small teams that specialized in data services. These individuals and teams tended to handle all RR-related questions, broadly defined, that came into the library. These librarians share similar job titles, such as data services librarian. Some data services teams include individuals with PhDs instead of library degrees. The most frequently mentioned services were workshops, courses, and consultations.

"I don't necessarily think all subject liaisons or reference librarians can be all the data things either. … In my library, we have this pretty discrete boundary between the two."

"We don't have a class on research reproducibility or anything of that nature, so I feel like what we do offer is ingrained in a lot of the classes that we do. … Some of the classes that we teach are using Git and talking about how to document your code and how to do versioning of your code. When we're doing our data management classes, we're just ingraining those concepts in there but we don't call out a whole class on it."

Interestingly, participants referenced a range of RR resources and concepts when describing their RR services. We didn't ask participants to comprehensively list RR resources and concepts from their work, but the ones mentioned help illustrate the range and type of activity that falls under the umbrella of RR library services (see Table 1).

Table 1 : RR resources and concepts mentioned during the interviews, in alphabetical order.

Institutional context, including partnerships

The participants worked at research institutions with researchers and research infrastructure, which meant they had individuals and departments to collaborate with. Participants leveraged collaborations for funding and technology infrastructure, and making connections with other institutional stakeholders. These collaborations allowed participants to specialize where there was an institutional need and combine their expertise with external individuals and departments.

The participants viewed institutional partners as essential to their RR work. They spoke knowledgeably about potential partners and were aware of how their services fit into the bigger institutional context. They expressed concern about overlapping services as well. Their partners tended to be research administrators, individual researchers, deans, associate deans, department chairs, IT departments, offices of research (or equivalent), and institutional review boards. In most cases, the institution and the library were directionally aligned, but there was one instance in which a restrictive institutional data sharing policy hindered the library's efforts.

"One thing we do not want to do and something that we've been long conscious of, especially when first starting as a data services department within the library, is avoiding being in a condition where we're competing with somebody else or there's a turf war."

"Anything [that is] broader organizationally than just running something within the library will not happen at [redacted] without partners."

"[I]f you're going by the [current institutional] policy, nobody can share any data unless it was already public to begin with…And then, finally…one of our very high level people was like 'Well, our policy says this but, if journals and funders are saying this other thing, then that should just supersede it, so just forget the policy.' And it was like, Okay, well, we need to write that down because I can't, in the library, just be like 'oh forget the policy, do this other thing.' So we've been really struggling with that piece of it."

"I was super interested when I went…to meet with the VPR [Vice President for Research] and talk about how we could partner together on this because I’d had such amazing reception in this area at [redacted]. And that was, I think, my first experience realizing how [the] structure of different institutions really could help or to hinder some of these efforts."

Assessment of library-related RR activities

Six participants noted that their RR work was either not assessed or not assessed well, and five said that meaningful assessment for RR work was difficult to do. Current assessments were mostly limited to easy-to-collect data such as workshop evaluations, attendance statistics, consultation numbers, and repository deposit numbers. Most participants noted that this was not ideal. More meaningful assessments would measure changes in researcher behavior over time. This type of assessment was perceived as a challenge, in part because such assessments require long-term tracking and cannot easily be pinned to specific library interventions.

"Assessment is really tricky because I think that, in general, people try to think about what they can measure, rather than what they should measure. It's easy to do things like how many people participated, or count things, but what really is the value there? I think it's a little bit trickier to operationalize the things that are actually meaningful outcomes of interventions."

"I definitely have concerns that a focus on trying to measure reproducibility is going to be limited to things that are easy to measure, and not get as much at the mindset of researchers and how they engage in their practice."

Challenges associated with RR activities (other than assessment)

Aside from assessment, the main challenges were lack of awareness, time, and money. There was recognition among participants that researchers don't necessarily connect RR to the library, and even when they do, they might not think of the library and its services at the right time. Time was mostly discussed in the context of other services: If RR was a priority, then something had to give to make room for it. Participants also noted that it takes time to develop services and engage with professional development.

We didn't ask participants to provide specifics about the costs of their RR activities, but some participants noted costs associated with hosting a data repository, as well as hiring individuals with specialized expertise. The individuals who do data-related work may have higher salaries than other librarians. One participant referred to data positions as "unicorns," in that the roles are unique and involve specialized skills. Furthermore, these positions can be difficult to replace, which can create a sustainability problem. Two participants struggled to maintain data services at their library after the departure of individuals who were doing the work.

"There can definitely be a time cost. Like anything, it's going to come down to if you're working in this, there's the opportunity cost of not potentially working in a different area."

"If you build a service on some unicorns…can we really rehire that same type of person? So you have to look at some of those things and think about: are they foundational or are they an outgrowth of somebody’s expertise or interest or passion?"

"I think people just don't think of the library. It's a big challenge to figure out how to do outreach that gets to the right people, but also at the right time. It’s one of those things where people might hear it and they totally forget about it, and then the moment when they actually need help, they don't remember that they heard that. I feel like libraries are doing a lot of outreach and yet I still think that people just are not aware and I'm not really sure what that disconnect is."

"Another thing that I think is a challenge is that it would be nice if we could reach people before they had a problem. Reproducibility becomes an issue once the paper is already published and it can't be reproduced but the problems arise way before that. So by the time you know that you have a problem, it's kind of too late to fix it."

When RR is framed around funder and publisher requirements, it can become closely linked to compliance. This challenges librarians who do not necessarily want to associate with that aspect of RR—not wanting the library to be seen as “wielding a stick” or pushing more undesirable work onto researchers. Most participants preferred to focus on RR as good scientific practice.

"The other thing that I think is difficult is nobody likes compliance. So I don't want to fully align ourselves with this compliance thing and be seen on the end of the stick. I'd rather be seen on the end of the 'Okay. It’s okay, we'll help hold your hand and get through this.'"

Vision of RR activities in the future

There was agreement that RR is becoming increasingly important to funders, institutions, and researchers. The NIH Data Management and Sharing Policy (National Institutes of Health 2020. "Final…"), which takes effect in January 2023, was explicitly mentioned by five participants as something they are preparing for.

There was also a belief that in the future more health sciences librarians will need to maintain a baseline level of competency around RR principles, similar to how many librarians currently have a basic understanding of copyright. The most aspirational vision had the library influencing tenure and promotion policy to reward researchers who implement good RR practices.

"I think there will be more and more requirements that most librarians be able to hold an intelligent discussion about reproducibility. Much like we require librarians to have a basic understanding of copyright."

"What I see that will happen in the next couple of years, especially if the funders and the journals continue on the path that they're going and funds become more scarce again…I don't think there are going to be able to be librarians who say that ‘I don't have something to do with this. That's a focus of somebody else's job.’"

"[Y]ou better believe I'm putting stuff in [librarian promotion guidelines] about supporting open science and pre-prints and data sharing and code sharing and how these are the scholarly products that we're going to value equally as much as we are going to value journals."

"One of the things that I think is really important is changing the perspective of researchers and having them view librarians as collaborators and not as service providers, and being co-equal in a lot of ways. Even when I write papers, I hate to say the librarian supported the researcher because I don't want librarians to be viewed in a support role."

Recommendations

We asked each participant to offer recommendations for other librarians who were becoming involved in RR initiatives at their institutions. Most of the recommendations fell into one of four categories: 1) learn from others who are doing similar work, 2) understand your institutional context, 3) identify partners, and 4) model RR in your own work.

1. Learn from others who are doing similar work

"I think we could and should and rely on each other to some extent. This is a world where there is a pretty strong mandate coming our way, and we can help each other not reinvent the wheel."

2. Understand your institutional context

"[I]s this an area of interest for their library? Is it an area within their institutional context where the library is going to make sense to provide services around reproducibility?"

"Understand the landscape. Is this something your population understands and cares about? Try to see what requirements are and is there anything already going on on-campus that you can partner with."

3. Identify partners

"Make sure you have strong campus partners. … Most offices of research have vice presidents, associate vice presidents who are, in fact, very interested and dedicated to rigor and reproducibility."

4. Model RR in your own work

"There aren't so many ways you can get at changing academic culture, but modeling the future you want to see is definitely one of those ways. I definitely believe in working open."

"I think if we actually did this for ourselves, we would be better able to both speak to the value of it, but also to be able to model it."

Discussion

Understanding RR as a library service

All participants offered reasonable definitions of RR and noted that there was a difference between reproducibility and replicability. However, most services weren't marketed with those words, and most of their researchers didn't use the terminology. It is more important to be able to weave principles and behaviors that facilitate RR into relevant conversations. RR, then, may have limited semantic value in defining and marketing many library services. For the participants, it was more important to teach something like good file-naming practices than present the principles of RR.

One participant distinguished between librarians as research collaborators and librarians as research support. We didn't explore this distinction during the study, but it illustrates a tension between the service and partner model of librarian work. More research could be done to understand this distinction, and explore librarian and researcher views and assumptions about the work librarians are doing in the area of RR and data.

Assessing librarian impact on RR

As noted, participants acknowledged that assessment was important, but meaningful assessment was challenging. There was a general sense that the real impact of their RR activities was unknown or hidden within the complex research workflow. But that certainly didn't mean that the services were a failure or even that more time should be allocated to proving positive outcomes. We heard librarians talk about unique institutional partnerships that probably wouldn't have come about without mutual interest in RR and other data services. We also heard of librarians joining interdisciplinary data and policy groups. The institutional partnerships created while doing this work are often great value adds to the library and the institution. Future research could focus on the types and frequency of assessments, explore the value of RR work in more depth, and develop more robust assessment models.

Anticipating the NIH Data Management and Sharing Policy

The new NIH Data Management and Sharing Policy, which was not in effect during the interviews, will likely impact researcher behavior and increase researcher interaction with the library. Exactly what kind of impact is difficult to foresee, but it’s probable that more researchers will begin planning for data earlier in the research lifecycle, and may reach out to the library for assistance while planning for and sharing their data. One of the policy’s supplemental documents, Selecting a Repository for Data Resulting from NIH-Supported Research , refers to librarians as a source of expertise: "Researchers may wish to consult experts in their own institutions (e.g., librarians, data managers) for assistance in selecting among data repositories" (National Institutes of Health 2020. "Supplemental…"). Our results highlight a specific period of time in which librarians involved in RR activity were striving to figure out how the new policy might impact both their work and the workflows of researchers. Future research could explore the impact of the new policy on library services.

Investing in RR

Due to the resources required—time, funds, and personnel—librarians should thoughtfully consider whether RR services are a worthwhile investment at their library. For librarians at institutions where research is being conducted, there is likely value in becoming informed about RR principles and resources to better help researchers address funder and publisher requirements. However, at an institutional level, there might be existing departments and services that restrict or expand library service possibilities.

As noted earlier, it might be difficult to find individuals who can do RR work. This could create a sustainability problem for libraries that seek to increase engagement with RR. But even in restrictive environments, it could still be possible for librarians to apply RR principles to their own work.

Limitations

Our study has limitations. We used interviews to collect data, which restricted the number of participants. We analyzed notes instead of transcripts which may introduce bias. We attempted to minimize this bias by having one author take all the notes (the other author reviewed the first two to confirm the interviews were being accurately captured), and by incorporating many quotes transcribed directly from the interview recordings. We only talked to librarians whose work we knew related to RR, and were thus unlikely to hold contrary views about the relevance of RR services in libraries.

Conclusion

Through interviews with medical and health sciences librarians, we learned that there are a variety of RR services in health sciences libraries. These services are highly dependent on institutional contexts and the skillset of the individual librarians providing the services. RR services are often integrated within existing data services, but individual librarians can model RR principles in their own work, irrespective of service models. More research is needed to further understand a) how librarians can meaningfully assess their contributions to RR initiatives, b) what skills librarians need to perform the work, c) the staff resources required to do the work sustainably, d) the type of RR work librarians are doing in non-health sciences disciplines, and e) the impact of the NIH Data Management and Sharing Policy on library services.

References

Collins, Francis S., and Lawrence A. Tabak. 2014. "Policy: NIH Plans to Enhance Reproducibility." Nature 505: 612-613. https://doi.org/10.1038/505612a .

Flores, Jodi R., Jason J. Brodeur, Morgan G. Daniels, Natsuko Nicholls, and Ece Turnator. 2015. "Libraries and the Research Data Management Landscape." In The Process of Discovery : The CLIR Postdoctoral Fellowship Program and the Future of the Academy , edited by John C. Maclachlan, Elizabeth A. Waraksa, and Christa Williford. Educational Resources Information Center. https://www.clir.org/pubs/reports/pub167 .

Giustini Dean, Kevin B. Read, Ariel Deardorff, Lisa Federer, and Melissa L. Rethlefsen. 2021. "Health Sciences Librarians' Engagement in Open Science: A Scoping Review." Journal of the Medical Library Association 109(4): 540-560. https://doi.org/10.5195/jmla.2021.1256 .

Kim, Soojung, and Sue Yeon Syn. 2021. "Practical Considerations for a Library’s Research Data Management Services: The Case of the National Institutes of Health Library." Journal of the Medical Library Association 109(3): 450-458. https://doi.org/10.5195/jmla.2021.995 .

LaPolla, Fred W., Caitlin Bakker, Nina Exner, Tisha Montnech, Alisa Surkis, and Hao Ye. 2022. "Rigor and Reproducibility Instruction in Academic Medical Libraries." Journal of the Medical Library Association 110(3): 289-293. https://doi.org/10.5195/jmla.2022.1443 .

Murray, Matthew, Megan O'Donnell, Mark J. Laufersweiler, John Novak, Betty Rozum, and Santi Thompson. 2019. "A Survey of the State of Research Data Services in 35 U.S. Academic Libraries, or "Wow, What a Sweeping Question." Research Ideas and Outcomes 5: e48809. https://doi.org/10.3897/rio.5.e48809 .

National Academies of Sciences, Engineering, and Medicine. 2019. Reproducibility and Replicability in Science . Washington: The National Academies Press. https://doi.org/10.17226/25303 .

National Institutes of Health. n.d. "NIH Awards by Location and Organization." Accessed June 21, 2022. https://report.nih.gov/award/index.cfm .

National Institutes of Health. 2015. "Implementing Rigor and Transparency in NIH & AHRQ Research Grant Applications." https://grants.nih.gov/grants/guide/notice-files/NOT-OD-16-011.html .

National Institutes of Health. 2020. "Final NIH Policy for Data Management and Sharing." https://grants.nih.gov/grants/guide/notice-files/NOT-OD-21-013.html .

National Institutes of Health. 2020. "Supplemental Information to the NIH Policy for Data Management and Sharing: Selecting a Repository for Data Resulting from NIH-Supported Research." https://grants.nih.gov/grants/guide/notice-files/NOT-OD-21-016.html .

National Science Foundation. 2014. "A Framework for Ongoing and Future National Science Foundation Activities to Improve Reproducibility, Replicability, and Robustness in Funded Research." https://www.nsf.gov/attachments/134722/public/Reproducibility_NSFPlanforOMB_Dec31_2014.pdf .

Rethlefsen, Melissa L., Mellanye J. Lackey, and Shirley Zhao. 2018. "Building Capacity to Encourage Research Reproducibility and #MakeResearchTrue." Journal of the Medical Library Association 106(1): 113-119. https://doi.org/10.5195/jmla.2018.273 .

Samuel, Sara M., and Mark MacEachern. 2023. “Research Reproducibility Activities in Health Sciences Libraries.” OSF. https://doi.org/10.17605/OSF.IO/UAFH8 .

Sayre, Franklin, and Amy Riegelman. 2018. "The Reproducibility Crisis and Academic Libraries." College and Research Libraries 79(1): 2. https://doi.org/10.5860/crl.79.1.2 .

Sayre, Franklin, and Amy Riegelman. 2019. "Replicable Services for Reproducible Research: A Model for Academic Libraries." College and Research Libraries 80(1): 260. https://doi.org/10.5860/crl.80.2.260 .

Steeves, Vicky. 2017. "Reproducibility librarianship." Collaborative Librarianship 9(2): 80-89. https://digitalcommons.du.edu/collaborativelibrarianship/vol9/iss2/4 .

Tenopir, Carol, Dane Hughes, Suzie Allard, Mike Frame, Ben Birch, Lynn Baird, Robert Sandusky, Madison Langseth, and Andrew Lundeen. 2015. "Research Data Services in Academic Libraries: Data Intensive Roles for the Future?" Journal of eScience Librarianship 4(2): e1085. https://doi.org/10.7191/jeslib.2015.1085 .

Vitale, Cynthia R. H. 2016. "Is Research Reproducibility the New Data Management for Libraries?" Bulletin of the Association for Information Science & Technology 42(3): 38-41. https://asistdl.onlinelibrary.wiley.com/doi/full/10.1002/bul2.2016.1720420313 .

Yoon, Ayoung, and Teresa Schultz. 2017. "Research Data Management Services in Academic Libraries in the US: A Content Analysis of Libraries’ Websites." College & Research Libraries 78(7): 920. https://doi.org/10.5860/crl.78.7.920 .