NIH Policy on Rigor and Reproducibility

You’ve all heard about the reproducibility crisis in science. But you may not be aware of a (relatively) new National Institutes of Health (NIH) policy designed to address the issue. The NIH Policy on Rigor and Reproducibility became effective for proposals received on or after January 25, 2016 and applies to most NIH and Agency for Healthcare Research and Quality (AHRQ) grant applications. We just learned about the policy ourselves thanks to the combined efforts of UCSD library and research staff to raise awareness on their campus (and here’s a noteworthy mention in a Nature review of 2015 science news). To aid researchers in meeting the new criteria, UCSD produced this handy guide that we (and they) would like to share with the wider community.

The new policy does not involve any changes to data sharing plans. It is related and important enough, however, that we inserted a statement and link in the “NIH-GEN: Generic” template (Please note the Rigor and Reproducibility requirements that involve updates to grant application instructions and review criteria [but not Data Sharing Plans]).

The policy does involve:

  • Revisions to application guide instructions for preparing your research strategy attachment
  • Use of a new “Authentication of Key Biological and/or Chemical Resources” attachment (example from UCSD library website)
  • Additional rigor and transparency questions reviewers will be asked to consider when reviewing applications

These policies are all meant to achieve basically the same goals: to promote openness, transparency, reproducibility, access to, and reuse of the results of scientific research. We’re grateful to the folks at UCSD—Dr. Anita Bandrowski, Ho Jung Yoo, and Reid Otsuji—for helping to consolidate the message and for providing some new educational resources.

Roadmap team-building exercises: US edition

roadmap_wallLast week we hosted Marta Ribeiro, the lead developer for DMPonline, for an intense, donut-fueled planning meeting to define our co-development process and consolidate our joint roadmap. The following is a debriefing on what we accomplished and what we identified as our next steps.

The project team is established, with Brian Riley joining as the DMPTool technical lead. Marta is busy completing the migration of DMPonline to Rails 4.2 to deposit the code into our new Github repository: DMPRoadmap. There’s nothing to see just yet—we’re in the midst of populating it with documentation about our process, roadmap, issues, etc. As soon as everything is in place, we’ll send word so that anyone who’s interested can track our progress. This will also allow us to begin sussing out how to incorporate external development efforts to benefit the larger DMP community. In addition, Marta is mentoring a pair of summer interns who are undertaking the internationalization work and building APIs. Meanwhile, Brian will finish building the servers for the Roadmap development and staging environments on AWS with another new member of the UC3 team: Jim Vanderveen (DevOps/Developer). Additional core team members include Stephanie Simms and Sarah Jones as Service/Project Managers, Marisa Strong as the Technical Manager, and the CDL UX team (many thanks to our UX Design Manager, Rachael Hu, for spending so much time with us!). UC3 and DCC will also rely on their existing user groups for testing and feedback on both sides of the pond.

Other groundlaying activities include a web accessibility evaluation for DMPonline to ensure that the new system is accessible for disabled users and exploring what we (and others) mean when we talk about “machine-readable DMPs.” Stephanie just received an RDA/US Data Share Fellowship to develop use cases for making DMPs machine readable, in consultation with the Active DMPs Interest Group and the research community at large. In line with this effort, she’ll be participating in an interdisciplinary and international workshop on active DMPs next month, co-hosted by CERN and the RDA group. We’re actively seeking and summarizing thoughts on the topic so please send us your ideas!

We conclude this edition with a draft of our project roadmap (below); it lists all of the features that we’ll be adding to the DMPonline codebase before we release the new platform. Most of the features already exist in the DMPTool and were slated for future enhancements to DMPonline. Stay tuned for our next update following a UC3 exchange visit to Glasgow/Edinburgh in mid June to prioritize the roadmap and commence co-development work.


  • Migration to Rails v.4.2
  • Bring DMP Assistant’s internationalization upstream for multi-lingual support
  • Adding the concept of locales so specific organizations, funders, and templates can be defined and filtered out for certain users/contexts
  • Shibboleth support through eduGain
  • OAuth link for ORCID
  • APIs to create plans, extract guidance, and generate usage statistics
  • More robust institutional branding
  • A lifecycle to indicate the status of plans and allow institutional access to plans
  • Support for reviewing plans
  • Public sharing option > Public DMPs library
  • Flag test plans (to exclude them from usage stats)
  • Email notification system
  • Admin controls for assigning admin rights to others
  • Export template with guidance
  • Copy template option for creating new templates
  • Copy plan option for creating new plans
  • Toggle switch for navigating between Plan area and Admin area

Maintenance and bugs

We just announced a maintenance window for Wednesday, 4 May 2016, 3:00–4:00pm (PST). We’ll be updating some support scripts related to last month’s migration to AWS that help ensure the stability of the system. Following the maintenance, these bug fixes will also be in place:

Bug fixes

  • We fixed an issue that prevented admins from uploading files for institutional logos.
  • We created a static FAQ page on GitHub to replace the unreliable askbot service.
  • We added a date field to the API call to get a list of users. This call enables admins to track new users over time.
  • We updated links on the Help pages and other outdated info throughout the platform (e.g., we have a new CDL Director: Günter Waibel!).

We’ll continue on a 2-week release cycle until all of the bugs are wiped out. You can follow our progress in the GitHub issue tracker and keep a lookout for updates. We also invite direct bug reporting and enhancement requests via GitHub issues.

Roadmaps galore

Data management planning is moving and shaking at all scales—local, national, international—these days. We had excellent conversations at IDCC about coordinating responses to proliferating data policies and sharing experiences across borders and disciplines. All of the slides and materials from the international DMP workshop are available here.

So far the community has responded positively to our proposal for building a global infrastructure for all things DMP. Our big-picture plans include a merged platform based on the DMPonline codebase and incorporating recent internationalization work by the Portage Network in Canada (check out their bilingual DMP Assistant). We’re completing a gap analysis to add existing functionality from the DMPTool to DMPonline and will issue a joint roadmap in the coming months. Drawing together these disparate development efforts also presents an opportunity to set best practices for future work (stay tuned). This will allow us to consolidate value upstream and ensure maximum benefits to the entire community.

To facilitate our capacity-building efforts, we submitted a proposal entitled (what else) “Roadmap” to the Open Science Prize. You can read the Executive Summary on their website here and peruse the full proposal here (also view our snazzy promo video below). The prize seemed like the perfect opportunity to reposition DMPs as living documents using the biomedical research community as a pilot group. We’ll know by the end of April whether our bid is successful. Regardless of the outcome, we would love to know what you think about the proposal.

And finally, an update on the near-future roadmap for the DMPTool. We just added some new API calls in response to requests for more usage statistics and to facilitate integration projects with other data management systems. Admins can now get info about templates used to create plans at their institution (including private plans!) and a list of institutional templates. Check out the updated documentation on the GitHub wiki. The next order of business is working through the backlog of bug fixes. You can follow our progress in the GitHub issue tracker. Once the bugs are eliminated, we’ll circle back to high priority feature enhancements that contribute to our long-range plans.

DMPs are going global

…well international at least, with global aspirations. The US-based DMPTool and UK-based DMPonline have collaborated from the beginning to provide data management planning services and training on our respective sides of the pond. As more and more funders, institutions, and nations—the entire EU, for instance—adopt data sharing policies, we find ourselves supporting data management planning initiatives farther and wider.

To meet the exploding demand and facilitate connecting the dots (e.g., promoting community standards for DMPs), we’ve decided to formalize our partnership and move toward a single platform for all things DMP. You can learn more about our evolving partnership in this joint paper that we’ll be presenting at the International Digital Curation Conference (IDCC) at the end of Feb. Stay tuned for updates about a joint roadmap and timeline in the coming months. Our individual roadmaps will remain in place for now.

As always, we invite your feedback! And if you happen to be attending IDCC, consider joining us and the DART Project for an international DMP workshop on Thurs, Feb 25 (registration info).


NASA’s Global Digital Selfie 2014

DMPTool maintenance and a roadmap

First things first, we want to inform you about upcoming maintenance. The DMPTool will be unavailable on Wednesday Feb 3, 2016 5:00 – 6:00pm (PDT) while we migrate the database to AWS. The application will also be moving to the cloud so you can expect another maintenance message in the near future. We apologize for the inconvenience.

In other news, we’re excited to share the following roadmap for future development. Thanks to all of your suggestions about how to improve the DMPTool, we’ve identified and prioritized some feature enhancements to meet evolving data management planning needs:

  • API work for integration projects with other data management systems, to extract guidance and customizations, and to generate additional user statistics
  • Formatting tools for template creation and customization views
  • Search function for lists of templates, customizations, and partner institutions
  • Function for customizing auto-generated email messages (e.g., when a user submits a plan for review)

Migrations and bug fixes are already underway. We’ll provide a timeframe for the roadmap in the coming months as we clear the backlog and move on to enhancements. You can also track our progress and help shape future plans by submitting feedback through our GitHub Issue Tracker.

New NSF-BIO template

On October 1, 2015, the NSF Directorate of Biological Sciences issued Updated Information about the Data Management Plan Required for Full Proposals. Changes to the guidelines include some reorganization and clarification of the components of the DMP. There is a new section, “Future Proposals,” highlighting that DMP implementation will be considered during evaluation of future proposals. The new guidelines also contain a handy list of data management resources and training opportunities, including links to DataONE, Data Carpentry, and Software Carpentry, as well as mention of services provided by university libraries. According to the NSF Bio Buzz blog, the updated DMP guidelines are part of the newly released Proposal and Award Policies and Procedures Guide, which applies to proposals submitted, or due, on or after January 25, 2016.

As with the previous revision to the NSF-BIO guidelines in 2013, we handled the changes to the template in the DMPTool by deactivating the old one and creating a new template. DMPTool users who created plans using the old template(s) will continue to have full access to those plans. Users creating new NSF-BIO plans will be presented with the new 2015 template.

We will continue monitoring the Bio Buzz blog for future updates.

New DOE “Generic” template

We’ve added a drop-down list for the U.S. Department of Energy (DOE) and created a new “Generic” template. The existing “DOE: Office of Science” template remains the same.

These template changes come in response to the recent release of the agency-wide DOE Policy for Digital Research Data Management, which took effect on October 1, 2015 and applies to all Unclassified and Otherwise Unrestricted Digital Research Data. The suggested elements for data management that were being piloted by the Office of Science will now apply to all DOE-sponsored research programs. Going forward, each sponsoring office must include the requirements for DMPs in all solicitations and invitations for research funding, with details about how and when a DMP should be submitted. The generic template contains updated DOE links that direct researchers to consult the appropriate sponsoring office for specific requirements and guidance. The new links also include a helpful list of Data Management Resources at DOE Scientific User Facilities.

Stay tuned for more template news from the DMPTool as agencies continue to revise existing policies and release new ones.

New template: NIH Genomic Data Sharing

The National Institutes of Health issued new Guidance for Investigators in Developing Genomic Data Sharing Plans along with some helpful sample plans (dated July 14, 2015). The DMPTool team has been monitoring the responses to the OSTP memo by federal agencies, but this alert came to us via the DMP admin email list. Please continue to let us know when you hear anything at all (see the links from a previous post “How you Can Help”)!

We added a National Institutes of Health drop-down list to the DMPTool that contains the new NIH-GDS: Genomic Data Sharing template in addition to the NIH-GEN: Generic template. The basic data management requirements for most NIH grants remain unchanged (pending further notice); researchers can continue to use the generic template for most grants.

The new guidance pertains to those proposing research that will generate large-scale human and non-human genomic data. It describes the type of information that should be provided in a genomic data sharing plan and when the plan should be submitted, including instructions for IRB review, appropriate uses of the data, and suggested/required data repositories. The new guidance is an update to the existing NIH GDS Policy that became effective on January 25, 2015.

New templates for IMLS

The Institute for Museum and Library Services (IMLS) changed its data management plan requirements, by expanding the scope to include research data, other digital content, and software tools and applications. Although IMLS uses just one form for these new requirements, we decided to split it into three DMPTool templates. Each of the new DMPTool templates for the IMLS consists of one section on Copyright and Intellectual Property Rights and section(s) specific to the type of digital products from the proposed project.

IMLS changed its requirements in 2014. Unfortunately, it has taken us a while to add them to the DMPTool, as we discussed the best way to represent them. One template or three? Keeping them as one had several disadvantages, with the most significant disadvantage being its length. We also think that most people will need only one of the sections for any given project. Unfortunately, splitting them into three caused problems with the automatic numbering that the DMPTool added to sections of the plans when saved to PDF or RTF. The numbers didn’t match the section numbers in the IMLS requirements. We’ve removed the automatic numbering, and now have released the new templates. We hope these are useful to researchers applying for grants from the IMLS.