Cross-posted from our UC3 blog

Welcome to the second post of UC3’s New Year blog post series, where different services of UC3 take a look at the coming year.  If you haven’t already read it, check out the first one on digital preservation.

Over in the world of Data Management Planning, we’ve got a lot of exciting work this year to share!

• DMP Tool Rebuild
• Get Involved with API Integrations
• Matching to Published Research Outputs
• DMP Chef

DMP Tool Rebuild

Our main project continues to be working on the rebuild of the DMP Tool.  While we initially hoped to have it ready early this year, we’re now targeting the summer of 2026.  This gives us more time to make sure it’s at a high level of quality, and also releases it at a time that will hopefully be less disruptive to people who teach classes using the DMP Tool.  There’s a chance it will take longer than the summer though – we’re focused on quality over speed.

We’ve done 3 rounds of user testing so far on the site, and each time has given us a lot of valuable information.  We’ve gotten a lot of positive feedback about new features we will be offering, such as alias email addresses, adding collaborators to templates, a revamped API, and much more.  Other changes, though, have caused some confusion for people used to the current tool, and through testing we have found opportunities to improve the workflow and usability of the new site.  These are the types of changes that mean the rebuild will take longer than initially planned to complete, but we think are worth the time to get right.

As we’ve said before, we’re limiting updates to the current tool so we can focus our limited resources on the rebuild; but of course we also want to keep the tool live and helpful during the transition.  We’re fixing any major issues that come up, such as keeping it up to date with new ROR API and schema, and addressing user tickets as quickly as possible.  We are trying to keep funder templates up to date as well, but the frequency of new information and potential changes has made it difficult to perfectly capture all updates to federal guidelines.  We want to make sure we have the most relevant information possible on the tool without changing templates too often (as that can lose organization guidance), so we’ve been collecting updates from our Editorial Board members for a template release in the near future.  If you see any instances where a template in our tool does not match a funder template, please reach to us by email so we can get it corrected.

Get Involved with API Integrations

With our rebuild is coming a complete revamped API to take advantage of our new machine-actionable functionality.  We’re currently looking for partners that would like early access to our new API in order to develop new integrations for our rebuild.  Our goal is that the new API can do anything the user interface can do, which means the sky (or more relevant, the cloud) is the limit for possible tools.  If you’ve been wanting to connect to our API for some sort of automation that our current API did not offer the capability for, we’d love to hear from you. You can hear more about past pilot integrations and how to work with our API at this recording of our webinar from the Machine-Actional Plans pilot project.  We’ll be following the common API standard being developed with the Research Data Alliance, meaning many integrations with our tool should work for other DMP service providers as well.  If you have an idea for an integration you’d like to build on our new API, please reach out to dmptool@ucop.edu! 

Matching to Published Research Outputs

We’ve talked before about a major project to use machine learning models to help match DMPs to their eventual research outputs, like datasets and software publications, to help make data from published DMPs easier to find and re-use.  This work has continued and we plan to release it with the rebuilt DMP Tool.  Since our last update, we’ve made some significant steps towards this goal, including:

• Moving the infrastructure onto our own servers to prepare for integration into the DMP Tool
• Adding new sources of data, such as grant award pages that list published outputs
• Getting the normalized corpus into OpenSearch to aid us in the matching process
• Expanding our ground truth dataset of true matches and non-matches to help test our matching algorithm
• Utilizing a Learning to Rank model that will improve over time as it learns from accepted and rejected matches
• Building out the user interface for how users will see potential matches and accept or reject them

Improvements we plan to work on over 2026 include:

• Adding in related outputs based on accepted outputs (i.e., finding matches to any Accepted works in addition to matching against the DMP itself)
• Looking at options to improve the matching algorithm, such as vector search with an embedding model
• Working with the COMET team on tooling that can extract award IDs from published outputs, which will improve the quality of matching to DMPs that include an award ID

We’re excited for people to get to use this tool with the rebuild and start accepting and rejecting potential matches so we can learn from this and improve the matching algorithm further over time.  People will also be able to manually add DOIs as research outputs, like they can on the current tool, which will also help train the model over time on what we missed as potential matches.  This will be available for all DMPs that have been published, i.e., registered for a DMP ID.  Accepted works will be added to the metadata for the plan as related identifiers.

DMP Chef

Another exciting area we’re exploring is the use of generative AI to assist in writing Data Management Plans.  We’ve partnered with the FAIR Data Innovations Hub to work on the DMP Chef, a project to explore using large language models (LLMs) to draft DMPs.  Our goal is not to take away the key decisions in data management planning from a researcher, but instead to simplify the process as much as possible by asking a few critical questions, combining that responses with funder requirements that need to be met, and using those to produce a draft of a DMP for their review and edits.

We have promising early results, with both automated statistics and human evaluations showing the LLM-drafted DMPs can be comprehensive, accurate, and follow best practices.  Commercial models are performing better than the open-source models, but since we want to remain open-source, we’re looking at ways to improve the open-source models through additional retrieval augmented generation and other options.  And we’ll be testing carefully how accurate and helpful the output is, as well as looking at ways to help ensure researchers read and edit the plan as needed, rather than just accept the output right away.

DMP Source	Overall Satisfaction rating (1-5)	Average Error Count per DMP	Accuracy in guessing LLM vs Human
Human	3.1	7.2	65%
LLMs (combined)	3.4	4.9	43%
Llama 3.3	2.6	7.5	70%
GPT-4.1	4.2	2.3	15%

Over the course of 2026, we plan to keep testing and improving this model, starting with NIH and NSF plans.  The ultimate goal is a general use model that can be used within the DMP Tool for any funder to get a first draft of either a whole DMP or specific sections a researcher is struggling with.  We have a working prototype tool for DMP generation we will use for testing purposes, with integration into the DMP Tool planned for further out.  If you’d like to be part of testing out this new tool, please sign up for our user panel.

Thanks for reading about our major initiatives for the year!  Keep an eye out on this space for the next post in our series, about our 2026 plans for persistent identifiers.

We are grateful to the Institute of Museum and Library Services, the National Science Foundation, and the Chan Zuckerberg Initiative for each supporting core components of these initiatives.

TL;DR

• The rebuild of the technology behind the DMP Tool offered a chance to refresh the user interface
• We conducted 12 user testing sessions to have real users walk through wireframes of our new tool designs to offer feedback and find issues
• People liked the new designs but had a lot of small areas of confusion around some aspects like sharing and visibility settings
• We made tons of small changes based on feedback and continue to make updates for better usability
• Fill out this short form to have the option to join future feedback sessions

Why we needed new designs

As mentioned in our last blog post, the team behind the DMP Tool has been working on a rebuild of the application to improve usability, add new features, and provide additional machine-actionable features.  To provide all of this advanced functionality, we needed to do a pretty big overhaul of the technology behind the DMP Tool, and it was a good time to give the design a more modern upgrade as well, adding new functionality while hopefully making existing features easier to use.

How we made the first drafts and tested them

Over the past few months, we’ve worked closely with a team of designers to create interactive wireframes—prototype mockups that allow us to test potential updates to the user interface without fully developing them. These wireframes are crucial for gathering feedback from real users early, ensuring that our vision for a better tool meets their expectations.  While a lot of thought and planning went into these initial designs, we wanted to make sure people were finding the new site as easy and intuitive as possible, while still offering new, more intricate features.

To do this, we recruited three groups of people, 12 total, who work on different parts of the tool to test out these designs:

• 5 researchers, who would be writing DMPs in the tool
• 4 organizational administrators, who would be adding guidance to template in the tool
• 3 members of the editorial board or funder representatives, who would be creating templates in the tool

We recruited volunteers from the pilot project members, from our editorial board, from social media, and from asking those we recruited to share the invitation with others. We conducted virtual interviews with each person individually, where we let them explore the wireframe for their section, gave them tasks to complete (e.g., “Share this DMP with someone else”), and asked questions about their experience.  For the most part we let people walk through the wireframes as if they were using it for real, thinking out loud about what they were experiencing and expecting.

What we found from testing

It was illuminating for the team to see live user reactions from these sessions, and watch them use this new tool we’re excited to continue work on. 

We loved to hear users say how excited they were for a particular new feature or how much they liked a new page style.  At times it could be disheartening, watching a user not find something that we thought was accessible, but those findings are even more important because it means we have an area to improve.  We made a report about the findings after each group of users and worked with the designers on how to address the pain points.  Sometimes the solution was straightforward, while other times we wrestled with different options for weeks after testing.

Overall, we found that people liked the new designs and layout and could get through most tasks successfully.  They appreciated the more modern layout and additional options. But there were many areas that the testers identified as confusing or unclear.  There are specific examples, with before-and-after screenshots, in the Appendix.  Some of the top changes made revolved around the following areas:

• Decreasing some text in areas that felt overwhelming, moving less important information to other pages or collapsed by default
• Adding some text to areas that were particularly unclear, such as what selecting “Tags” for a template question would do
• Connecting pages if people consistently went somewhere else, such as adding a link to sharing settings on the Project Members page since that’s where people looked for it first
• Moving some features to not show until they’re needed, such as having Visibility settings as an option in the publishing step and not the drafting step
• Clarifying language throughout when things were unclear, such as distinguishing whether “Question Requirements” was about what the plan writer was required to write when creating their DMP or whether that was about the template creator marking whether a question is required or had display logic
• Having additional preview options when creating a template or adding guidance to understand what a question or section would look like to a user writing a DMP
• Making certain buttons more prominent if they were the primary action on a page, like downloading a completed DMP that originally was hard to find

Even though the main structure worked well for people, these small issues would have added up to a lot more confusion and obstacles for users if we hadn’t identified them before releasing.  

Wrapping up and moving forward

The whole team learned a ton from these sessions, and we’re grateful to all the participants who signed up and gave their time to help us improve the tool.  This sort of testing was invaluable to find areas to improve – we made dozens, if not hundreds, of small and large changes to the wireframes based on this testing, and we hope it’s now much better than it was originally. We’re still working on updates as we build our designs for more areas of the site, but feel better now about our core functionality.

If you’d like to be invited to participate in surveys, interviews, or other feedback opportunities like this for the DMP Tool, please fill out this brief form here: Feedback Panel Sign-Up. For anyone that signed up but wasn’t selected for this round, we may reach out in the future! 

We loved seeing how excited people are about this update, and we can’t wait to share more.  The most common question we get is – when is it releasing!  That’s going to be quite some time, and we don’t have more to share yet, as we’re still too early in the development process.  But stay tuned here for more updates as we do! 

We want to thank Chan Zuckerberg Initiative (CZI) for their generous support for rearchitecting our platform. We wouldn’t be able to make all of these helpful updates along with our back-end transformations without it.

Appendix: Specific Examples

Important note: The “updated wireframes” shown here are not final designs. We have not yet completed a design pass for things like fonts, colors, spacing, and accessibility; this is just a quick functionality prototype so we could get early feedback. Even the functionality shown here may change as we develop based on additional feedback, technical challenges, or other issues identified. Additionally, these wireframes are mockups and do not have real data in them, so there may be inconsistent or incorrect info in affiliations, templates, etc; we were focused on the overall user interface in testing, not specific content.

Sharing settings

For those who want some more details and specific examples, here are a few of the top areas of confusion we found:

There was sometimes confusion in how to share a plan with others, and what the distinction is between a Project collaborator (e.g., another researcher on the grant who may not be involved in the DMP) and a DMP collaborator (e.g., a peer who is giving feedback on writing the DMP but not on the project).  The current live tool has both “Project Contributors” and “DMP Collaborators” on the same page which we thought contributed to this confusion, so we wanted to separate those who can edit the DMP into a separate Sharing section.  However, testers had a hard time finding these sharing settings, and often went to the Collaborators page to grant DMP access.  So, we added a link to these settings where people were looking (the new section in the green box), and added more detail to the sharing page about whether they were invited or had access due to being a collaborator, changed some language within this like “Collaborator” to “Project Member,” with the option to change access.

Current tool:

Initial wireframes:

Updated wireframes:

Card layout

Many parts of the tool used a new, more modern card format for displaying lists of items to choose from.  This allowed us to show more information than in a list, and adapt to smaller screens. However, we saw in some areas that people had trouble scanning these cards to find what they were looking for, like a plan or template, when they expected to search in alphabetical order.

For example, picking a template in the first draft used a boxier card format. People found it harder to find the template they were looking for, since they wanted to quickly scan the titles vertically.  So we changed it to a different format that should be easier to scan, even if it doesn’t show as many on one page.  Note we also now have the option to pick a template other than from your funder, a common request in the current tool.  

Current tool:

Initial wireframe:

Updated wireframe:

Flow through the tool

People appreciated that they could move around more freely in the new design, as compared to the more linear format of the current tool. However, that also occasionally made people feel “lost” as to where they were in the process of writing a DMP. Especially as there is now a “Project” level above each plan to help support when people have multiple DMPs for the same research project.  So we added more guidance, breadcrumbs, and navigation while still allowing the freedom of movement throughout the process.

For example, while writing a plan, users will now be able to see the other sections available and understand where they are in the Project tree.  We also reduced some of the text on screen due to people feeling overwhelmed with information, putting some best practices behind links that people can visit if they wish to, and moved the Sample Answer people were most interested in to above the text box for better visibility.

Current tool:

Initial wireframe:

Updated wireframe:

Layout changes

In addition, there were tons of small changes throughout, changing layouts, wordings, and ordering of options in response to areas of confusion.  Some places we scaled back a bit of functionality since the number of new options were overwhelming, while other places we added a bit more that people needed.

In the first draft of the wireframes, the visibility settings of the plan were on the main overview page of the plan.  This was concerning to users since they were still drafting at this stage, and even if they may want it public once they published it, the setting in this location made it seem like it was public now.  Instead we added a status and setting on the overview page, but the visibility setting does come up until a person gets to the Publish step, somewhat like the current tool that has those options later than in the plan writing stage.

Current tool:

Initial wireframe:

Updated wireframe:

We made similar change to creating a template, moving the visibility settings to be selected in the publishing stage instead of being in a Template Options menu people didn’t always see right away.  They expected to set that visibility at the time they published it, so that’s where we moved that option to be, consistent with how the plan creation flow works.