This is a PLOS Computational Biology Benchmarking paper.

During my training and now, as a group leader, I have created several open source research software tools. My group publicly commits to maintaining these for a minimum of 5 years after the publication of the associated manuscript. In practice, several of our projects have been maintained for longer given that they continue to be used. Ideally, software should be maintained for as long as it is being actively used. We also provide some basic support to users, in the form of addressing their queries. In this perspective, I will first define exactly what is meant by maintenance and when we are committing to it, then describe how we make it possible, and finally end with a call for the community and institutions to be explicit about what users can expect from each piece of research software.

Level 0: One-off analysis. This comprises the day-to-day development of scripts. These will be limited in scope to the current application and it is accepted that they will often contain minor bugs such as mishandling boundary conditions. Part of the process of computational science is to find (and correct) these errors to prevent erroneous conclusions. Most analyses will only be intermediate steps towards a publication and will be discarded en route.

Level 1: Extended Methods Code (for support of a results-driven publication). When we publish a paper that focuses on biological results, we will publish the analysis scripts alongside it. The goal is to act as an extension to the Methods Section, disambiguating any unclear statements made in natural language, while also making it easier for others to reproduce and build on the work. However, we expect that only a small minority of the readers of the paper will use (or even download) the code. While this code generally started as one-off analysis code (Level 0), it was reviewed for potential flaws. Furthermore, we added a modicum of documentation and took care to ensure that the code is understandable by an outside reader and that it does not rely on the specific installation and environments we use (e.g., no internal paths). Nonetheless, this code produces a single result and is not primarily intended to be used by others.

Level 2: A Tool for others to use. A tool for others to use is the most demanding type of code that we produce. Here, the goal is that the highest possible number of readers of the paper become users of the tool. For long-term robustness, it is important that such Level 2 code be as user-friendly, well-documented, and platform-independent as possible.

It is just as ill-advised to release under-engineered software as a Tool (thus, misleading potential users as to the quality of the underlying code) as to expend resources to optimize and make robust code that will only be used a small number of times. As described in the main text, we aim to have tools that produce high-quality error messages by detecting cases such as insufficient temporary disk space. It could be wasteful, however, to expend the effort necessary to add this functionality to Extended Methods Code.

Having established that our commitment is to maintaining and supporting code that we release as Tools, I will now describe 7 practices we use to achieve it. Although the commitment is forward-looking, most of its effect is to change our behavior prior to and immediately after publication. The goal is that the software continues to be available and useful while minimizing the long-term effort from the part of the authors and minimize the disruption when the main author of the tool leaves the group.

One of the difficulties in providing tools is that one loses the perspective of how a beginner approaches the problem. Therefore, we ask that incoming students attempt to install any software using only publicly available sources and report any issues. It is important to communicate to students that finding faults with documentation is an important outcome of their efforts in order to avoid that they take shortcuts that are not available to the wider community (most obviously, asking the original authors of the tools for offline help).

As users often face similar issues, it is inefficient to tackle each of them individually, in a one-off manner. In particular, providing support by private email benefits only the immediate recipients, even as others (perhaps in the future) are likely to encounter the exact same issue. Therefore, we try to provide support using public fora, such as a mailing-list, GitHub Issues, or YouTube videos. When answering a single individual, we often also copy the response into the documentation.

To achieve this, you need to first make it clear to users what the preferred support channels are. When users contact us directly by email, we steer them towards the public forum. This strategy is also employed by corporations that provide services free of charge and provide support only on forums where a significant fraction (often the majority) of the support is provided by other users.

A high-quality error message is one that enables users to quickly diagnose and, if possible, fix the underlying issue. Where possible, we treat confusing error messages as bugs. Good error messages not only provide a better user experience, but they can reduce the amount of support that needs to be provided.

For example, when a user reports a long and confusing error that results from a lack of disk space, it is tempting to simply reply that they need to increase their temporary storage capacity. However, when feasible, we also add the extra code to detect and report this situation. A good message would read like “not enough temporary disk space in temporary directory: /tmp”—note that by explicitly printing the temporary directory being used, we also help the user identify the case where the temporary directory has been misconfigured. We hope that this will enable the next user facing the same problem to solve it by themselves without the need to contact us.

It is possible to systematically test for these issues by running tests in artificially constrained environments, but, in practice, we rely on user reports.

Beta users are incredibly important for catching bugs, including usability and documentation bugs. Thus, we make our software available even prior to publication and advertise it online and at conferences. We often attract testing users who report bugs and other issues.

I presented several principles that I use in my group to provide long-term support for the tools we publish at minimal long-term cost. A common theme is to not only fix any immediate problem encountered or reported, but also to attempt to avoid similar issues in the future or to equip users to solve them themselves. Often, when using a platform such as Github Issues, the user reporting the original bug will close the issue as soon as their problem is solved. While this is well-intentioned, I will reopen the issue with a note that there is a possible improvement to the software (for example, that an error message could have provided more information) and only close the issue once this improvement has been implemented.

When we first release a public test version, and immediately after the manuscript is published, users will report many issues, including lacuna (and outright mistakes) in the documentation. There is a risk that junior students get discouraged if they perceive this user feedback as negative critiques of their work. It is important to stress that receiving bug reports is actually a very positive sign as it indicates engagement and leads to improvement of the code. As reported issues get tackled, the stream of reports should slow down even as usage continues to increase. Questions are also easier to handle when we can point users to previously written documentation rather than having to write an individualized answer.

The perception that bioinformatics software is low-quality damages the reputation of the field as a whole. Therefore, I hope that others will copy the commitment that my group makes. I further hope that similar pledges become institutionalized: funders (including peer reviewers evaluating grant proposals) should evaluate the track record of applicants in keeping their software up-to-date while considering whether to provide new funding.

I thank Daniel S. Katz (National Center for Supercomputing Applications, University of Illinois), Konrad Hinsen (Centre de Biophysique Moléculaire, Orléans), and all members of my research group for feedback on earlier versions of this manuscript.