Go, C++ or Java for DNA sequencing?

A DNA sequencer takes a DNA sample, such as human tissue, and applies chemical processes to eventually read small fragments of the DNA sample and output them as large files. These files are then fed into software pipelines that reconstruct the original DNA sequence from those fragments. Such sequencing pipelines need large amounts of storage, on disk and/or in RAM, and can strongly benefit from parallel execution to improve runtime performance. Data sets for human DNA samples are usually in the order of several hundreds of GB of uncompressed data, and runtimes are typically in the order of several hours for single samples.

While Go is not primarily designed for parallel programming, it nevertheless has features that end up being beneficial for parallelism as well, especially the inclusion of a work-stealing scheduler for goroutines and a concurrent, parallel garbage collector. For this reason, we have recently included Go as one of several candidate programming languages in an evaluation of their suitability for expressing sequencing pipelines. We have also included C++, which supports reference counting as a parallelism-friendly memory management option through its std::shared_ptr template, and Java because of its various alternative concurrent and/or parallel garbage collectors. Go has a better runtime performance than both these alternatives, and its memory use is close to the best results, without requiring major optimisation efforts.

Links

The DNA sequencing tool being evaluated is elPrep, which has been open sourced at https://github.com/exascience/elprep. We have also developed and released Pargo, which is an open-source library for parallel programming in Go that is used in elPrep. Pargo can be found at https://github.com/exascience/pargo.

The previous version of elPrep and its performance characteristics, before it was ported from Common Lisp to Go, is described in an open-access online journal article at PLOS One.