henhaobio
Sign in

PLOS ONEScientific Communication & EducationPeer-reviewed journal article

From peer-reviewed to peer-reproduced: a role for data standards, models and computational workflows in scholarly publishing

Alejandra Gonzalez-BeltraniD, Peter Li, Jun Zhao, Maria Susana Avila-Garcia, Marco RoosiD, Mark Thompson, Eelke van der Horst, Rajaram KaliyaperumaliD, Ruibang Luo, Tin-Lap Lee, Tak-wah Lam, Scott C. Edmunds, Susanna-Assunta SansoneiD, Philippe Rocca-SerraiD

PDFDOIPreprintCodeData
In-depth full-text digest available · Jump to digest ↓

Abstract

MotivationReproducing the results from a scientific paper can be challenging due to the absence of data and the computational tools required for their analysis. In addition, details relating to the procedures used to obtain the published results can be difficult to discern due to the use of natural language when reporting how experiments have been performed. The Investigation/Study/Assay (ISA), Nanopublications (NP) and Research Objects (RO) models are conceptual data modelling frameworks that can structure such information from scientific papers. Computational workflow platforms can also be used to reproduce analyses of data in a principled manner. We assessed the extent by which ISA, NP and RO models, together with the Galaxy workflow system, can capture the experimental processes and reproduce the findings of a previously published paper reporting on the development of SOAPdenovo2, a de novo genome assembler.\n\nResultsExecutable workflows were developed using Galaxy which reproduced results that were consistent with the published findings. A structured representation of the information in the SOAPdenovo2 paper was produced by combining the use of ISA, NP and RO models. By structuring the information in the published paper using these data and scientific workflow modelling frameworks, it was possible to explicitly declare elements of experimental design, variables and findings. The models served as guides in the curation of scientific information and this led to the identification of inconsistencies in the original published paper, thereby allowing its authors to publish corrections in the form of an errata.\n\nAvailabilitySOAPdenovo2 scripts, data and results are available through the GigaScience Database: http://dx.doi.org/10.5524/100044; the workflows are available from GigaGalaxy: http://galaxy.cbiit.cuhk.edu.hk; and the representations using the ISA, NP and RO models are available through the SOAPdenovo2 case study website http://isa-tools.github.io/soapdenovo2/. Contact: philippe.rocca-serra@oerc.ox.ac.uk and susanna.assunta-sansone@oerc.ox.ac.uk

In-depth summary

Claude Opus 5.5 · xhigh effort · 8 Nov 2026 · from the full text of version 1 (8 Dec 2014) · CC BY 4.0

Galaxy workflows reproduced SOAPdenovo2 bacterial assembly results almost exactly, while ISA, Nanopublication and Research Object models exposed reporting errors leading to a published correction.

Published computational results can be hard to reproduce even when the data, scripts and documentation are openly deposited and refereed, and narrative methods hide design details and resource identifiers. This case study tests whether structured metadata models combined with executable workflows can support peer review by checking whether an article can be repeated and whether its reporting is accurate.

Key results

  1. ShownGalaxy workflows re-ran SOAPdenovo2, SOAPdenovo1 and ALL-PATHS-LG on the two bacterial genomes (S. aureus, R. sphaeroides) and closely matched the published assembly statistics. The SOAPdenovo2 workflows needed two extra steps that the original authors' scripts did not contain.

    The SOAPdenovo2 workflow outputs were described as almost identical to the original Table 2 (reproduced values are in Table 1). SOAPdenovo1 and ALL-PATHS-LG reruns showed minor discrepancies. The added steps split scaffolds at gaps and ran the GAGE statistics script. Reproduction took about half a man-month even though data and scripts were open. · 0.2 Reproducing the results from the paper with…; Discussion; Tables

  2. ShownBuilding nanopublications meant tying each claim to table evidence. A reduced run-time assertion that the curators drafted while reading the abstract proved wrong. Separately, inaccuracies found during this model-assisted review led the original authors to publish a correction.

    9 assertions were encoded as 9 nanopublications. Of three assertions first drafted by curators from abstract statements (A1 higher coverage, A2 lower memory, A3 lower run time), A3 proved incorrect: Table 4 of the original paper showed SOAPdenovo2 running slightly longer than the other two assemblers. Peak memory on the human data was 2/3 lower than SOAPdenovo1. · 0.4 Publishing findings as Nanopublications; Discussion

  3. ShownThe ISA minimal-reporting guidelines require resolvable identifiers for data files and software. Applying them uncovered missing files, unresolvable file references and an ambiguously named human reference genome in the original deposit.

    Through correspondence with the authors, 'hg19' was resolved to GRCh37, GenBank Assembly GCA 000001405.1, which separates it from 7 later releases. Example missing files are listed in Table 2. The ISA-Tab archive was also released as RDF via linkedISA, mapped to OBI and PROV-O. · 0.3 Modeling the experimental process using ISA; Tables

  4. ShownTreating the assembler comparison as a designed experiment, with software and genome size as factors and six response variables, showed an unbalanced design that lacked replication in several groups, so variance could not be estimated.

    Modelled as a 3×3 factorial. Response variables: genome coverage, contig N50, scaffold N50, errors, run time (hours) and peak memory (GB). Two bacterial genomes vs one mid-size and one large genome. Suggested additions: Apis mellifera (236Mb) and dog (2.4 gigabases) genomes. The original authors cited machine occupancy and cost as obstacles. · 0.3 Modeling the experimental process using ISA; Discussion

  5. ShownA workflow-centric Research Object bundled the Galaxy workflow, links to its input data and provenance statements as uniquely identified, citable parts. It complements ISA (study design) and the nanopublications (findings).

    Built with the command-line RO Manager, checked against a 5-rule completeness checklist and published in the public RO repository under a PURL. The authors note some overlap with linkedISA in how workflow inputs and outputs are captured. · 0.5 Preserving computational workflows and…; 0.6 ISA, NP and RO models: a complementary set…; 0.10 The Research Object model

Methods, data and reproducibility

Case study that re-ran the SOAPdenovo2 paper's assembly analyses as Galaxy workflows and then curated the article in three forms: an ISA-Tab description converted to RDF with linkedISA, nanopublications of key findings built with OntoMaton/NanoMaton, and a workflow-centric Research Object holding the workflow, inputs and provenance.

  • S. aureus GAGE sequencing data set, re-assembled in Galaxy with SOAPdenovo2, SOAPdenovo1 and ALL-PATHS-LG · S. aureus (bacterium)
  • R. sphaeroides GAGE sequencing data set, re-assembled in Galaxy with the three assemblers · R. sphaeroides (bacterium)
  • B. impatiens GAGE insect genome data set; not re-assembled because of server memory limits · B. impatiens (insect)
  • YH Asian (Chinese Han) human genome reads from the Illumina platform; modelled in ISA but not re-assembled in Galaxy · human
  • Human reference genome hg19/GRCh37, used for the alignment step · human · GenBank Assembly ID: GCA 000001405.1
  • GigaDB record with the SOAPdenovo2 scripts, data and results · 10.5524/100044
Methods & tools
Galaxy (GigaGalaxy instance), SOAPdenovo2, SOAPdenovo1, ALL-PATHS-LG, GAGE assembly validation script, ISA-Tab / ISAcreator, linkedISA (ISA-Tab to RDF), OntoMaton, NanoMaton, RO Manager (command-line), Ontologies: OBI, STATO, SIO, PROV-O, Nanopublication and Research Object models
Software & environment
Galaxy server (GigaGalaxy) set up on GigaScience hardware; SOAPdenovo2 and its pre/post-processing tools were wrapped through their command-line interfaces. No software versions are stated.
Assay conditions & compute
The public Galaxy server could not supply the up to 155 GB of memory needed for the human assembly. An AWS cr1.8xlarge instance (up to 244GB memory) at USD 3.5 per hour was estimated at USD 200-300 per YH assembly, excluding storage.
Reproducibility gaps
The B. impatiens and YH assembly workflows could not run on the public server. The original deposit had missing files and unresolvable references, and did not include intermediate data. The RO and the correction article are linked through goo.gl short URLs.
Limitations (6)
  • Only the two bacterial assemblies were re-run. The B. impatiens and human YH workflows could not run because the public Galaxy server lacked the up to 155 GB of memory needed, and re-running YH on cloud hardware was estimated at USD 200-300. · stated by the authors
  • The ontologies used (OBI, STATO, SIO) had no terms for response variables such as genome coverage, memory and runtime, so a new ontology was created. Nanopublications also carry the interpretation of whoever translates the text into triples. · stated by the authors
  • The authors consider it unrealistic for researchers to master semantic representation models, so routine use in publishing needs new, scalable authoring tools. Reproducing a well-documented paper still took about half a man-month. · stated by the authors
  • The workflow-centric Research Object and the linkedISA conversion partly duplicated each other in capturing workflow inputs and outputs. · stated by the authors
  • The evidence comes from a single computational article with openly deposited data. Whether the effort and benefits carry over to wet-lab studies or less well-documented papers is not tested. · noted in this summary
  • The text describes agreement between reproduced and published statistics only qualitatively (almost identical, minor discrepancies) and sets no acceptance tolerance. Per-metric values appear only side by side in Table 1. · noted in this summary

Read this if you design reproducibility checks, data-publication standards or semantic metadata (ISA, nanopublications, Research Objects) for journals and want a worked genome-assembly case study.

Reproducibility ledger

Nobody has reported trying to reproduce this paper yet. Did you replicate an experiment or run the code? Your report saves the next lab a week.

Sign in to file a reproducibility report.

Related papers

Similarity Map
SelectedSame fieldCross-field
THISPiccolo '16Riffle '26Krieger '19Schölzel '21Xu '24Bhagwat '26
55% semantic matchBioinformatics
Stephen R Piccolo, Michael B Frampton·bioRxiv·Mar 24, 2016