Cookies on this website

We use cookies to ensure that we give you the best experience on our website. If you click 'Accept all cookies' we'll assume that you are happy to receive all cookies and you won't see this message again. If you click 'Reject all non-essential cookies' only necessary cookies providing core functionality such as security, network management, and accessibility will be enabled. Click 'Find out more' for information on how to change your cookie settings.

BACKGROUND: Modern biological science generates a vast amount of data, the analysis of which presents a major challenge to researchers. Data are commonly represented in tables stored as plain text files and require line-by-line parsing for analysis, which is time consuming and error prone. Furthermore, there is no simple means of indexing these files so that rows containing particular values can be quickly found. RESULTS: We introduce a new data format and software library called wormtable, which provides efficient access to tabular data in Python. Wormtable stores data in a compact binary format, provides random access to rows, and enables sophisticated indexing on columns within these tables. Files written in existing formats can be easily converted to wormtable format, and we provide conversion utilities for the VCF and GTF formats. CONCLUSIONS: Wormtable's simple API allows users to process large tables orders of magnitude more quickly than is possible when parsing text. Furthermore, the indexing facilities provide efficient access to subsets of the data along with providing useful methods of summarising columns. Since third-party libraries or custom code are no longer needed to parse complex plain text formats, analysis code can also be substantially simpler as well as being uniform across different data formats. These benefits of reduced code complexity and greatly increased performance allow users much greater freedom to explore their data.

Original publication

DOI

10.1186/1471-2105-14-356

Type

Journal article

Journal

BMC bioinformatics

Publication Date

01/2013

Volume

14

Addresses

University of Edinburgh, King's Buildings, West Mains Road, Edinburgh, EH9 3JT, UK. jerome.kelleher@ed.ac.uk.

Keywords

Animals, Humans, Drosophila Proteins, Random Allocation, Computational Biology, Genomics, Genome, Human, Automatic Data Processing, Computer Simulation, Software, Libraries, Digital, Databases, Factual, Genome, Insect, Search Engine