Currently, Next Generation Sequencing (NGS) has become a widely used technology to identify variations associated with the disease for research and diagnostic. The variant analysis workflow on NGS data yields text files in VCF format. This format is inefficient to query large cohorts. To solve this problem, one uses one of the following technologies: 1) ready-to-use variant management systems (e.g., GTRAC, GenomicsDB, Gemini); 2) native relational database management systems (e.g., MySQL) or 3) NoSQL database systems such as Clickhouse or MongoDB.

In this poster we compare the performance of different systems in these categories (GTRAC, GenomicsDB, Gemini, MySQL, and Clickhouse) and present best practices and recommendations.

We used 1000 Genome Project dataset (1092 VCFs), each includes ~39.7 million variants. Total size of the database is about 76 GB. We used a server with 24-CPUs, 128GB RAM. The query set included: 1D range query to look for variants in a genomic range (chr:start_pos-end_pos), and 2D range queries to search for variants in a range and in group of samples.

Results: GTRAC has the best compression, but it lacks major functions for frequent queries. Clickhouse and GenomicsDB have the least insertion time per sample. Gemini and MySQL require to build indices after populating the tables and after each insertion. All tools in general have acceptable query/retrieval time. MySQL has best query time due to best indexing, but its space consumption and the insertion time is prohibitive for huge datasets. Comparing Clickhouse, Gemini and GenomicsDB, we observe that Clickhouse performs slightly better.

Co-authors: Mohamed El-Kalioby,