E-Science Products

Software products maintained by the E-Science group. Most products are tools for the publication and analysis of scientific data bases. However we also maintain tools for accessing VO compatible services.

This is a list of GitHub repositories with software developed in the AIP E-Science group.

Web Development

Data Warehousing

  • DBIngestor: The DBIngestor library provides wrappers to data ingestion into various DB systems, such as MySQL, SQLite3, and ODBC compatible systems. Use this C++ library to ingest data in any format by only implementing the appropriate reading routines. The library provides an ingestion buffer that allows for multiple rows to be ingested at one time for better performance. Further assertion and conversion functions are provided for elementary data transformation and processing. These can easily be extended by custom functions. Get DBIngestor from GitHub at: https://github.com/adrpar/DBIngestor
    Extensions for the binary file formats FOF and PMss are available at https://github.com/kristinriebe/FofIngestor and https://github.com/kristinriebe/PmssIngestor.
  • AsciiIngest: A general ASCII file row based ingestion tool using the DBIngestor library. This tool allows definition of custom ASCII data formats similar to the MS SQL Server format files. However it is possible to define custom checks, transformation functions, derived data rows, and constant data rows. It is easy to extend the assertion and convertion functions, by implementing ones own functions in the DBIngestor library. Further the DBIngestor library provided data buffering for higher ingest performance and many interfaces to various DB systems. Get AsciiIngest from GitHub at: https://github.com/adrpar/AsciiIngest
  • PaQu: A parallel querying facility for MySQL that works together with the MySQL Spider engine built on a fork of shard-query. PaQu allows transparent parallel SQL queries on TB sized database tables distributed in a cluster of MySQL databases. For a description of the PaQu concepts see the Cosmosim documentation. Get PaQu from GitHub at: https://github.com/adrpar/paqu

Developments for MySQL/MariaDB

  • MySQL Query Job Queue: This MySQL daemon plugin provides a job queue to which SQL queries can be submitted. These will then be executed according to priority and number of already running queries. This plugin is fully supported by Daiquiri. Get the MySQL Query Job Queue from GitHub at: https://github.com/adrpar/mysql_query_queue
  • MySQL Sphere: A port of the widely used pgSphere 1.1.1 library to MySQL. This library is especially important in the implementation of a VO TAP interface for MySQL. Get MySQL Sphere from GitHub at:  https://github.com/adrpar/mysql_sphere
  • MySQL SPRNG: An implementation of the parallel pseudo random number generator SPRNG for MySQL clusters. Get MySQL SPRNG from GitHub at: https://github.com/adrpar/mysql_sprng
  • MySQL Dump VO: A fork of the mysqldump tool distributed with MySQL 5.5.28 supporting the dumping of MySQL database tables in the VOTable 1.3 (ASCII and BINARY) format. Get mysqldump-vo from GitHub at: https://github.com/adrpar/mysqldump-vo
  • AIP E-Science MySQL UDF collection: A collection of useful UDFs such as angular distance, hilbert curve key generation, distributed STDDEV calculation, sleep, and a strrpos implementation. Get the UDF collection from GitHub at: https://github.com/adrpar/mysql_udf
  • MySQL Validate SQL UDF: Plugin for exposing the SQL validation codes from within MySQL without executing sub queries first. Get the Validation UDF from GitHub at: https://github.com/adrpar/mysql_validateSQL


  • LibHilbert: Library for N-Dimensional Hilbert Mapping providing an implementation of the Chenyang, Hong, Nengchao 2008 IEEE algorithm. For better performance, the Hilbert generating genes are statically compiled into the library, allowing for up to 20-dimensional Hilbert Keys. Get LibHilbert from GitHub at: https://github.com/adrpar/libhilbert