Accelogic, LLC — Department of Energy SBIR Phase I: 27a

Description

Millions of dollars are spent every year in generating the tons of data that are necessary for addressing nuclear physics grand challenge problems at DOE facilities- Due to inherent technical and financial limitations at the “last- mile” of the data delivery pipeline, only a small fraction of these expensive Peta bytes of data is available for immediate access by end-users at any given time- This not only renders the users’ work inefficient, but also risks the waste of significant amounts of information that could very well contain the keys to answering grand challenge questions- This project aims at solving this issue by providing the ground-breaking data compression technology able to multiply the capacity of existing (and future) last-mile “live storage” facilities by factors larger than 4x and up to 9x without any additional investment in hardware- The technology, which will be packaged as an open-source set of tools, will be integrated with ROOT/IO and will provide the users with full control in the form of easy to understand/use/modify/adapt according to the specific needs and technical sophistication of each user- The data compression technology is based on a novel and very rich theory of application-driven losslesscompression that systematically analyzes the data in search of hidden redundancies that are not easily detected by traditional general-purpose compression- In addition, the theory provides fundamental mechanisms for achieving unprecedented lossless compression factors from a thorough understanding of the peculiar —and rich— interdependencies among the precisions of the data- This theory is complemented by very efficient and portable software implementations that we have developed in the last five years and which have already made paradigm- shifting impact of programs at NASA and DoD- Those implementations include the fastest compressors/decompressors and highest compression factors in the world- Already successful for network-data and in-memory compression, this project will realize the power of the proposed data compression technology for nuclear physics I/O and storage-