Commonwealth Computer Research Inc — Department of Defense SBIR Phase I: The US military and intelligence community has been successfully fusing the data it gather

Description

The US military and intelligence community has been successfully fusing the data it gathers into actionable intelligence. However, the volume of data is increasing such that it cannot be processed on a single server, calling for distributed data fusion algorithms that operate across a cloud. As data grows to the point of requiring distributed storage, machine learning algorithms capable of producing situational awareness must rise to the challenge of working with distributed storage as well. The problem is to design distributed fusion algorithms which not only do as well as single-server solutions, but which leverage larger volumes of data to produce higher quality analytics. This proposal outlines an architecture that works with distributed data sources without needing data to be directly shared between compute nodes. Data fusion without shared memory is a difficult task; however we develop techniques to minimize the amount of information sent between nodes while maintaining high quality fusion. We propose to use models for which both model learning and inference can leverage distributed storage and computation. Inference should be fast and detached model instances readily deployable to local servers for real-time use, while maintaining data and model integrity with the cloud.