Intelligent Automation, Inc. — Department of Defense SBIR Phase I: In recent years data intensive processing is beyond the capability of any individual machi

Description

In recent years data intensive processing is beyond the capability of any individual machine and requires clusters, which means that massive data problems are fundamentally about organizing computations on dozens, hundreds, or even thousands of machines. This is exactly what MapReduce does, and the challenge is really how we can exploit MapReduce to bring valuable information to the Warfighter, for example, in combating terrorism to monitor at risk individuals and groups. In this proposal Intelligent Automation Inc. proposes a multi-layer workflow, initiated by Warfighter questions (plus metadata), which invokes other services, implemented as MapReduce processes in a Hadoop-based ecosystem. This workflow ends back at the Warfighter"s PDA, where ranked meaningful information is presented as a result of a data fusion process, with the system"s end goals being accuracy and speed. We also propose many improvements to the Hadoop kernel including dealing with bottleneck Map jobs, improvements for iterative (as opposed to parallel) tasks, and tandem parallel databases-Hadoop operations.