LYNNTECH INC. — Department of Defense SBIR Phase I: N231-037

Description

The Navy’s Naval Sea Systems Command (NAVSEA) has the need to be able to immediately identify unmanned airborne systems (UAS) of interest while understanding their environments. Some UAS of interest are lacking in diverse exemplary images both in term of point of view and environment. Furthermore, visible images alone may be insufficient to achieve NAVSEA’s performance targets, necessitating the usage of thermal infrared (TIR) image sensors as well. Currently, many of the generative modeling techniques are applied solely to the visible wavelengths. Generative modelling approaches using Generative Adversarial Networks (GANs) have shown some modicum of success augmenting visible image datasets. Hence, there is a need to create a GAN for RGB+TIR images of UASs that could augment current training sets in a functionally useful way in a variety of contexts. The final generative output would need to be adaptable to accept various formats of imagery, with varying resolutions and sensors. Various quantitative and qualitative metrics should be identified and explained, with priority to predicting utility of generated datasets for machine learning applications. Lynntech’s Intelligent Systems group has several years of experience researching and developing GANs for applications involving various IR image bands, which is a challenge due to the emissive nature of thermal IR component, and the lower spatial resolution, yet greater bit depth of the IR sensor data relative to visible cameras. Therefore, based on this and the need of NAVSEA Lynntech, Inc. proposes to develop a Synthesizer of Queryable Unmanned Aerial System Heterogenous EO/IR Datasets (SQUASHED) which is a scalable software suite capable of creating (1) multi-group UAS target extraction and pose estimation,  (2) posed-based instance segmentation-map generation with various backgrounds, (3) photorealistic UAS insertion to novel instance backgrounds with a generative adversarial network (GAN) while blending with the different imaging/environmental conditions of the background and/or sensor characteristics of the camera, and (4) domain transfer of different imaging modalities (thermal, RGB) with a conditional GAN. The focus of the SQUASHED R&D effort is to realize a tool that can fabricate image frames of relevant UAS platforms, with a goal to ultimately produce mission-focused synthetic data that has more training utility than available sets. This synthetic dataset will be queryable in the sense that it with be extensively labeled according not only according to class type, but many other characteristics and metadata, i.e., orientation, pixels on target, and even weather. The ultimate utility of such a scalable SQUASHED resource will be evaluated by the specificity of any derivative machine learning trained with this resource on evaluation sets chosen by NAVSEA.