SOAR TECHNOLOGY, LLC — Department of Defense SBIR Phase II: SB163-007

Description

Extending research on scalable Transformer models performed on previous SBIR-funded work for DARPA, Soar-Tech proposes an investigation into Scalable Language Association Models for Domain-specific Utterances in Novel Contexts (SLAMDUNK), a Phase II SBIR research effort focused on developing novel synthetic data generating pipelines and extending these pipelines with novel index-learning, token grouping, sparse approximation, and model mixture techniques that will enable orders-of-magnitude increases in the performance of Transformer models and will explore the application of these scalability techniques in improving the performance of emerging multilingual Transformer models. SLAMDUNK addresses core scalability issues in modern language models including applying index learning to accelerate performance when in search and retrieval applications, applying token grouping and sparse approximation methods to reduce the number of floating-point operations required to execute language models, and applying mixture techniques to combine results obtained in smaller language models. We anticipate SLAMDUNK to result in research that demonstrates a dramatic reduction in the time and cost required to apply Transformer models in both English and multilingual information processing domains.