THIN-Bayes

Neural networks have been shown to provide rich and complicated inferences from time-series data over first-principle approaches. However, with inference moving to the edge, and IoT platforms shrinking, realizing AI-based inference on-board is challenging. While communication bandwidth, energy budget, and form factor of these platforms have gone down, the workload and complexity of neural networks have skyrocketed, requiring systematic software and tools to guide on-board TinyML implementation. Keeping the challenges in mind, we developed THIN-Bayes, a completely open-source black-box optimization framework for training and deploying ultra-lightweight models on extremely resource-constrained platforms using TensorFlow Lite Micro.

THIN-Bayes is designed based on three insights: 1. hardware proxies deviate significantly from true values for low-end microcontrollers, thereby requiring exact hardware metrics; 2. the linear programming formulation stemming from hardware-in-the-loop neural architecture search is not gradient-friendly. 3. the type of model should also be a hyperparameter for low-end microcontrollers. Based on ARM Mango, THIN-Bayes features a tightly integrated ultra-lightweight model zoo and a gradient-free, hardware-in-the-loop, and parallelizable Bayesian neural architecture search framework, receiving hardware metrics directly from the target hardware during optimization. We demonstrate the efficacy of our framework by optimizing neural-inertial navigation models and earable human activity detection models for microcontrollers, two of the most challenging applications of inertial sensors. The inertial-odometry models found by THIN-Bayes are 31-134x smaller than state-of-the-art neural-inertial navigation models, while the activity detection models are 98x smaller and 6% more accurate over the state-of-the-art. THIN-Bayes is an important step towards bringing in challenging AI applications to TinyML platforms.