Legion: Tailoring Grouped Neural Execution Considering Heterogeneity on Multiple Edge Devices

Abstract

Reducing the inference latency by distributing the tasks that cannot be handled by a single edge device across multiple edge devices can be a promising solution to meet tight deadlines. Several technologies have been proposed for this purpose, and most focus on determining an optimal fused layer configuration. However, with deep learning networks becoming deeper and the emergence of services requiring fast deployment, the rapid search for optimal fused layer configurations has become more important. This study proposes Legion, a lightweight optimizer for the fast deployment of neural network operations into multiple edge devices. The Legion reasonably improves latency by determining an optimal fused layer configuration with minimal profiling execution using multiple key features of a predictor, wild card, and saturation target candidate selection. The Legion exhibited up to 39.8 times faster optimal fused layer configuration search with only a 3.9% performance difference compared to a full search-based configuration search for the execution of several popular networks on a maximum of six RK3399-based system boards.