Paper: IEEE Access – DNN partitioning for inference throughput acceleration at the edge

Thomas Feltin February 14, 2023 0 Comments

Intro

I am very excited to present this work, published in the IEEE Access journal, which presents an alternative to standard AI workload acceleration mechanisms at the edge (hardware acceleration, model compression, cloud off-loading). This work, in collaboration with Cisco, presents tools and mechanisms to enable inference throughput acceleration by leveraging the available compute of idle edge devices, hereby improving performance, preserving privacy by keeping computation in a trusted edge environment, and saving on hardware expenses and waste of computational resources.

The paper: DNN partitioning for inference throughput acceleration at the edge

Deep neural network (DNN) inference on streaming data requires computing resources to satisfy inference throughput requirements. However, latency and privacy sensitive deep learning applications cannot afford to offload computation to remote clouds because of the implied transmission cost and lack of trust in third-party cloud providers. Among solutions to increase performance while keeping computation on a constrained environment, hardware acceleration can be onerous, and model optimization requires extensive design efforts while hindering accuracy. DNN partitioning is a third complementary approach, and consists of distributing the inference workload over several available edge devices, taking into account the edge network properties and the DNN structure, with the objective of maximizing the inference throughput (number of inferences per second). This paper introduces a method to predict inference and transmission latencies for multi-threaded distributed DNN deployments, and defines an optimization process to maximize the inference throughput. A branch and bound solver is then presented and analyzed to quantify the achieved performance and complexity. This analysis has led to the definition of the acceleration region, which describes deterministic conditions on the DNN and network properties under which DNN partitioning is beneficial. Finally, experimental results confirm the simulations and show inference throughput improvements in sample edge deployments.

Citation

@ARTICLE{10042405,
author={Feltin, Thomas and Marché, Léo and Cordero-Fuertes, Juan-Antonio and Brockners, Frank and Clausen, Thomas H.},
journal={IEEE Access},
title={DNN partitioning for inference throughput acceleration at the edge},
year={2023},
volume={},
number={},
pages={1-1},
doi={10.1109/ACCESS.2023.3244497}}