Today’s deep learning networks have accomplished a great deal but are running into fundamental limitations – including their need for enormous compute power. A large, complex model can cost millions of dollars to train and to run, and the power required is growing at an exponential rate. New algorithms are essential to break through this performance bottleneck.
California-based Numenta this week announced a major breakthrough, based on a principle of the brain called sparsity. Researchers at the company developed a new algorithm by comparing “sparse” and “dense” networks (illustrated above) for a speech recognition task, using the Google Speech Commands (GSC) dataset. The team ran their programs through field-programmable gate arrays (FPGAs), a type of integrated circuit designed to be configured by a customer or designer after manufacturing, supplied by Xilinx.
Based on the metric of words processed per second, the sparse networks yielded more than 50 times the acceleration over dense networks on a Xilinx Alveo circuit board. In addition, Numenta demonstrated their network running on a Xilinx Zynq – a smaller chip where dense networks are too large to run – enabling a new set of applications that rely on low-cost, low-power solutions. As a result, when measured by words per second per watt, Numenta has shown a 2600% saving in energy efficiency.
This proof-of-concept demonstration validates that sparsity can achieve substantial acceleration and power efficiencies for a variety of deep learning platforms and network configurations, while maintaining competitive accuracy. It provides an exponential improvement in terms of larger and more complex networks using the same resources.