BNN

Project Details

Project Lead
Minje Kim 
Project Manager
Minje Kim 
Project Members
Mrinmoy Maity, Brahmendra Sravan Kumar Patibandla, Kai Zhen, Sun Woo Kim, Sanna Wager, Aswin Sivaraman, Lijiang Guo  
Institution
Indiana University Bloomington, School of Informatics and Computing  
Discipline
Computer Science (401) 

Abstract

This research aims at proving the conjecture that there is an efficient bitwise version of neural networks and we can learn such a network that spends a minimal amount of resources in small device, whilst still enjoying the stat-of-the-art performance. A growing need for embedded systems is to efficiently run an Artificial Intelligence (AI) engine on the chip, so that it can process and analyze various kinds of real-time sensor signals around the device to predict the context, behavior, and status of the user. However, the state-of-the-art AI depends largely on the complex models, e.g. Deep Neural Networks (DNN) with easily over millions of parameters, whose run-time operations are burdensome for an embedded system with often constrained resources such as power and memory. The proposal explores Bitwise Neural Networks (BNN) to achieve both performance and efficiency goals. BNN is a transformative research direction that can entirely replace the fixed- or floating-point variables and the operations on them with binary variables and bitwise arithmetics. For example, every participating BNN parameter is represented with a bipolar binary value, i.e. +1 and -1. Consequently, computations on them are much more efficient in hardware, e.g. multiplication between real numbers is replaced with a single bitwise operation, XNOR. To prevent the possible performance degradation due to the extremely condensed nature of BNNs, one of the goals in this research is to develop training strategies for BNNs to maximize their performance, whose upper bound is that of a comprehensive DNN. At the same time, another goal is to demonstrate their efficiency by implementing BNNs on simple FPGA boards. The performance of the proposed systems will be tested on various real-time tasks, such as keyword spotting, human activity monitoring, signal denoising, etc.

Intellectual Merit

The first intellectual merit is to advance the foundational technology to train a BNN, which is an entirely different combinatorial optimization problem from training a usual neural network. Although deep learning is one of the most active research areas in AI and machine learning, research efforts to train a more efficient network for the run time got a relatively less attention, since it has been considered as an implementation issue. Meanwhile, some recent preliminary research results on small benchmark datasets are promising in training such a network with discrete parameters. Yet, a fully bitwise Recurrent Neural Network (RNN) to process real-time sensor signals is largely unexplored. Another intellectual merit comes from the hardware system that will demonstrate the cost-efficiency of the proposed network, which will serve as a practical prototype technology particularly simulating the resource-constrained devices.

Broader Impacts

This project promotes the critical technologies to improve the efficiency of AI systems, so that they are more widely affordable in small-scaled systems that tightly interact with human, such as wearable and pervasive devices. The embedded systems will also serve as a powerful building-block to enrich the potential of related technologies, e.g. Internet of Things and Cyber-Physical Systems. Many health-related applications are time- and resource-sensitive (e.g. medical monitoring chips on body, speech enhancement on hearing-aids, etc), so they can benefit from the results of the research activities, too. In a long term, the research can be a stepping stone to the future AI system, which is not a big black box remotely hidden in the background anymore, but a more distributed and robust network of various intelligent units. It consists of many cheaper responsive front-end nodes and some more comprehensive ones in the cloud that make a higher-level decision with less burden. As a faculty member of the new Intelligent Systems Engineering (ISE) department, the PI will participate in developing the curriculum to incorporate the proposed research. We are also establishing the Research Experiences for Undergraduates (REU) activities within the School of Informatics and Computing (SoIC), where the PI plans to mentor undergraduates in the summer in the areas related to the proposal. The new ISE Undergraduate program also stresses academic year undergraduate research and the PI will be actively involved in this by establishing the details for the first students entering this fall. As an SOIC faculty member, the PI has a commitment to women and minority students. Hence, the PI will ensure that the undergraduate activities address under-represented communities.

Scale of Use

I need a few GPU units (the more the better) for my ongoing experiments on BNN. This won't occupy a lot of main memory or CPUs, but the GPU allocated to me might be fully occupied from time to time. Since this is an ongoing project, I'd like to use those units up to 6 months if possible.