The Promise of ONNX
A Machine Learning framework is a tool that enables a developer to create models to achieve a given task such as image classification or natural language processing. Some of the most familiar examples of ML frameworks are PyTorch and TensorFlow.
ONNX (Open Neural Network eXchange) is an open-source standard that defines a common set of operators and a file format that enables AI developers to use models with a variety of frameworks, tools, runtimes, and compilers.
In this post I will talk about:
Why was ONNX created?
What is ONNX?
The Promise of ONNX
Fulfilment of the Promise
Why was ONNX created?
The advent of Machine learning/ Deep learning caused an exponential increase in the number of frameworks/ libraries that were introduced to handle the execution of deep learning/ machine learning models.
This increase was primarily driven by,
(a) Hardware vendors who developed their own frameworks to achieve vertical integration and make it easier to develop models for their hardware
and
(b) Software vendors who developed their own frameworks to achieve the best performance for specific use cases
This explosion in the number of available frameworks to develop deep learning models led to the fragmentation of the ecosystem and made working across frameworks or hardware ecosystems difficult. A choice had to be made on where the models would be developed and trained and moving to a different framework or a different hardware ecosystem later in the data science pipeline was near impossible. This was due to the compatibility of frameworks with compilers for different hardware but we will save the compiler story for another time.
ONNX was born to remove this inertia in interoperability between frameworks and between different hardware ecosystems.
Throughout the rest of this article, we will explore if the solution to this problem of too many frameworks is to introduce another framework into the mix?
What is ONNX?
ONNX is an open specification that consists of definitions for the following:
File format for a computational graph model
Model: Is an Acyclic computation data flow graph that holds version information and metadata.
Graph: Contains the name of the data flow graph in the model, a list of computation nodes that form the graph, inputs and outputs to the graph.
Computation Node: Each computation node in the graph that takes zero or more inputs of defined types, one or more outputs of defined types and the type of operation the specific node performs on its inputs.
Standard data types
ONNX as a standard supports the following list of datatypes:
Tensor type:
- Int8, Int16, Int32, Int64
- Quantized Int
- Uint8, uint16, uint32, uint64
- Float16, float, double
- Bool
- String
- Complex64, complex128
Non-tensor types:
- Sequence
- Map
- Operators (Built-in/ Custom)
Operators
Each computation node in the ONNX Graph performs a specific operation and produces one or more outputs. The ONNX standard defines the official list of operators that are supported by ONNX graphs. This list of operators is constantly evolving and is kept up to date through ONNX Opsets.
Each incremental ONNX Opset adds support for new operators and improves the existing list of operators.
This specification is meant to be an open standard that everyone can agree upon and hence enable translation from any framework to ONNX.
The Promise of ONNX
The core design philosophies of ONNX are:
- Interoperability
- Portability
ONNX intends to be this intermediary between different frameworks by providing an efficient interface to translate models from one framework to another. Figure 4 below lists the tools that convert models to ONNX format from some of the most common frameworks.
The Fulfilment of the Promise
In reality, the conversion of models from any framework to ONNX has been a challenge. One of the primary hurdles is the difference in surface area of these frameworks.
The challenge for the conversion libraries is keeping up with the shifting landscapes of both ONNX and their source libraries for every release (~4–6 times a year)
Conversion of models from one framework to another boils down to being able to represent the underlying math operations from the original model. Figure 5 shows the number of unique operators defined in each framework. ONNX supports ~13% of the operators supported by PyTorch. This significantly limits the number of models that can be actually converted to ONNX.
It is important to note that 1000+ operators on frameworks like PyTorch also include a degree of duplication in the functionality in specific ops and also begs the question, should frameworks like ONNX support the global superset of all operations?
To solve this issue of operator coverage ONNX has adopted a different strategy rather than chasing the never-ending list of operators on all these frameworks.
ONNX Functions help express complex operations using primitive/ existing operators in ONNX. Every upcoming ONNX release is going to be adding new functions and it is easier than ever to add your own functions as needed. This helps limit the number of operators that need to be maintained and also benefits compiler developers by reducing the scope and effort involved to optimize a smaller number of operators for their hardware.
Operators are the basic building blocks used to define ONNX models. With a rich set of operators, ONNX can describe most DNN and ML models from various frameworks. Functions enable expressing complex operators in terms of more primitive operators. The ONNX specification includes a core set of operators that enable many models. It is a non-goal to add all possible operators, however more operators are added as needed to cover evolving needs.
I cannot overstate the resemblance of the above with the CISC Vs RISC wars of the previous era. We not only have the front row seats to see how this plays out but also the opportunity to truly influence the direction of this impactful industry that is already shaping the future of technology by contributing to open source projects and pushing the boundaries of the AI software.
What a time to be alive!!!
