91视频APP

Last week, our co-founder and Chief Technology Officer, Simon Knowles, was invited to give a talk at the 3rd Research and Applied AI Summit (RAAIS) in London. RAAIS is a community for entrepreneurs and researchers who accelerate the science and applications of AI technology. The event is organized annually by Nathan Benaich, a London-based investor, technologist and scientist focused on companies creating transformational solutions for the world鈥檚 most challenging problems.

Let me start off having you remind us how you define an Intelligent Machine?

Let鈥檚 imagine you built an autonomous car which drove more safely than a human, but nevertheless one day it crashed. This event does not imply a lack of intelligence. After all, humans crash. The machine may have made the best driving decisions possible, given the information it had. But if you try it again with exactly the same test scenario and the machine crashes in exactly the same way, then you probably wouldn鈥檛 regard it as intelligent, because it didn鈥檛 learn from the first experience. So here鈥檚 my attempt to define intelligence in one sentence: the capacity for rational judgment based on imperfect knowledge, adapted with experience. Intelligence is required where perfect results aren鈥檛 computable, usually because there鈥檚 not enough data available. And there has to be some learning component for a machine to be considered intelligent.

We hear a lot about chips designed for training and chips designed for inference. In this context, what is the 91视频APP IPU designed for?

I don鈥檛 think it is helpful to think of there being a hard distinction between training and inference, from a computational viewpoint. Any intelligence computer requires just two components: an inference engine and a knowledge model. Learning is a form of inference, in which we are trying to condense experience (data) into the knowledge model. We do this by inferring the parameters and perhaps the structure of the model from the training data. We can then use the learned model to infer some unknowns from some new input data, such as what motor actions to take in response to incoming video frames.

So training is inference which adapts the model, and there is also inference which does not adapt the model, but computationally they are very similar. We should not need two types of machine architecture.  There might be large machines and small machines, but they can have the same architecture and run the same software efficiently.

As I said earlier, there is no intelligence without learning. But it鈥檚 possible that some 鈥渆dge鈥� machines could delegate their learning function to a computing 鈥渃ore鈥�. For example, an autonomous car might send condensed sensor data back to the cloud and receive periodic model updates.  Personally, I think this is a mode which may justify machines of different capacity, but not machines of different architecture. The great beauty of having a common architecture is that all the software can be common.

You mentioned in your talk that you shouldn鈥檛 design a processor today just for neural networks. Instead, you claim that you need to have a 20 year time horizon. How do you do that?

Inventing a new processor architecture and rendering it as a leading-edge chip is difficult and expensive. Building all the software on top to make it useful and easy to use is similarly difficult and even more expensive, so unless it鈥檚 going to be useful for 20 years or more, don鈥檛 bother.

If you had set out to build an intelligence processor in 2005 you might have focused on the computational needs of support vector machines. By 2010, you would probably have been more concerned with random forests. Today, the high fashion is deep neural networks.  What will be the leading paradigm in 5 years time?

When we started thinking about building a machine to accelerate intelligence processing at 91视频APP, we knew that we had to look beyond today鈥檚 deep neural networks. Neural networks are clearly important, as are many of the earlier paradigms. Our IPU has to out-perform GPUs and CPUs at all these tasks. But perhaps more importantly, it has to provide a flexible platform for the discoveries yet to come.

So we have tried to stand back from specific algorithms, and to consider what is fundamental about intelligence as a computational workload. We knew that if we successfully captured the fundamentals in our design, then our IPU would automatically perform well on neural networks et al.

So, what is the fundamental workload for machine intelligence compute?

In all forms of machine intelligence computation we are trying to infer knowledge models from data and then to use these learned models to infer new results from further data. So to a large extent the workload is defined by the nature of the knowledge models.

Knowledge models and the algorithms required to perform inference on them are most naturally represented as graphs. A graph comprises vertices connected by edges. The vertices contain the learnable parameters of the model, and also some code which defines how to update those parameters in response to new inputs, and what derived information to provide to other vertices so that they may update in turn. The edges define causal pathways of information flowing through the model.  In a deep neural network, the edges often pass tensors which are stacks of feature maps. In a Bayesian net or a Markov random field they pass probability distributions.

The first relevant characteristic of these graphs is that they are large, often thousands or millions of vertices, so there is scope for a parallel processor to achieve high performance by computing on many independent vertices at once. This is serendipitous, because to build higher performance processors in this age of power-limited silicon we must use parallelism not speed.

The second characteristic of the models is that they are sparse 鈥� most vertices are connected to only a small subset of the other vertices. A machine designed for intelligence processing must recognise this sparsity architecturally, because it has a major impact on memory access patterns and communication patterns between processors.

In fact the effective sparsity of memory access increases when we embed graph models in physically realizable memories. This is because the graphs usually have high 鈥渇ractal dimensionality鈥�, and the reduction in dimensionality when embedded in a linear address space induces sparsity. For example, a vertex in the graph might be connected to 100 neighbours which are equally close. But if I store the state of that vertex in a memory with one address, it can have only two immediate neighbours. The effect of embedding a high-dimensional graph in a low-dimensional memory is to cause neighbours to be scattered across the memory. This is a latent problem for very wide vector machines such as GPUs.  Processing on graphs favours much finer-grain machines which can scatter and gather data efficiently.  This is why IPU has many more processors than a GPU, each designed to process narrower vectors.

The third fundamental characteristic of knowledge models is that they capture statistical approximations of their data experience. They are usually stochastically learned from noisy and incomplete data, and when we use them it is to obtain approximate inferences. A consequence is that individual model parameters and activation states are never very accurately resolved. A large ensemble of them in a model can work together to deliver a highly resolved output, but no single internal component is precise.  From a machine design viewpoint this favours low-precision floating-point arithmetic. In fact, this is probably the first major application for very high performance computing on very low precision data. It鈥檚 the polar opposite of traditional High Performance Computing (HPC).

Another fundamental is the re-use of model parameters. Convolution is all about spatial re-use and recurrence is all about re-use over time. Such re-use captures spatial or temporal invariances in the data, but equally important is that if you re-use parameters, you expose them to more data so they learn faster.

The last fundamental characteristic I鈥檒l mention is that the graph structures can be regarded as static 鈥� at least for long periods of time. We don鈥檛 need to change the connections in our knowledge models during computation, just like you don鈥檛 rewire the ethernet cables in a datacenter when you change applications, and the axons in your brain don鈥檛 rewire themselves as you think (actually they do, but only very, very slowly).

Having a static graph is vital to building efficient highly-parallel machines, because it allows the compiler to do three NP hard tasks involved in mapping a parallel program to a parallel machine:

• balancing computational work across processors,
• partitioning and allocating memory,
• scheduling message traffic between processors.

There are many other fundamental characteristics of the machine intelligence workload that we have considered; I mentioned a few more in my talk.

You said in your talk that 91视频APP takes a different approach to memory. Could you explain?

For machine intelligence workloads, the performance of today鈥檚 GPUs and CPUs is limited by their bandwidth to external memory. If you invented a GPU-like processor with 10x the arithmetic performance, there is no known technology which will deliver the necessary 10x in external memory bandwidth to keep up with it. To achieve a step change in system performance we have to do something different.

Traditional external DRAM systems, such as DDR4 or GDDR5, are already left behind because bandwidth is too low and not scaling. They are being replaced with High Bandwidth Memory (HBM), which uses vertically stacked DRAM die physically very close to the processor die, on a silicon substrate with very high wiring density. This is what today鈥檚 state-of-the-art GPUs are using. It is 鈥渉eroic鈥� technology, very difficult to manufacture at good yield. Also it brings the DRAM into the thermal envelope of the processor, which is bad news for leakage and steals power from the processor budget, so the processor must run slower. Today鈥檚 second-generation HBM2 systems are struggling towards 1TByte/s, but this is still a hard limit on compute performance.

The radical alternative is to bring the model memory onto the processor chip. Allocating, say, half the die area to SRAM delivers an order of magnitude improvement in bandwidth and latency over an HBM2 external memory system. Then partitioning this SRAM into many blocks distributed across a massively parallel processor-on-chip yields another order of magnitude.  This is the approach we have taken with the IPU.

You obviously can鈥檛 integrate as much SRAM on an IPU as you can attach as DRAM to a GPU.   But we can fit plenty enough to hold large and complex machine learning models entirely on chip. And the ability to access them at 100x the bandwidth and 1/100^th the latency delivers great performance dividends. The final part of the jigsaw is that IPUs are designed to be clustered together such that they appear to software like one larger chip. So if you need huge models you can distribute them over multiple chips.

What if we still run out of memory? It turns out there鈥檚 huge scope for reducing the memory footprint of machine intelligence processing, not least because several of the techniques currently used to mitigate DRAM bandwidth and latency problems cause memory bloat.

For example, GPUs use a transformation called 鈥渓owering鈥� to turn convolutions into matrix-matrix multiplication. And they require large training batches to reach their best efficiency.  Each of these can bloat the memory by an order of magnitude. IPUs require neither.

We can save another order of magnitude in memory footprint by trading re-computation for memorization. For example, most of the memory used during DNN training on a GPU is to save activations on the forward pass, so that we can use them during back-propagation. An efficient alternative is to just save periodic snapshots, then re-compute the missing activations from the nearest snapshot when required. This trade provides an order of magnitude memory footprint improvement for around 25% increase in compute. An IPU has less memory but far greater compute than a GPU, so it鈥檚 a good trade.

Most importantly, all these memory optimizations can be done 鈥渦nder the hood鈥� by the compiler, so they don鈥檛 require clever programming.

One of the takeaways from your talk is that you provided a new definition of what it means for a chip to be efficient 鈥� 鈥�Silicon efficiency is the full use of available power.鈥� You鈥檝e mentioned keeping the memory local. How else do you maximize use of the available power?

One way is by serializing computation and communication, which may seem counter-intuitive. Surely it鈥檚 more efficient to overlap them? Not necessarily so in a power-limited world.

Consider the design of a new multi-processor. You aim to overlap inter-processor communication with computation, so you allocate your power budget between these two according to your expected workload. Then you run an application which has a different balance and the result is that the program executes in the same time but consumes less power, because either the communication or the computation part was bottle-necked and the other under-used its power budget.

Now consider an alternative design, in which you assume that computation and communication do not overlap, so you provision them both to use all the chip power. Now whatever the balance of a real workload, the program has the potential to complete in minimum time, at constant full power. In this age of 鈥渄ark silicon鈥� this is a more efficient design.

You also talked about Bulk Synchronous Parallel. How does that fit in?

An IPU machine is a pure, distributed, parallel processor, so we can build little ones for embedding in robots and big ones for data-centres and, in the future, huge ones and it will still be the same machine to software.

An IPU consists of very many high-performance, independent processors, each with their own local memory, which can send messages to each other through a perfect stateless interconnect. This interconnect allows an IPU to support any type of knowledge model and inference algorithm, including ones that we haven鈥檛 thought of yet.

The static nature of the computation graph allows the compiler to divide the program into independent sub-programs running on the many processors. But we need a mechanism for maintaining overall program order, so that messages are sent between the sub-programs at the right time.

There is a paradigm for doing this: it鈥檚 called Bulk Synchronous Parallel (or BSP), and was invented in the 1990鈥檚 by Leslie Valliant and Bill McColl. It works by alternating between a 鈥渃ompute phase鈥�, in which every processor just does compute on its own local memory, and an 鈥渆xchange phase鈥� in which they just send and receive messages, memory-to-memory. Notice that this serializes computation and communication, which is now efficient as discussed earlier. Once around each pair of phases, the whole machine synchronises so all processors remain in step. There is hardware to make this synchronization super-cheap, on chip as well as across chips in a cluster.

The beauty of BSP is that it effectively creates a sequential machine at the whole-program level, but supports massive parallelism in each of the steps. It鈥檚 also the only parallel program execution paradigm which is automatically free of concurrency hazards, so you can鈥檛 write a program with races or deadlocks or livelocks.

The sequence of compute and exchange phases is extracted from the graph automatically by the compiler, so the BSP mechanics are not something the programmer has to think about. It鈥檚 just the IPU way of mapping a graph program to an efficient, scalable, and safe massively-parallel machine.

I know that you have lots of potential customers who are eager to try your IPU technology 鈥� when will we be able to get our hands on an IPU?

We will be shipping products to our early access customers before the end of this year and more general availability will start early next year. We are looking forward to helping engineers deliver the first generation of really intelligent machines and providing researchers with a more flexible and potent engine for the discovery of future general intelligence models and inference algorithms.

Benaich is an investor, technologist and scientist focused on companies creating transformational solutions to the world鈥檚 challenging problems. Previously a partner at Playfair Capital, Nathan invested in 33 companies including Mapillary, Ravelin and Starship Technologies. Nathan previously earned a Ph.D. in computational cancer biology at the University of Cambridge and a BA from Williams College.

You can watch the talk in full, below: