Curious about what an NPU (neural processing unit) is and what it does? Here's the lowdown, including how it stacks up against other processors and why it's becoming a game-changer.

What Does NPU Mean?

NPU stands for neural processing unit.

It's a specially designed and manufactured piece of silicon that can accelerate machine learning calculations faster than a more generalized central processing unit (CPU) or graphics processing unit (GPU).

Although they usually appear more in datacenters and servers, NPUs are now appearing in recent consumer processors to accelerate local calculations of artificial intelligence (AI) software, like large language models, similar to ChatGPT.

What Are NPUs Used For?

NPUs have been popping up in a range of devices for years, including Apple's iPhones and iPads, "Copilot+" PCs and laptops, and more.

While Apple's Bionic chips with onboard NPUs perform tasks like video stabilization and photo correction, they're now seeing use for generative AI functions such as drawing and text generation. These uses also include responding to real-language questions using large language model AIs.

Typically, these kinds of tasks happen in the cloud due to the extreme processing demands, but the latest generations of NPUs allow for some of those tasks to happen directly on a device.

That efficiency improves (lowers) the latency of their responses and reduces the overall power and data requirements for these tasks, making them faster and less battery-intensive.

NPU vs. CPU vs. GPU vs. SoC

One of the reasons that "NPU" as a term can be a little confusing is because it sounds so much like a range of other terms associated with processors and modern technology. An NPU is not a CPU, though it can be part of an SoC, which can also contain a CPU and GPU. Let's break down these terms to clear things up.

• NPU: A neural processor unit that handles machine learning and AI calculations and is either a standalone chip, or built into a SoC or CPU as a core component of it.
• CPU: The central processing unit and the main general purpose processor within most modern electronic devices that handles the decision making and is the most impactful when it comes to general performance. A faster CPU will help your device do almost everything faster and more efficiently.
• GPU: A graphics processing unit, sometimes called a graphics card in desktop PCs. This handles 3D and video rendering, so it will transcode video when you stream it online, and make your games appear on screen. This can be a standalone chip, a standalone add-in card (graphics card), or be built into a CPU or SoC alongside an NPU.
• SoC: A "system on chip" is a piece of silicon that contains a range of processors and components, effectively offering a full "system" on a single "chip." You'll more commonly find these on compact devices like smartphones and tablets, where the idea is to remove additional wiring between components to improve latency, efficiency, and performance.

Beyond NPU: What to Know About GPNPUs

A more modern concept has emerged in recent years, known as a GPNPU. These processors combine a graphics and neural processor on a single chip.

This consolidation can allow an NPU to use the general-purpose capabilities of a GPU to accelerate AI calculations beyond what the NPU can manage by itself.

Some modern devices have been advertising their TFLOPS (Tera Floating-Point Operations Per Second) calculation speed for AI work as a metric for how fast the device is at AI calculations, and those TFLOPS are often split between the NPU and GPU, as both can contribute.

TFLOPS is a unit of measurement that determines a computer's ability to perform calculations to within one trillion floating-point operations per second. Yeah. That'sfast!

GPNPUs may be the next evolution of that, helping to push AI calculations further than before, and potentially allow the NPU to contribute to 3D rendering with AI calculations for upscaling and frame generation.