Blaize GSP Deep Dive: Graph Streaming for Low-Latency Edge AI
Overview
The Graph Streaming Processor (GSP) from Blaize, a US-based edge AI company, takes a different route to efficient inference than either dataflow-SRAM designs or in-memory compute: it executes the neural network as a graph, streaming intermediate results between operations so that activations rarely leave the chip. The first-generation part packs 16 programmable cores delivering a reported 16 TOPS (INT8) in about a 7 W envelope on Samsung 14 nm silicon. Blaize’s company-published comparisons claim roughly 50x lower memory bandwidth, up to 10x lower latency and as much as 60x better system-level efficiency than GPU-plus-CPU pipelines for its target workloads. This deep dive explains the architecture behind those claims, the software stack, and where graph streaming genuinely pays off.
Key takeaways
- The GSP schedules the neural network as a task graph across 16 cores, executing depth-first so intermediate activations stay in on-chip memory instead of round-tripping to DRAM.
- Reported silicon figures: 16 TOPS (INT8) at roughly 7 W on Samsung 14 nm – modest peak compute, spent unusually efficiently.
- Company-published claims of ~50x lower memory bandwidth, ~10x lower latency and up to 60x system efficiency vs GPU-CPU pipelines are vendor comparisons – treat them as directional, not measured guarantees.
- Fine-grained hardware task scheduling lets multiple models share the chip concurrently, which suits sensor-fusion and multi-net pipelines.
- Software is Picasso SDK (C/C++, ONNX ingestion, NetDeploy quantisation) plus the low-code AI Studio; ecosystem breadth trails CUDA by a wide margin.
What the Graph Streaming Processor is
Every neural network is a directed graph of operations. Conventional accelerators flatten that graph into a sequence of kernels executed breadth-first: run layer 1 across the whole frame, write results to memory, read them back for layer 2, and so on. The GSP instead ingests the graph natively. A hardware scheduler decomposes it into fine-grained tasks and dispatches them across 16 general-purpose streaming cores, with data flowing from producer to consumer tasks as soon as partial results exist. The architecture deliberately hides its internals – vector widths and pipeline depths never surface to the programmer – so code targets the graph abstraction, not the microarchitecture.
Depth-first execution: why activations stay on-chip
The consequential design choice is depth-first scheduling. Rather than materialising a full activation tensor per layer, the GSP pushes a tile of data down through several layers of the network while it is still resident in on-chip memory. Intermediate activations – the dominant memory traffic in vision CNNs at high resolution – are consumed almost as soon as they are produced. That is the mechanism behind Blaize’s company-published claim of around 50x lower memory bandwidth than a GPU-CPU equivalent, and it has two second-order effects: power drops, because DRAM traffic is the most expensive thing an edge device does, and latency drops – the reported ~10x – because results for a region of the frame emerge before the whole frame has been processed. Batching, the GPU’s standard trick for utilisation, becomes unnecessary; the GSP is built to be efficient at batch size 1, which is exactly the regime latency-critical edge systems live in.
Reading the published numbers
Sixteen TOPS (INT8) looks small next to a 200+ TOPS datasheet, and that is the point of the architecture argument: headline TOPS measures multiply capacity, while delivered frames-per-second within a power and latency budget measures usefulness. Blaize’s up-to-60x system-efficiency figure is a vendor comparison against a specific GPU-plus-CPU baseline on selected workloads – reasonable as a directional signal that data movement, not arithmetic, dominates edge economics, but not a number to transplant into your own sizing. The ~7 W envelope is reported at the chip level; module and box power (host, decode, IO) is higher. As with every accelerator on this page, insist on measured latency and FPS for your model, at your resolution, at batch 1.
Software: Picasso SDK and AI Studio
The Picasso SDK is the developer path: ONNX and framework-trained models come in through NetDeploy, which quantises (INT8, with mixed-precision options) and converts them to the GSP’s graph-native format; C/C++ APIs and OpenVX-style pipelines handle integration, and the same graph abstraction covers pre- and post-processing so more of the pipeline runs on-chip. AI Studio is the low-code layer – model zoo, transfer learning and deployment workflows aimed at teams without deep embedded expertise. The honest caveats: the developer community is small compared with CUDA or even Hailo’s, operator coverage requires per-model verification, and porting effort is the real cost line in any evaluation.
Products and form factors
- Xplorer accelerator cards – M.2 and small PCIe cards for adding GSP inference to existing industrial PCs and gateways.
- Pathfinder system-on-module – GSP plus Arm host cores for building standalone smart devices.
- Edge appliances and partner boxes – integrated systems for video analytics, including multi-chip configurations for higher stream counts.
GSP at a glance
| Attribute | Reported value | Caveat |
|---|---|---|
| Cores | 16 graph-streaming cores | General-purpose, scheduler-driven |
| Peak compute | 16 TOPS (INT8, reported) | Architecture argues delivered FPS/W matters more |
| Power | ~7 W (chip-level, reported) | Module/system power is higher |
| Process | Samsung 14 nm | First-generation silicon |
| Memory bandwidth claim | ~50x lower vs GPU-CPU (company-published) | Vendor baseline, workload-specific |
| Latency / efficiency claims | ~10x lower latency, up to 60x system efficiency (company-published) | Directional; verify on your models |
Where the GSP fits – and honest limits
The GSP’s natural home is latency-critical, multi-model edge work at batch size 1: industrial inspection where a verdict must land within a machine cycle – the same budgets discussed in our line-rate quality inspection article – plus ADAS-style sensor fusion, smart-city intersections running several nets concurrently, and retail analytics in fanless enclosures. In Indian cabinet deployments the ~7 W class matters doubly: less heat in a 45 C roadside box, and on-device processing that keeps identifiable video local under the DPDP Act. The limits: modest absolute compute rules out large models, quantisation is mandatory, and the software ecosystem is the thinnest of the three vendors we cover. Weigh it via Hailo vs Axelera vs Blaize, and sanity-check whether an accelerator is right at all in edge AI accelerators vs GPUs.
Frequently asked questions
How is graph streaming different from Hailo’s dataflow architecture?
Both minimise DRAM traffic, but differently. Hailo compiles a fixed layer pipeline into distributed SRAM; Blaize keeps a dynamic hardware scheduler that dispatches graph tasks across general-purpose cores at runtime. Hailo’s approach maximises efficiency for one compiled network; Blaize’s favours flexibility and concurrent multi-model execution.
Is 16 TOPS enough for real workloads?
For quantised vision networks at batch 1, often yes – delivered FPS within a latency budget depends more on memory behaviour than peak TOPS. It is not enough for large transformers or high-resolution generative models; that is not the GSP’s target.
Should I trust the 50x/10x/60x claims?
Treat them as company-published comparisons against a specific GPU-plus-CPU baseline on selected workloads. The direction is credible – data movement dominates edge power – but the multipliers for your workload will differ. Benchmark your own models.
Can multiple models run on one GSP simultaneously?
Yes – fine-grained hardware task scheduling is one of the architecture’s core features, allowing several networks plus pre/post-processing stages to share the cores concurrently rather than time-slicing whole-chip contexts.
What does deploying a model on Blaize involve?
Export to ONNX (or a supported framework format), run it through NetDeploy in the Picasso SDK for quantisation and graph conversion, validate accuracy against your calibration data, then integrate via C/C++ APIs or build the pipeline in the low-code AI Studio.
Ready to deploy?
Talk to an RDP architect about power, cooling and lead time.