Inference
8 articlesDisaggregated Inference: Splitting Prefill and Decode in 2026
Prefill is compute-bound, decode is memory-bound, and 2026 serving stacks split them onto dedicated GPU pools with KV cache shipped between - worth 2-3x throughput at node scale and far…
GPU Server India Sizing Checklist for AI Inference
When sizing GPU servers for AI inference in India, consider memory capacity, workload requirements, and compliance with data protection regulations. The NVIDIA H200's 141 GB HBM3e memory and adherence to…
Sizing 70B LLM Inference on GPU Servers in India
A 70B LLM inference plan should start with memory, concurrency, latency, power, and data-residency constraints, not only GPU count. For Indian teams, RDP GPU Mart can turn those constraints into…
GPU Sizing for Agentic AI Workloads (2026)
Agentic AI changes the sizing question: it isn't just a GPU problem. The GPU still runs the model, but the orchestration layer — scheduling sub-tasks, routing tool calls, passing state…
How Many GPUs Do You Need for a 100-User Private ChatGPT (On-Prem)?
A private, on-prem ChatGPT-style assistant for 100 users typically runs on one to two datacenter GPUs. Because "100 users" rarely means 100 people typing at once — active-to-total ratios are…
Sizing GPU Compute for Computer Vision / Video Analytics
Computer-vision and video-analytics GPU sizing is throughput-driven, not memory-driven: capacity is set by streams × resolution × frame rate × model complexity, and constrained by the GPU's video-decode and inference…
How Much VRAM Does a 70B / 405B LLM Need for Inference?
For inference, a 70B-parameter model needs ~140 GB of GPU memory at FP16, ~70 GB at FP8, and ~43 GB at 4-bit — so one to two datacenter GPUs. A…
GB300 NVL72: Anatomy of a 120 kW Rack-Scale AI Factory
Overview The NVIDIA GB300 NVL72 (Blackwell Ultra) marks the point where the rack, not the GPU, becomes the unit of compute. Seventy-two Blackwell Ultra (B300) GPUs and 36 Grace CPUs…