GPU Sizing for E-commerce Recommendation and Personalization (2026)
Overview
Sizing GPU infrastructure for e-commerce recommendation and personalization in 2026 comes down to three numbers: peak queries per second, a latency budget under roughly 100 milliseconds, and the memory footprint of your embedding tables and ranking models. Most Indian mid-market retailers land between a single L4/L40S-class inference server and a small 4–8 GPU cluster, not a training-scale deployment. Industry trackers such as eMarketer expect AI-driven platforms to influence a rapidly growing share of retail transactions through 2026, which makes right-sizing inference capacity a procurement question, not a research one.
Key takeaways
- Real-time recommendation serving targets p99 latency under ~100 ms; that budget, not raw throughput, usually determines GPU class and count.
- Embedding tables for large catalogs can exceed GPU VRAM; plan CPU-RAM or tiered offload before adding GPUs for memory alone.
- A two-stage retrieve-then-rank design lets one L4/L40S-class GPU serve thousands of requests per second for candidate retrieval.
- Batch pre-compute remains cheaper for slow-moving catalogs; reserve real-time GPU inference for session-aware personalization and search.
- Hosting inference in India simplifies DPDP Act compliance for behavioural data and keeps round-trip latency low for Indian shoppers.
What actually runs on the GPU in a 2026 recommendation stack
A modern e-commerce personalization stack has four GPU-relevant stages: candidate retrieval (approximate nearest-neighbour search over item embeddings), ranking (a deep model scoring a few hundred candidates per request), re-ranking with business rules, and increasingly a generative layer that writes personalized copy or powers conversational discovery. Frameworks such as NVIDIA Merlin and open-source stacks built on FAISS, TensorRT and Triton Inference Server cover the first three stages. Only retrieval and ranking are latency-critical; the generative layer can run asynchronously on the same cluster during off-peak windows.
The sizing mistake we see most often is treating the recommender like an LLM. Ranking models are typically tens to hundreds of millions of parameters — small by 2026 standards — but they are invoked on every page view, cart event and search query. Throughput per rupee, not model size, is the constraint.
Latency budgets and what they imply for hardware
Real-time personalization is generally held to an end-to-end budget of 100–200 ms, of which the model-serving slice is 20–50 ms. Within that envelope, a single L4-class (24 GB) or L40S-class (48 GB) GPU running INT8/FP8 engines can score several thousand candidates per request at high concurrency. H100/H200-class parts only become necessary when you consolidate many models — search ranking, recommendations, ads, fraud — onto one serving tier, or when a generative shopping assistant with a 7–13B LLM joins the same pool. For that class of workload, see our GPU server sizing checklist for AI inference.
Embedding tables: the hidden memory problem
For catalogs in the tens of millions of SKUs with per-user embeddings, table sizes of hundreds of gigabytes are normal. These do not need to sit in VRAM. Production systems shard hot embeddings into GPU memory and serve the long tail from CPU RAM or NVMe, which is why inference nodes for recommenders are usually specified with 512 GB–1 TB of system RAM and fast local NVMe. Storage layout matters as much as GPU choice; our guide to GPU storage planning for checkpoints and indexes covers the same tiering logic.
Batch pre-compute vs real-time inference
Not every surface needs a live model call. Email campaigns, home-page carousels for anonymous users, and slow-moving category pages can be pre-computed nightly on a fraction of the hardware. The honest decision rule: if the recommendation cannot change within a session, batch it. Retailers that split traffic this way typically keep 60–70% of impressions on batch output and reserve GPU serving for search, session-aware ranking and cart-abandonment moments where freshness demonstrably lifts conversion.
The India deployment angle
Behavioural and purchase data used for personalization is personal data under the DPDP Act, with penalties up to ₹250 crore for significant breaches. The Act does not mandate blanket localisation, but consent, purpose-limitation and breach-notification duties are easier to evidence when inference and feature stores sit in-country. Quick-commerce and ONDC-network sellers also benefit from single-digit-millisecond proximity to Indian users. With IndiaAI Mission capacity pushing common-pool GPU pricing toward ~$1/GPU-hour, the build-vs-rent comparison for a steady 24×7 recommender increasingly favours owned inference nodes at utilisation above roughly 40–50%.
Reference sizing tiers
| Tier | Traffic profile | Typical hardware | Serving pattern |
|---|---|---|---|
| Starter | <500 req/s peak, single market | 1× L4 or L40S-class GPU, 256–512 GB RAM | Two-stage retrieve + rank, batch for cold surfaces |
| Growth | 500–5,000 req/s, search + reco + ads | 2–4× L40S or 2× H100-class, 512 GB–1 TB RAM | Consolidated Triton pool, FP8 engines |
| Platform | >5,000 req/s, multi-tenant, GenAI assistant | 8× H100/H200-class node(s) | Dedicated LLM pool + ranking pool, autoscaled |
| Batch-only | Campaigns, nightly refresh | Shared or off-peak capacity | Scheduled pre-compute, no latency SLO |
Frequently asked questions
Do recommendation systems need H100-class GPUs?
Usually not for serving. Ranking models are small; L4/L40S-class inference GPUs handle most retail traffic. H100/H200-class parts earn their cost when training large two-tower models frequently or when an LLM-based shopping assistant shares the serving tier.
What latency should a product-recommendation API target?
A common production target is 20–50 ms for the model-serving step inside a 100–200 ms end-to-end page budget. Beyond that, measured conversion impact drops, which is why p99 latency — not average — should drive sizing.
How much VRAM do embedding tables need?
Only the hot shard needs VRAM. Production recommenders keep frequently accessed embeddings on-GPU and serve the tail from CPU RAM or NVMe, so a 24–48 GB GPU with 512 GB system RAM often outperforms a larger GPU with thin host memory.
Is batch pre-compute obsolete in 2026?
No. Batch remains the cheapest option for surfaces that do not change within a session — email, anonymous home pages, category carousels. Most retailers run a hybrid, reserving real-time GPU inference for search and session-aware ranking.
Does the DPDP Act force personalization data to stay in India?
It does not impose blanket localisation, but consent and breach obligations — with penalties up to ₹250 crore — are easier to operate when behavioural data and inference infrastructure are hosted in-country, which is why regulated and large consumer platforms increasingly keep both in India.
Ready to deploy?
Talk to an RDP architect about power, cooling and lead time.