Training Behind Your Own Firewall: The On-Prem Case for Private AI in India
Overview
A general foundation model knows the world; it does not know your claims history, your loan book, your imaging archive or your maintenance logs. The value comes from adapting it on your own records — and that is exactly the data you are least able to send elsewhere. This is the practical argument for training on hardware you own: not ideology, but the observation that the most valuable training data is usually the most regulated. This article sets out when on-premises training is the right call in India, what it genuinely costs, and where renting still wins.
Key takeaways
- Your differentiating data is your regulated data — which is why residency and training strategy are the same conversation.
- DPDP does not mandate blanket localisation, but penalties to ₹250 crore per violation push regulated workloads in-country.
- On-prem removes the transfer question rather than answering it — the simplest defensible position for an auditor.
- Rented capacity still wins for bursts — IndiaAI compute at roughly $1/GPU-hour is hard to beat for experimentation.
- The hybrid pattern dominates: experiment on rented capacity, train and fine-tune sensitive models on owned hardware.
The data that matters is the data you cannot move
Consider what a domain model actually needs. A BFSI risk model wants real transaction and default history. A hospital’s model wants real imaging and notes. A manufacturer’s wants real defect images from its own lines. In each case the training corpus is personal, commercially sensitive, or both. Teams frequently discover that the pilot ran fine on synthetic or public data, and the production version — the one that would actually be useful — stalls in legal review. Deciding where training happens is therefore not an infrastructure afterthought; it determines whether the project ships.
What DPDP actually requires
Be precise here, because the Act is often misquoted. The DPDP Act does not impose blanket data localisation; it permits cross-border transfer except to specifically restricted countries. What it does impose is accountability, with penalties up to ₹250 crore per violation. The practical effect is a shift in the burden of proof: you may transfer, but you must be able to justify and control it. Keeping training on infrastructure you own collapses that burden — there is no transfer to document, no processor contract to police, no jurisdictional question to answer. For sector regulators layered on top of DPDP, that simplicity is worth a great deal.
The honest cost picture
| Dimension | On-premises | Rented / subsidised capacity |
|---|---|---|
| Cost shape | Capital up front, low marginal run cost | Operating cost per GPU-hour |
| Indian context | Power, cooling, space, ops staff | IndiaAI compute near $1/GPU-hour |
| Data residency | Provable, no transfer | Contractual, must be verified |
| Best for | Sustained training on sensitive data | Bursty experimentation, benchmarking |
| Scaling limit | What you bought | What is available and funded |
| Latency to data | Data already on site | Egress and staging time |
Where rented capacity genuinely wins
Owning is not automatically correct. The IndiaAI Mission has empanelled more than 38,000 GPUs with a stated target near 100,000 by end-2026, backed by roughly $1.25 billion (₹10,372 crore), at subsidised rates around $1 per GPU-hour. For architecture search, benchmarking, or a one-off large run, that is excellent value and no capital is stranded. The mistake is assuming it resolves governance: subsidised compute is still someone else’s infrastructure, so personal data placed there needs the same contractual and audit scrutiny as any cloud. Use it for work that does not carry your sensitive corpus.
What a single owned machine can reach
The counter-argument to owning is usually capacity — that serious models need a cluster. That has become less true. Memory-offload approaches extend what one machine reaches: on a reference 8-GPU node, published figures show 70B FP32 fine-tuning moving from zero concurrent sessions on GPU alone to seven with NVMe offload, and 180B and 405B becoming possible at all. That does not make a workstation a supercomputer, and offload trades throughput for capacity — but it does mean a single owned, in-country box can fine-tune models that would previously have forced you to rent. See our fine-tuning capacity analysis for the full table and the architecture behind it.
A decision frame
Ask three questions in order. One: does the training corpus contain personal or regulated data? If yes, the default is on-premises and the burden shifts to justifying anything else. Two: is the workload sustained or bursty? Sustained training amortises capital; occasional runs favour rented capacity. Three: can you operate it? Owning means power, cooling, monitoring and someone accountable for uptime — real costs that spreadsheets often omit. Most Indian enterprises land on a hybrid: rented or subsidised capacity for experimentation and benchmarking, owned in-country hardware for the sensitive fine-tuning that creates the actual advantage.
Frequently asked questions
Does DPDP require me to keep AI training data in India?
No. The Act permits cross-border transfer except to specifically restricted countries, so there is no blanket localisation mandate. What it adds is accountability, with penalties up to ₹250 crore per violation, which pushes regulated workloads toward in-country infrastructure in practice.
Is on-premises training cheaper than renting?
It depends on utilisation. Owning is capital up front with low marginal cost, which suits sustained training; rented capacity at roughly $1 per GPU-hour under IndiaAI is better for bursty experimentation. Include power, cooling and operations staff in any comparison.
Can one owned machine fine-tune large models?
More than it used to. Published figures for a reference 8-GPU node show 70B FP32 fine-tuning going from zero sessions GPU-only to seven with NVMe offload, and 180B and 405B becoming feasible — with the caveat that offload extends capacity rather than accelerating a run.
Does subsidised government compute solve data residency?
Not by itself. It is still infrastructure you do not own, so personal data placed there requires the same contractual controls and audit scrutiny as any third-party platform. It is best used for workloads that do not carry your sensitive corpus.
What is the most common working pattern?
Hybrid. Teams use rented or subsidised capacity for architecture search, benchmarking and one-off large runs, and keep fine-tuning on sensitive records on owned in-country hardware where residency, access control and auditability can be demonstrated.
Ready to deploy?
Talk to an RDP architect about power, cooling and lead time.