Agentic AI in Banking: GPU Infrastructure Under FREE-AI

Overview

Agentic AI — systems that plan, call tools and take multi-step actions rather than answering a single prompt — changes bank infrastructure in two ways at once. It multiplies inference volume, because one business action becomes many model calls, and it multiplies the audit surface, because every intermediate decision becomes something a regulator may ask about. In India, the RBI’s FREE-AI framework makes the second point concrete. This article covers what agentic deployment does to GPU sizing and what the governance requirement adds to the architecture.

What agentic actually costs in inference

Model a single customer request through a conventional assistant: one prompt, one generation. Now model the same request through an agent: a planning call, one or more retrieval and re-ranking passes, two or three tool invocations each with a model call to format arguments and interpret results, a verification or self-check pass, and a final synthesis. Five to ten model invocations for one business outcome is typical, and reasoning-style planning steps generate long outputs.

The consequence is that GPU capacity for an agentic rollout is not the capacity for the equivalent chatbot rollout with a margin. It is several times larger, and the shape is different: more short prefill-heavy calls interleaved with a few long generations. That mix is exactly what motivates the architecture in disaggregated inference, and it is worth modelling before committing a budget — the sizing method in GPU sizing for agentic AI workloads applies directly.

What FREE-AI requires operationally

The RBI’s FREE-AI committee report structures its guidance as seven sutras — including safety, transparency, accountability, fairness and explainability — across six pillars covering governance, risk management, data stewardship, model lifecycle management, stakeholder engagement and innovation enablement, operationalised through 26 recommendations. For an infrastructure team, three of those translate into concrete system requirements.

Model lifecycle management means every model version in production is inventoried, its training or fine-tuning provenance recorded, and its behaviour monitored against a baseline. Data stewardship means the data an agent retrieves at inference time is itself governed — lineage, access control, retention. Accountability and explainability mean a specific decision must be reconstructable after the fact. None of these are satisfied by an inference endpoint alone; they are satisfied by the logging, storage and control plane around it. The controls detail sits in BFSI private AI GPU server controls and auditability.

Per-step auditability changes the design

The hardest requirement to retrofit is trace completeness. To explain why an agent declined a limit increase, you need the exact prompt, the model version, the retrieved documents and their versions, each tool call with its arguments and response, and the intermediate reasoning — all preserved with integrity guarantees and retained for the applicable period.

That has real infrastructure cost. Traces for a high-volume agent are large, and they contain customer data, so they inherit the same residency and access controls as the source records. Practically: budget durable storage for traces alongside model serving, keep the trace store in-country, and design retention as policy rather than as whatever the log system defaults to. Teams that treat tracing as an observability afterthought discover during the first audit that the evidence was sampled or expired.

Sizing agent traffic

The latency row is the one that most often breaks a pilot. If a customer will tolerate three seconds and the agent takes six steps, each step has roughly 500 milliseconds including retrieval and tool execution. That is a demanding target that usually forces prompt caching, smaller router models for simple steps, and aggressive KV cache reuse across steps within a session.

Graded autonomy is the risk control

The governance answer to agent risk is not a better model; it is bounded authority. A workable progression starts with read-only agents that retrieve and summarise but change nothing, moves to recommend-and-approve where a human commits the action, and only then to bounded execution for low-value, reversible operations with hard limits and automatic rollback.

Each tier has an infrastructure implication. Read-only agents need retrieval governance. Recommend-and-approve needs a queue, a reviewer interface and a record of what was shown to the approver. Bounded execution needs transaction limits enforced outside the model, idempotency and a reversal path. Building the control plane for the tier above the one you are deploying is the cheapest way to avoid re-architecting when the pilot succeeds.

Where the compute should sit

For Indian banks the residency question resolves fairly cleanly. Customer data flows through every agent step, DPDP penalties reach up to Rs 250 crore per violation, and RBI expectations around data localisation for payment system data are long-standing. Combined with FREE-AI’s accountability requirements, the simplest defensible position is in-country inference on infrastructure the bank controls, with traces stored alongside.

The economics reinforce it. Agentic traffic is steady-state and high-volume, which is exactly the profile where owned capacity beats per-token cloud pricing over a depreciation cycle. Bursty work — model evaluation, fine-tuning experiments — can still be rented. The broader risk and planning frame is in BFSI AI risk and GPU infrastructure planning, and the regulatory detail in FREE-AI and model risk rules.

Frequently asked questions