EY Forward Deployed · technical round
Hub Recruiter
Technical round · practice

Agents, retrieval and evals

The engineering round: Mudit Gupta or one of his Managing Directors, an architecture discussion, a Partner round after. This page is the AI engineering substance. The governance material lives in the deck and gets one answer here, because at this level they are testing whether you can build the thing, not whether you can document it.

How to use this page
  1. These are positions with receipts, not a script. Learn the shape and the specifics, then say it your way.
  2. The specifics are the point. "We use caching" is nothing. "Cache reads cost about a tenth of input tokens, and if your hit rate is zero you have a timestamp in your system prompt" is an engineer talking.
  3. At Manager they want to know you build. Reach for the mechanism, then the tradeoff you owned.
  4. Fill every [bracket] before the call. Those are facts only you know.
Workflow firstagency is earned, not assumed
Context is a budgetnot a place to put things
Retrieval ≠ generationevaluate them separately
Gate, don't measurean eval that can't fail a build is a dashboard
Tools are the APIagent quality is tool quality
Somnathex-IBM Research, OR background, Sibos
Before you start

The read

Three people score three different things. Mudit is deciding whether you are safe in front of a bank. The Managing Director or Senior Manager is deciding whether you can actually build. The Partner at the end is deciding whether a client would pay for you. This page is mostly for the second one, because that is the round candidates lose.

R
Tell me how you think about building with language models.
J
The model is the easy part. What decides whether a thing works is everything around it: how much context you spend and on what, how good the tools are, whether retrieval actually returns the right passage, and whether your evals can fail a build. I spent this year building that scaffolding rather than prompting, and the bank years mean I already know what the second line will ask of it.
Open with the thesis. "The model is the easy part" tells an AI practice leader you have shipped. Everything after it is specifics.
Possible interviewer · researched 3 Nov 2026

Somnath Mukherjee

Twelve years at EY, entered through Advisory in the Financial Services Office. White Plains, New York, so this is an on-shore conversation, not GDS. Before EY: four years as a system engineer at IBM, then three and a half years as a Data Specialist in Advanced Analytics and Data Mining at IBM Research. Master of Computer Applications, West Bengal University of Technology. They are a BI and data-engineering practitioner who moved into analytics and then ML, not an LLM researcher.

Two things to check before the call

They may no longer be a Senior Manager. Several sources show their current title as Managing Director, promoted around June 2026, while older scrapes still say Senior Manager. Look at the live profile yourself. If they are an MD, this is a hiring-manager or skip-level conversation, two ranks above the Manager seat, and you should treat it as a judgment interview rather than a peer technical screen.

They have no publications, patents or public code. That is a verified negative, not a gap in the search. Do not walk in planning to reference their work. The IBM Research role was a practitioner line inside the research org, not a research-scientist one. Note also that Sumanta Mukherjee, one letter away, is an IBM Research India scientist with a long publication list. Different person. Do not mix them up out loud.

Their one recent public statement, and your best hook

They attended Sibos 2026 in Miami, 28 September to 1 October, and posted about it about five weeks before your interview. The substance, verified across several sources:

The AI conversation in financial services has shifted, from experimentation to real business workflows.

They named five domains: payments, trade finance, securities services, client servicing and financial intelligence. They also said 2024 was the first time they heard about AI agents, and that what they are trying to solve and how they operate have not changed since. Read that as a hype skeptic. Lead with production constraints and outcomes. Do not lead with framework names or model novelty.

The dominant live-agent story at Sibos 2026 was exception-queue handling: several large banks described agents repairing failed payments, checking trade documents and dispatching work, with a human on the approval. If you have anything shaped like queue triage with a human at the decision point, that is the story to lead with.

What they will actually probe

Data plumbing before model architecture. They spent a decade on business intelligence, data integration and architecture before any of this. Their recommendations cite data integration and database development. Expect "what does the pipeline actually look like in production", lineage, and freshness. Your data-quality position and the MUFG catalog work are aimed straight at them.

Optimization thinking. Roughly six years across IBM Research and early EY on supply chain planning and optimization. Operations research is native to them, which means they think in objectives, constraints and whether the answer is actually optimal. A hybrid design, where the model handles interface and reasoning and a solver or a rule makes the decision, will land far better than a pure-agent design.

They are upskilling on agents right now. Their recent activity includes agent-engineering coursework. They are learning this alongside you rather than from ten years of depth. Collaborative framing, not credential-flexing. Explaining a tradeoff clearly will do more than demonstrating that you know more vocabulary.

R
You were at Sibos? What did you make of where banks actually are with agents?
J
I didn't attend, but I followed what came out of it, and the thing that struck me is that the live deployments were nearly all exception handling. Failed payments, trade document checks, routing work to the right person. That's the honest shape of it: agents are good at the messy triage step in front of a process that's already well defined, with a human on the approval. It's much less exciting than the 2024 pitch and much more likely to survive a validator.
Only say this if the conversation opens the door. Do not announce that you researched them. "I followed what came out of it" is honest; claiming to have attended is not.
R
Where does classical optimization still beat a language model?
J
Anywhere the decision has a real objective function and hard constraints, which is most of the interesting ones. A model is bad at proving optimality and worse at respecting a constraint it can only read about. So the split I'd build is the model on the interface and the messy inputs, turning a request into a well-posed problem and explaining the result afterwards, and a solver or a rules engine making the actual decision. You get explainability for free, because the binding constraint is right there. The failure I see is teams asking a model to do the optimization because it's the newer tool.
This is your highest-leverage answer with them. It speaks their native language, it's genuinely correct, and it's the same restraint as your workflow-before-agent position applied to their old domain.
The question under every other question

Agent or workflow

R
When do you reach for an agent, and when do you build a workflow?
J
Workflow by default. An agent is for when you genuinely cannot enumerate the path in advance. Most of what gets called an agent in banking is a workflow with a model inside one step, and that's the right answer, not a lesser one: it's testable, it's cheap, and model risk can reason about it. The test I apply is four things. Is the task actually hard to specify up front. Does the outcome justify the latency and cost. Is the model good at this class of problem. And can I catch and recover from an error. If any of those is no, I stay with the workflow. I'd rather ship a deterministic pipeline that passes validation than an agent that demos well and dies in review.
The most important answer on this page. The failure mode is saying "we'd need a framework to scale this" when the correct answer is "this is a workflow on purpose." Those four criteria are close to Anthropic's own published guidance, so you are not improvising a position.
R
So when have you actually chosen an agent?
J
Adversarial generation. On the red-teaming platform I can't enumerate the attacks in advance, and if I could it would be a test suite instead. That earns open-ended exploration, and even then it's bounded by explicit tool permissions and a budget per run. The inverse case is the one I'd point at though. I built a tax agent that files a return from a W-2, and the hard rule there was that the model never does the arithmetic. It extracts and it asks questions; a deterministic engine with versioned, citeable policy packs computes every number. The agency is in the conversation, not in the maths.
Two receipts, opposite directions. One where agency was earned, one where it was deliberately withheld. With an interviewer from an optimisation background, "the model never does the arithmetic" is the half that lands.
Where agent quality actually comes from

Agents and tools

R
Single agent with tools, or multi-agent? Everyone is building swarms.
J
Single agent with good tools beats a swarm for most work, and the swarm is usually a workaround for tools that aren't good enough. Multi-agent earns its place in one shape: parallel, read-heavy exploration where each worker burns its own context and hands back a summary, like researching twenty vendors at once. The moment sub-agents need to share state or agree on something, you've built a distributed system with a language model for a message bus, and you now own consensus and partial failure. I'd rather spend that effort on the tool surface.
Demonstrates: judgment against hype. "A distributed system with a language model for a message bus" is a line that gets repeated.
R
What makes a tool good? That sounds like it should be trivial.
J
It's where most of the quality lives, and it isn't trivial. Granularity first: a tool should map to something the model can decide to do, not to one of your internal endpoints. The description is a prompt, so it gets written like one, including when not to use it. Error messages are also prompts, which is the part people miss. A tool returning "400 bad request" teaches the model nothing. One returning "start_date must be before end_date, you sent the reverse" gets a correct retry. And the schema should be strict, with additional properties off, so you're not parsing hopeful JSON. If an agent is behaving badly, I look at the tool surface before I touch the system prompt.
This answer separates you from people who have read about agents. "Error messages are prompts" is the insight. Have an example ready of a tool you rewrote and what it fixed.
R
Any practical gotchas in the loop itself?
J
A few that cost me time. When the model calls several tools in one turn, every result has to come back in a single message. Split them across messages and it quietly stops calling them in parallel, so you lose the speedup and never see an error. Failed calls need to come back as results flagged as errors, not be dropped, or the model has no idea what happened. And the loop needs checkpointing, because a long run that dies at step nine should resume, not restart. Those are the kinds of things you only learn by running the loop for real.
Demonstrates: hands-on. These are not blog-post facts; they are things you hit. The parallel-results detail in particular is a strong signal.
The skill that replaced prompt engineering

Context engineering

R
Context windows are huge now. Is context management still a real problem?
J
More of one, because the temptation is to fill it. Treat the window as a budget you spend, not a place you put things. Quality degrades well before you hit the limit, especially on retrieval into the middle, and every token is latency and money on every turn. So I'm deliberate about what earns a place: the system prompt, the tool definitions, the few results that matter, and a running summary rather than raw history. Long agent runs are mostly an exercise in deciding what to throw away.
Demonstrates: current practice. "A budget you spend, not a place you put things" is the framing. Avoid the phrase "context window is big now so it doesn't matter" at all costs.
R
Concretely, how do you keep a long run from falling over?
J
Three different tools and they're not interchangeable. Clearing drops old tool results and thinking blocks out of the history entirely, which is right when the content is stale and nobody needs it again. Compaction summarizes the earlier conversation into a block and carries that forward, which is right when the thread still matters. And an external memory store is right when something has to survive the session at all. The mistake is summarizing when you should have deleted, because you pay tokens to carry a summary of something irrelevant.
Demonstrates: precision. Most candidates use "compaction" for all three. Naming the distinction is a differentiator.
R
Caching. Does it actually move the needle?
J
It's the biggest free win, and most teams leave it on the floor. A cache read runs around a tenth of the input price, a write a bit above full, so the economics are obvious on any repeated prefix. The catch is that it's a prefix match. One byte changes anywhere and everything after it is invalidated, so the architecture is stable things first, frozen system prompt and a deterministically ordered tool list, and anything volatile after the last breakpoint. The failure I look for first is a timestamp or a session id in the system prompt, or a dictionary serialised in a different order each run. I got bitten by that enough times that on the clinical co-pilot the cache hit rate is a CI gate condition: if it halves against baseline, the build fails rather than the invoice telling me next month.
Demonstrates: real operational experience, and it answers the Partner's cost question before it's asked. The "check the usage field" detail proves you have debugged this.
Where most bank deployments actually fail

Retrieval

R
Walk me through how you'd build retrieval for a bank knowledge base.
J
Hybrid, then rerank. Dense embeddings alone miss exact terms, and in banking the exact term is usually the point, a product code or a counterparty name. So on the clinical co-pilot it's BM25 for the sparse side and dense vectors in pgvector, merged, then a reranker cutting it to the top three that actually go in the prompt. Chunking follows the document's own structure rather than a fixed token count, because a chunk that splits a table from its header is worse than useless. And the corpus ingest refuses a document that arrives without a source and a licence, so provenance is enforced at write time rather than reconstructed later.
Demonstrates: you have built retrieval, not read about it. The "exact term is the point in banking" observation is domain-specific and lands with this audience.
R
How do you know whether it's working?
J
By evaluating the two halves separately, which is the thing people skip. Retrieval is judged on whether the right passage came back at all. Generation is judged on whether the answer is grounded in what came back. If you only measure the end answer you can't tell a retrieval miss from a reasoning failure, and you'll spend a week tuning prompts when the index was wrong. In practice retrieval breaks first, and far more often.
Short and sharp. This is the single most useful diagnostic opinion in RAG and it takes fifteen seconds to say.
R
When is RAG the wrong tool?
J
Whenever the answer lives in a system that can just be queried. If someone asks for a customer's current balance, you call the API. Embedding the balance is absurd and will be stale. Retrieval is for unstructured text where meaning is fuzzy. For anything with structure, give the model a tool and let it query. A lot of disappointing RAG projects are really just a database question wearing a vector store, and the other common one is a question that needed aggregation across a thousand documents, which retrieval of the top five will never answer.
Demonstrates: judgment about when not to use the fashionable thing. The aggregation point is the one that catches teams out.
R
And the data underneath?
J
That's the existential one. Hallucination is a tractable engineering problem. A wrong number in the warehouse with a citation attached is worse, because you have made it credible. So freshness is a first-class property, every answer carries lineage back to a source, and I'd rather the system say it doesn't know than answer from something stale. I spent part of my MUFG years on an enterprise data catalog, which is unglamorous and exactly the problem that decides whether any of this works.
Volunteer this. EY's own FS leaders publish about data readiness constantly, and candidates reliably forget it.
Named in the job posting

Evaluation

R
Everyone says they do evals. Most of it is vibes. What do you actually do?
J
Agreed, and the test is whether the eval can fail a build. Mine do. A golden set a domain expert signed off on, regression on every change, and the suite is an acceptance gate in CI rather than a dashboard somebody looks at on Fridays. For agents you have to score the trajectory as well as the answer, because a right answer reached through a tool call that should never have happened is still a finding, and in a bank it's the finding. Start small, too. Twenty cases that genuinely discriminate beat a thousand generated ones that all pass.
"Can the eval fail a build" is the whole answer. Say it first and the rest is supporting detail.
R
You use a model as judge. How do you trust the judge?
J
You don't, by default. A judge has measurable biases: it favours longer answers, it favours whichever response it sees first, and it rates its own family's output generously. So the judge gets its own evidence. I calibrate it against human-labelled cases and track agreement, randomise position, and use a dual-judge design so drift shows up as disagreement rather than silently. And I keep the judge's job narrow. Asking "is this answer grounded in the retrieved passage, yes or no" is reliable. Asking "rate the quality out of ten" is not.
Demonstrates: you have actually run judges rather than deployed one. The binary-versus-ten-point point is the practical core.
The part banks will press hardest

Failure modes

R
Prompt injection. How worried should a bank be?
J
Worried in one specific configuration. The dangerous combination is an agent with access to private data, exposure to content somebody else controls, and a way to send something outward. Any two are usually fine, all three is an exfiltration path, and retrieval is how the untrusted content gets in. So the mitigation is architectural rather than a better prompt. Break one leg: no outbound call without a human approving it, or no untrusted content in the same context as privileged tools. Treat retrieved text as data, never instructions. I have a whole eval category for this, direct, indirect and multi-turn injection, with cases mapped to the OWASP agentic entries, and the harm criterion is a real clinical interaction so the judge has something deterministic to score rather than a vibe.
Demonstrates: security literacy and intellectual honesty. "You cannot prompt your way out of this" is the correct and uncommon position.
R
Where do guardrails sit?
J
At the boundaries, not in the prompt. Input checks before the model sees anything, output checks before anything reaches a user or another system, and permission checks at the tool layer where they can actually be enforced. The rule I hold is that anything safety-critical has to be enforced in code, because an instruction in a system prompt is a strong suggestion, not a control. A validator will ask you which it is, and "we told the model not to" is not an answer that survives that meeting.
Demonstrates: the engineering-versus-prompting distinction, framed for a bank audience. Short answer, high signal.
R
Tell me about something you built that was wrong in production.
J
[Pick one real failure before the call: what broke, how you found out, what you changed, and what you added so it cannot happen silently again.] The shape of the answer: the detection came from a gate I had built rather than a user complaint, and the fix was a new case in the regression suite as well as a code change.
Deliberately blank. A borrowed failure story sounds borrowed. Forty seconds, real, ending with the test you added.
Never answer with a near-miss or someone else's fault. The question is how you find out you are wrong.
Named explicitly in the posting

Agentic coding

R
The posting mentions Claude Code and Codex. How do you actually work with them?
J
Daily, and with a clear split. I delegate anything where the specification is clearer than the typing: scaffolding, test generation, mechanical refactors across a lot of files, wiring an integration I've already designed. I keep the architecture decisions, the data model and anything security-shaped. The practice that makes it work is treating the repo as the context: a file that tells the agent how this codebase does things, so I'm not re-explaining conventions every session. And the review bar doesn't move. Generated code gets read, and the eval suite is what tells me it's right, not the fact that it compiles.
Demonstrates: exactly what a forward deployed engineer does all day. "Delegate where the spec is clearer than the typing" is a crisp rule. Have a number ready for how much of your week this is.
R
Does that scale to a client engagement, or is it a personal productivity thing?
J
It scales if the scaffolding is shared, which is the interesting part for a practice rather than an individual. The conventions file, the eval harness, the guardrail patterns and the tool definitions are the reusable assets. If every engagement starts from an empty repo, the second client costs what the first one did and nobody compounds. That's the thing I'd want to help build here, and it's why platform time versus embedded time is a question I'd ask you.
Turns a tooling question into a practice-building answer, which is what a hiring manager at a consulting firm is listening for.
Evidence · about 40 seconds each

Your three receipts

Have these cold. Each names what it is, the hard part, and what it proves. The hard part is the interesting bit; lead with it when they are short on time.

R
Show me the shape of something you built this year.
J
A continuous red-teaming platform for a clinical AI assistant. A red team agent generates attacks, a judge scores them, an orchestrator decides what to test next. The part worth talking about is what happens after a finding. A reproducer confirms the attack fires at least two times in three, and then it gets promoted into a permanent regression case that replays by response hash, with no model in the replay loop at all. So the corpus grows itself and the regression suite stays deterministic. I also refused the pass-fail binary there and made it three states, pass, fail and drift, because a test that passes because behaviour wandered is worse than no test.
Lead with the replay loop, not the agents. "The corpus grows itself and the replay has no model in it" answers the question every AI platform interviewer actually has: how do you know you did not regress. Have the drift line ready; it is the one that gets repeated.
R
Anything closer to a regulated decision?
J
A clinical co-pilot that writes an overnight pre-rounds brief from the chart. Retrieval is hybrid, BM25 for the exact terms and dense vectors in pgvector for meaning, merged and then reranked down to three passages. Eighty-one golden cases behind it, including adversarial ones like forged citations and tampered dates. Two things I'd point at. The five gating rubrics are deterministic Python, not a model, and they're calibrated against human labels so I can report precision and recall on the grader itself. And the gate fails the build if median cost per turn goes more than fifty percent above baseline or the prompt cache hit rate halves. That last one catches a prompt edit that bloats the cached prefix before it reaches production.
The cost and cache assertion is the rarest thing you own. Almost nobody asserts cost in CI. It also proves the caching answer earlier is lived rather than read. Say the three failure modes it catches if they lean in.
R
How do you know your evaluation is itself sound? A judge is just another model.
J
A benchmark harness comparing two architectures for live speech interpretation, voice-to-voice against a streaming cascade. It matters because of how the judging works. Scores are MQM-normalised for length with the raw penalty shown next to the number, so no reviewer has to trust a single figure. Significance uses a paired bootstrap, because the runs are paired and the scores are not normally distributed, so a t-test would be the wrong tool, with Benjamini-Hochberg correction across the rubric dimensions. And the merge gate only blocks on a drop that is both more than two points and statistically significant, with the whole verdict voided unless the judge agrees with human labels at a kappa above zero point six. I gated the gate on its own measured agreement.
This is your answer to "your LLM judge is just another unreliable model." With an interviewer whose background is optimisation and statistics, this is the strongest thing you own. Do not rush it.
R
And the bank platform on your resume?
J
Climate scenario analysis for the credit portfolio at US Bank. Physical hazard across more than thirty-six building types, a portfolio above two hundred fifty billion, feeding credit loss estimates. There was no budget for a vendor, so I made the case to build in-house on open source. I'll be precise: it's a quantitative simulation platform, not an agent system. What transfers is the other half. Getting it through model risk management and in front of the board risk committee was harder than the modeling, and that's the muscle an agentic system needs now.
Volunteer the limitation. Saying "not an agent system" before they are asked is what makes the agentic claims credible. See the vendor item in Still to decide.

Do not oversell these three

The insurance underwriting project. Its six-agent pipeline is simulated: every agent call is a timed delay over sample data, with no model calls at all. Presenting it as a multi-agent system would not survive one follow-up question.

The Riverside interview kit. LangGraph, pgvector and reranking appear only in slide copy. The three live endpoints are single-shot calls in a small Node server. It is a pitch artifact, not a build.

The sales-call analyser. A genuine tool-use loop, but 189 lines and two tools. Real, small, and worth exactly that much. The voice sales agent is the substantial one in that family.

One answer each, then move on

The two gaps

Governance gets one answer here rather than a section, because this round is about building. The Databricks gap is the real risk: Mudit leads EY's FS Databricks business and will find it within two questions.

R
How does an agentic system get past model risk management?
J
By building the evidence as you go instead of writing it afterwards: inventory entry, documented intended use and limitations, validation artifacts, a change log. The thing people miss is that the Fed replaced SR 11-7 with SR 26-2 in April, and SR 26-2 scopes generative and agentic systems out. That sounds like relief and isn't, because it means the bank's own governance has to carry them with no supervisory template. That gap is the work, and I've sat on the other side of that table at two banks.
One answer, then back to engineering. Most candidates will not know SR 11-7 was superseded at all. Do not expand unless asked.
R
Your platform work is Snowflake. We are Databricks on the FS side.
J
A gap I'd name before you had to, and I'm not going to pretend otherwise. What transfers is most of it: lineage, data products, retrieval, governance and the medallion pattern are the same problems with different names. What doesn't transfer is hands-on specifics, Unity Catalog, Delta Live Tables, the notebook and job model. I'd expect to be fluent inside the first engagement, and if it's useful I'll come back with a retrieval pipeline built on Delta with Unity Catalog governing access and my eval harness pointed at it. A day of work, and it would tell you more than another hour of talking.
Name it, size it, offer evidence. Never claim fluency you lack to the person who runs that business. If you make the offer, build it.
If they take up the screen-share

Ten minutes, in this order

You offer this in the recruiter screen, so be ready. Rehearse the order, not the words. Have every tab open before the call; never build context live.

Open and ready: eval suite, one failing test, a tool definition, an audit log record, the routing config. Close Slack and email. Share one window, not the desktop.

After each run

The scorecard

Still to decide

Edits live only in this browser. Print to keep a copy.
Facts as of 3 Nov 2026: SR 26-2 replaced SR 11-7 on 17 Apr 2026; Mudit Gupta leads EY's FS Databricks business; NYC Manager base band $150.7–261.6K. Caching, context-management and tool-use specifics reflect current Anthropic API behaviour. Receipts are drawn from the GauntletAI repos on this machine, verified against source on 3 Nov 2026. Interviewer lines are plausible, not transcripts of real EY calls.