EY Forward Deployed · technical depth
Practice page Training Feed Recruiter
Technical round · depth beneath the transcript

Every answer, three levels deep

Your transcript is the Level 1 layer: thirty seconds per answer, problem then constraint then decision then number. This page is what sits underneath each of those answers when the interviewer says "go deeper." Read the blue line out loud, then open the probes and answer each one before reading the model reply.

How to use this page.
  1. Read The round once. It says what is scored and how 30 minutes constrains you. The practice page from 3 Nov is the ten-minute version; this page is the depth beneath it.
  2. Do Your systems first. Résumé deep-dives are the most reported EY format and the probes there are the ones you will actually get.
  3. Then one technical area per sitting. Say the Level 1 line, open each probe, answer out loud, compare.
  4. Run the design drills on a whiteboard, timed to ten minutes each.
  5. Finish with rapid fire in Practice mode until none of the fifty needs a pause.
1 · What the round is

How technical rounds work for this kind of role

Per the recruiter's brief on 10 Nov: a 30-minute HireVue conversation covering technical depth, behavioral, and current plus prior experience, with time for your questions. Requisition is the Manager posting (1708792). Contact is now Elle Gray.

What the research says gets scored

SignalHow the interviewer detects itWhere you show it
Ownership"I decided", "I shipped", named trade-offs you defendedUS Bank build-versus-buy; AgentForge cost invariant
Production mindsetTesting, observability, failure modes, rollback mentioned unpromptedEvery answer ends with the eval or the control
Evaluation"How do you know it works?" answered with a harness, not "it looks right"263-case golden set; 80-case regression; dual judge
DecompositionClarifies before solving, names stakeholders, sequences by risk and valueDesign drills; "walking skeleton first"
Customer empathyExplains a system to a non-technical executive without jargonBoard Risk Committee; validators
Communication under pressureNever silent; narrates thinkingWhiteboard out loud

Formats you may meet, in order of likelihood at EY

  • Résumé deep-dive with one or two managers (most reported for EY technical consulting and AI roles): "pick a project and walk me through your thinking." 30 to 45 minutes. Sometimes one simple coding question first.
  • Architecture conversation: a client scenario, often lifted from your own résumé. The healthcare-RAG and policy-document designs below are the canonical shape.
  • Decomposition case (Palantir origin, now common at AI labs; not reported at EY but cheap to rehearse): an under-specified business problem, 45 to 60 minutes, scored on how you clarify, structure, prioritise, and adapt, not on the answer.
  • Practical coding (AI labs; rare at EY for this level): parse messy data, build a small CLI, a rate limiter, a small RAG over a folder. Readable, tested code over clever code.
  • Take-home (AI labs): 4 to 8 hours building a RAG or eval harness. Not reported at EY.

What 30 minutes means

  • Level 1 answers only, unless probed. Protect the design probe; let Gertie breadth be the thing that gets cut.
  • If HireVue turns out to be a recorded one-way format, each answer gets a short prep window and two to three minutes to record. Lead with the Level 1 line, add one receipt, stop. Transcripts are what get scored, so speak in full sentences and name the numbers.
  • Every technical answer carries one number and one control. "Zero knowledge-tier violations" is a number. "Access control enforced at retrieval" is a control.

If the interviewer is Somnath Mukherjee

Twelve years at EY through FSO Advisory, White Plains; before that IBM and IBM Research as a data and analytics practitioner, not an LLM researcher. Possibly a Managing Director since mid-2026, which would make this a judgment conversation two ranks above the seat rather than a peer screen; check the live profile. No publications or public code, so do not plan to cite their work. Their one recent public statement, from Sibos 2026: the AI conversation in financial services has moved from experimentation to real business workflows, in payments, trade finance, securities services, client servicing and financial intelligence; the live-agent story there was exception-queue handling. Read that as a hype skeptic: lead with production constraints and outcomes, never with framework names. The full read is on the practice page.

Failure modes the research names. Jumping to a solution before clarifying. "We did" instead of "I did." Silence while thinking. "It looks right" when asked how you know. Over-architecting instead of scoping a thin end-to-end version. Starting an answer with a stack list.
2 · Your systems, probed

The résumé deep-dive, three levels down

Each system: the Level 1 line, then the probes an engineer would ask next. The honest limits are written in, because the interviewer who finds one you did not name stops trusting the rest.

U.S. Bank: the Monte Carlo loss platform

Scenario analysis on about $165B of real estate loans. The bank had a $1M vendor commitment on the table; I made the case to build, in Python with Snowflake pipelines, compound-event modelling across 36-plus building types, feeding PD and LGD adjustments and CECL reserve impacts. One codebase shared by risk, credit and modelling. The hard part was not the maths; it was the documentation that passed independent validation. It is a quantitative simulation platform, not an agent system; Claude Code wrote and tested much of the code under my review.
Probes and model answers
"Walk me through the pipeline, end to end."

Inputs: loan-level exposure from the warehouse (geography, collateral type, vintage, servicer), S&P market and risk data through Snowflake. Hazard layer: event catalogues replayed against the exposure. Vulnerability: damage functions by building type (36-plus). Financial: loss to collateral, net of insurance, mapped to PD and LGD adjustments, then CECL. Outputs: expected loss, VaR and TVaR, 1-in-100, 1-in-250, 1-in-500 scenarios, concentration cuts. Monte Carlo over the event set; results versioned per run.

"What did validation actually test?"

Conceptual soundness: why these damage functions, why this event catalogue, sensitivity to the key assumptions. Outcome analysis: back-tests where history exists, benchmark against the vendor's numbers on an overlapping portfolio, stability across seeds and sample sizes. Implementation: production matches development, inputs reconcile to the warehouse, change control. I wrote the documentation to their template before they asked.

"How did you know the Monte Carlo had converged?"

Run-to-run variance of the tail metrics against sample size; pick the sample where the 1-in-250 moves less than a tolerance between seeds; document the seed and the tolerance. Replication of outputs is one of the first things a validator asks for, so the seed is logged with every run.

"Where is AI in this, honestly?"

In how it was built: Claude Code agents wrote and tested code, with me reviewing and owning the methodology and the documentation. The platform itself is deterministic simulation, which is why it could pass validation cleanly. The GenAI piece on my résumé is the risk-news monitor: grounded, cited, in weekly use.

"What would you do differently now?"

Build the evidence trail as a product from day one: the model inventory entry, the change log and the eval artefacts generated by the pipeline rather than assembled afterwards. And under SR 26-2 I would tier it by materiality explicitly and size the validation to that tier.

Trap. Letting it be described as an "AI agent-based" platform. It is not, and the first validator-minded interviewer will know. Say so first.

The risk and regulatory news monitor

Weekly monitor across nine risk categories and U.S. and global supervisors: automated relevance scoring, Claude-generated briefs, auditable monthly archives that trace every headline to its source. Grounded, so a reader can check any claim in one click.
Probes and model answers
"How do you score relevance?"

A rubric per risk category applied by the model with the category definitions in context, plus deterministic rules for supervisors and instruments. Scores are logged with the rationale so a reviewer can disagree with a specific line, not the system.

"How do you stop it inventing a headline?"

Generation only over retrieved items; every brief line carries the source link; a brief that cites nothing fails the build. The archive is the evidence trail. Zero hallucination is not a claim I make; a traceable one is.

"What is the cost and the latency?"

Batch, weekly, so latency is irrelevant and cost is dominated by input tokens; prompt caching on the stable rubric and summarisation of long items keep it to a few dollars a run.

AgentForge: continuous red-teaming for clinical LLM workflows

Five specialist agents (orchestrator, red team, judge, documentation, patch advisor) plus a watchdog, behind hard trust boundaries on a Postgres job bus. Every agent role runs on local open models on a DGX Spark and Mac fleet, so platform inference is $0 and only the customer's target model is billable, about three cents a run; a test enforces the all-local invariant. Every confirmed exploit becomes a deterministic regression case replayed on each new target deploy, with no LLM in the replay loop.
Probes and model answers
"Why a Postgres bus and not Kafka or Redis Streams?"

We already needed Postgres for the coverage map and cost meter; one typed jobs table with kind, payload, state, parent_id, attempts, last_error and dead-letter gave us idempotent agents and a queryable audit trail. Kafka would add operations for a throughput we never approached.

"How do you keep the judge honest?"

Independence enforced by the bus, not goodwill: the red team never sees the judge's rubric and the judge never sees the mutation chain. Dual judge, two independent model instances, agreement is a boolean. Rubrics are version-locked; the judge is re-evaluated weekly against a frozen golden set, and if accuracy drops the rubric version bumps rather than drifting silently.

"Why local models for the red team?"

Frontier APIs refuse offensive workflows by training, and that refusal posture can change with any vendor update. Local 70B-class models without that tuning keep the attack loop stable, and no PHI-shaped payload leaves the box, which removes a whole category of BAA questions for a hospital CISO.

"What stops it running up a bill or looping?"

A cost meter per agent per campaign halts mutation when signal-per-dollar drops below a threshold and forces the orchestrator to redirect. Every job carries attempts and a dead-letter timestamp. Frontier spend is tied to confirmed-exploit count, not mutation volume, so ten times the testing does not mean ten times the bill.

"How do you detect drift in the agents themselves?"

Watchdog: every agent heartbeats every ten seconds; the watchdog samples outputs for schema and behaviour drift and is the only agent allowed to issue restarts. Regression replays use character-shingle Jaccard similarity to flag drift in the target's responses.

"What does the threat model cover?"

Six categories: prompt injection (direct, indirect, multi-turn), data exfiltration and authorisation bypass, state and context corruption, tool misuse (unintended invocation, parameter tampering, recursion), denial of service and cost amplification, identity escalation. Each scored on attack surface, impact, difficulty and existing defences, and kept as a living document.

"Honest limits?"

The judge ran in stub mode for part of the build, documented as such; the target was a mock clinical co-pilot with four deliberate vulnerabilities; the public deploy was phase two. I'd rather say that than have it found.

Gertie: a governed agent for a school

A side project as a parent, and my most recent production agentic system. Deliberately not an autonomous agent: a graph-orchestrated workflow with bounded autonomy. It answers only from reviewed knowledge, routes everything else to a named owner, and a human approves replies first. 12-node orchestration graph, 14 guardrails including prompt-injection and knowledge-poisoning defences, per-step tracing, an approval queue, and a voice line (LiveKit, Deepgram, ElevenLabs) that speaks only the same grounded answers. First pilot eval: 263 cases across about 158 knowledge files, release bar of zero knowledge-tier violations, met.
Probes and model answers
"Why a graph and not a ReAct loop?"

Because the steps that must never vary (authorisation, knowledge-tier check, approval) have to be nodes the model cannot skip. A loop lets the model decide the control flow; a graph lets me decide it and let the model fill in the language. Autonomy is risk-tiered per channel, not granted globally.

"What are the 14 guardrails, roughly?"

Input side: injection screening, PII and off-scope detection, length and rate limits. Retrieval side: knowledge-tier enforcement, entitlement checks before any detail read, repeated on every turn, poisoned-write screening. Output side: schema validation, citation presence, no numbers that did not come from a tool, refusal templates that never echo foreign names or IDs. Process side: approval queue, kill switch, max iterations.

"How is the eval built?"

Golden cases per route with fixed reference replies, run with repeats so flaky behaviour shows as variance, independent expected-versus-observed assertions, plus adversarial authorisation, revocation and session-isolation cases. An unavailable source must never become a zero. CI checks the suite; the bar was violations, not an average score.

"Would the pattern scale to a bank?"

The controls are the enterprise pattern already: governed knowledge base, entitlement-aware retrieval, intent taxonomy with handoff to an accountable owner, evals as the release gate, tracing and cost controls. What changes at scale is the infrastructure around them: managed model access through a gateway, multi-tenant isolation, an identity provider instead of a portal session, a records system for the approval trail.

"The row-level-security incident?"

Diagnosed that the anonymous database role had full read access, verified the fix, and separately root-caused a two-month lead-capture outage to one new required field silently breaking a sync. The lesson I carry into AI work: access control has to travel with retrieval; no prompt downstream saves you if the store lets anyone read it.

Keep it at Level 1 unless asked. Gertie proves the governance instinct; the bank work proves the domain. Do not let the school story eat the design probe.

The multi-agent delivery pipeline, Cortex and RepIntel

A planner, architect, coder, reviewer, tester, deployer pipeline taking a ticket to a deployed app in a week, governed rather than freewheeling: human approval gates, parallel implementers capped by an independent reviewer agent, written project guardrails every change is checked against before merge, a standing gap log, Codex verifying each change independently. Cortex: five persistent assistants on private infrastructure with local inference and MCP memory tools. RepIntel: reputation-risk scores for 23 U.S. banks, matching 1.15M of 13.65M CFPB complaints to institutions.
Probes and model answers
"Why is the reviewer independent of the implementers?"

Same reason a validator is independent of a developer: an agent grading its own work shares its blind spots. The reviewer sees the diff and the guardrails, not the implementer's reasoning.

"How did you match complaints to banks at that scale?"

Deterministic first: normalised legal names, subsidiaries and brand aliases; fuzzy matching with thresholds second; a model only on the ambiguous residue, with a sample reviewed by hand to put an error bound on the match rate. 1.15M of 13.65M is the matched set, stated with its precision, not "all complaints."

"What does MCP do for you in Cortex?"

A typed boundary between assistants and tools: memory, health, file access, each scoped. The lesson relevant to banks: the tool descriptions are part of the attack surface, so they are pinned and the servers run with least privilege.

3 · Technical area

LLM fundamentals, the ones that come up

Level 1. A decoder-only transformer predicts the next token over a context window; attention is quadratic in sequence length, which is why context and the KV cache cost what they do. Everything I build treats the model as a probabilistic text engine whose outputs need grounding, evaluation and controls around them.
Probes and model answers
"Why does tokenization matter to an architect?"

Cost, context consumption and multilingual performance. Byte-pair encoding means a number, a table or code can cost far more tokens than its character count suggests; budgets and chunk sizes are set in tokens, not words.

"What is the KV cache and why do I care?"

Cached key and value tensors from earlier tokens so each new token costs linear rather than quadratic work. It is why long contexts eat GPU memory, why output tokens are priced higher than input, and why prompt caching exists: a stable prefix's KV state is reused instead of recomputed.

"Context window is a million tokens, so why chunk at all?"

Quality degrades before the hard limit ("lost in the middle"), latency and cost grow with input, and retrieval gives citations and access control that stuffing the window does not. Big windows change where the threshold is, not whether you retrieve.

"Temperature and top-p?"

Temperature scales logits before softmax: 0 is greedy and near-deterministic for extraction; 0.3 to 0.7 for conversation. Top-p keeps the smallest set of tokens whose cumulative probability passes a threshold, adapting to the model's confidence. For regulated outputs I pin both and log them, because replication is the first thing a validator asks for.

"Quantization?"

Storing weights, and optionally activations and KV cache, in fewer bits (FP8, INT8, 4-bit) to cut memory and bandwidth. On the DGX Spark I run 70B-class models quantized; the trade is a measured quality delta on my own evals, never assumed.

"RLHF versus DPO, in one breath?"

Both align a pretrained model to preferences. RLHF trains a reward model then optimises against it; DPO optimises directly on preference pairs without a separate reward model, simpler and cheaper. Either explains why production models sometimes over-refuse.

"Embeddings from the LLM or a separate model?"

Separate, purpose-built embedding models for retrieval; they are cheaper, faster and better at similarity. The generation model's internal representations are not what you index.

Terms to own

BPE
byte-pair encoding, the tokenizer family behind most production models
GQA / MQA
grouped or multi-query attention: fewer KV heads, smaller cache
MoE
mixture of experts: only some parameters active per token; cheaper inference per quality
prefill vs decode
processing the prompt versus generating tokens; TTFT is prefill, tokens-per-second is decode
O(n²)self-attention cost in sequence length
400–512 tokensa typical chunk; 10–20% overlap
0 / 0.3–0.7temperature for extraction / conversation
Trap. Reciting architecture. One sentence of mechanism, then what it changes in a design decision.
4 · Technical area

Retrieval-augmented generation, production grade

Level 1. Sourced retrieval first: documents chunked by structure, hybrid search, reranking, citations on every answer, a clear no-answer path that escalates to a human, and access control enforced at retrieval rather than in the prompt. Then a golden-set evaluation before launch, tracing in production, and cost and latency controls per workflow.
Probes and model answers
"Chunking strategy, and why?"

Default: recursive splitting at roughly 400 to 512 tokens with 10 to 20 percent overlap, page-level for PDFs and paginated policy documents, structure-aware (headings, clauses) for policies and contracts. Parent-child when answers need the surrounding section. Co-optimise with retrieval: recursive pairs with hybrid; semantic chunking pairs with dense-only. Measure where retrieval fails before adding complexity.

"Hybrid search and reranking, concretely?"

BM25 for lexical exactness (policy numbers, account types, product names) and vector search for meaning, run in parallel; take the top hundred or so candidates; cross-encoder rerank (Cohere Rerank or a BGE reranker) down to the five to ten the model sees. Reported end-to-end precision gains of 25 to 40 percent over naive single-stage retrieval. LLM reranking only as an optional third stage for the hardest queries.

"How do you evaluate retrieval separately from generation?"

Labelled query set of 50 to 100 questions with known relevant chunks; precision@k, recall@k and MRR; a target like precision@5 above 0.8, and if it underperforms look at the embedding model and chunking before touching the prompt. Then generation metrics on top: faithfulness, answer relevance, context precision and recall, the RAGAS set, LLM-judged and calibrated to a human sample.

"Where does the no-answer path trigger?"

Retrieval confidence below a threshold, no chunk passing the reranker, or the judge finding a claim without a supporting chunk. The response is a refusal with a route to a named owner, logged. In a bank a confident wrong answer costs more than an escalation.

"Freshness and versioning?"

Documents carry effective dates and versions; the index is rebuilt or patched on change with the version in metadata; answers cite the version; alerts when document age exceeds a threshold. Policy that changed last week must not be answered from last month's chunk.

"Multi-hop questions?"

Agentic RAG: decompose, retrieve per sub-question, synthesise, at higher latency and more model calls. Use it where single-shot retrieval measurably fails, not by default.

"Why not just fine-tune the knowledge in?"

Knowledge changes, you need citations and an audit trail, and a fine-tuned model becomes a model the bank has to validate. RAG keeps the knowledge in a governed store the data team already controls.

Terms to own

faithfulness
is every claim in the answer supported by retrieved context
context precision / recall
were the retrieved chunks relevant; did they contain the answer
cross-encoder
reranker that scores query and passage jointly; accurate, slower, so used on a short list
parent-child chunks
retrieve small, return the enclosing section
metadata filter
pre- or post-filtering retrieval by tenant, entitlement, date, product
top 100 → top 5–10hybrid candidates, then rerank
+25–40%reported precision from hybrid + rerank
50–100 queriesa usable retrieval test set
p@5 > 0.8a sane retrieval target
Trap. Describing RAG as "embed, store, retrieve, generate." That is the demo. Production is access control, citations, no-answer path, freshness and an eval set.
5 · Technical area

Agents, workflows and orchestration

Level 1. A workflow runs models and tools through predefined code paths; an agent lets the model direct its own steps. Precision and auditability steps stay in workflows; judgment and language steps get a bounded agent with explicit tools, budgets and a verifier. Agents earn autonomy through eval evidence, never by default. Say "workflow" first.
Probes and model answers
"Name the workflow patterns and when you'd use each."

Prompt chaining for fixed decomposable steps, trading latency for accuracy. Routing when inputs fall into distinct classes handled differently (triage by intent). Parallelization: sectioning for independent subtasks, voting for confidence. Orchestrator-workers when the subtasks cannot be predicted and a central model delegates. Evaluator-optimizer when there is a clear rubric and iteration helps. A fully autonomous agent only for open-ended problems where steps cannot be hard-coded and you can trust the model's decisions. Anthropic's framing, and it matches how I built Gertie: routing plus chaining, with one bounded agent for drafting.

"Single agent or multi-agent?"

Single by default. Multi-agent only when the task has distinct phases that benefit from different prompts, tools or trust boundaries, because it multiplies debugging complexity and tokens. AgentForge is multi-agent precisely for the trust boundary: red team and judge must not share context.

"How do you design tools for a model?"

Verb-noun names, a one-line purpose, typed parameters with documentation, a defined output format; fewer than five parameters and fewer than ten tools per agent; examples and edge cases in the description; formats close to natural text; poka-yoke so the obvious misuse is impossible. Tools return data, never instructions.

"Human in the loop, concretely, in a graph?"

An interrupt node before any state-changing tool call; state persisted to a durable checkpointer (Postgres, not memory) so the thread survives a restart or a reviewer who comes back tomorrow; the API returns immediately with a thread id; resume with the reviewer's decision. A max-iterations guard on every tool loop.

"What does MCP change?"

A standard typed boundary for tools and context, so an agent can reach Jira, SharePoint or a bank's APIs through governed servers instead of bespoke glue. It also makes tool descriptions part of the supply chain, so they are pinned by hash, allowlisted, and run with per-tool scopes and OAuth audience binding.

"Autonomy tiers for a bank?"

Collaborative assistance (drafts for a human), human-approved automation (prepares an action, a human commits it), autonomous operation (restricted, with a kill switch). Which tier a workflow gets depends on decision proximity and consumer-harm potential. Adverse-action language and override rationales are critical tier, so always human-reviewed.

Context engineering probes
"Context windows are huge now; is context management still a problem?"

More of one, because the temptation is to fill it. The window is a budget you spend, not a place you put things: quality degrades well before the limit, especially for retrieval landing in the middle, and every token is latency and money on every turn. What earns a place: the system prompt, the tool definitions, the few results that matter, a running summary rather than raw history.

"Concretely, how do you keep a long run from falling over?"

Three different tools, not interchangeable. Clearing drops stale tool results and reasoning out of the history entirely. Compaction summarises the earlier thread into a block and carries it forward when the thread still matters. External memory is for anything that must survive the session. The common mistake is summarising what should have been deleted, paying tokens to carry a summary of something irrelevant.

"Why does prompt caching break in practice?"

It is a prefix match: one byte changed anywhere invalidates everything after it. So the architecture is stable things first (frozen system prompt, deterministically ordered tool list) and volatile things after the last cache breakpoint. The failures to look for first: a timestamp or session id in the system prompt, or a dictionary serialised in a different order each run. A CI assertion on cache hit rate catches a prompt edit that bloats the prefix before it ships.

Terms to own

ReAct
reason, act, observe loop; the basic agent shape
checkpointer
durable store of graph state per thread; enables pause, resume, replay
interrupt
a pause point in the graph awaiting external input
A2A
agent-to-agent protocol; messages between agents need auth and integrity like any API
kill switch
halts an agent's actions and revokes its credentials immediately
<10 tools, <5 paramsper agent, per tool
12 nodes · 14 guardrailsGertie
3 autonomy modesassistance · approved automation · autonomous
Trap. Calling everything an agent. The interviewer who built one will ask what decides the control flow. Your answer: the graph, not the model.
6 · Technical area

Evaluation: the differentiator in AI roles

Level 1. Evals are the release gate, not a report after the fact. A golden set in CI, regression on every change, a red-team suite for injection, leakage and tool misuse, live judge-graded runs calibrated against human labels, and production monitoring for drift. The bar is written in the validator's language: violations, not an average score.
Probes and model answers
"Anatomy of an eval?"

Task (inputs plus success criteria), trial (one attempt; run several), grader, transcript (everything: outputs, tool calls, intermediate state), outcome (the final environment state, not the agent's claim). You evaluate the model and the harness together.

"Grader types and when?"

Code-based (unit tests, regex, schema checks): fast, objective, brittle when several answers are valid. Model-based (LLM-as-judge with a rubric or pairwise): handles open-ended output, non-deterministic, costs money. Human: highest quality, used mostly to calibrate the model judge on a stratified sample.

"pass@k versus pass^k?"

pass@k is the chance of at least one success in k tries, fine when one good answer suffices. pass^k is the chance all k succeed, what a customer-facing agent needs: 75 percent per trial gives only 42 percent pass^3. I report pass^k for anything a human will trust.

"LLM-as-judge pitfalls?"

Position bias in pairwise comparisons, verbosity bias, self-preference for its own family's outputs, format bias, and calibration drift over time. Mitigations: binary or rubric scoring over free scores, AB/BA swapping with position bias checked below a tolerance, aggregation over several calls, and periodic calibration against a held-out human-labelled set. A judge with high test-retest repeatability can still be badly biased; repeatability is not validity.

"How big an eval set?"

Start at 20 to 50 tasks sourced from real failures, grow to a few hundred stratified by difficulty; retire tasks once they saturate above roughly 80 percent; keep capability evals (hard, low pass rate, a target) separate from regression evals (near 100 percent, a drop means breakage). Read the transcripts: a flagship benchmark score jumped from 42 to 95 percent once grading bugs and ambiguous specs were fixed. Evals are infrastructure with an owner.

"Evaluating agents, not chatbots?"

Trajectory grading: tool choices, argument correctness, recovery from errors, end state. Simulated users for multi-turn (τ-bench style) checking end state, transcript constraints and interaction quality. New judge biases appear: trajectory-length bias, hallucination blindness, argument blindness. Deterministic checks wherever possible, judge only for the rest.

"Your regression in CI, how does it work?"

Golden outputs hashed; any behavioural change on a golden case fails the build; AgentForge replays confirmed exploits deterministically on every target deploy with no model in the loop; Gertie runs each golden with repeats so flakiness shows as variance. Cost is a tested invariant too: the all-local test fails if any platform agent calls a billable endpoint.

Terms to own

golden set
curated cases with agreed correct outputs; the acceptance test
capability vs regression
stretch targets versus guard rails against backsliding
saturation
when a task stops discriminating; retire it
calibration
agreement of judge scores with human labels; measured, then monitored
trajectory
the sequence of tool calls and states an agent took
263 cases / ~158 filesGertie pilot eval, zero knowledge-tier violations
80 casesclinical copilot regression suite
0.75³ = 42%pass^3 at 75% per trial
85–90%typical judge–human agreement when calibrated
5%production sample for human review
Trap. "We use an LLM judge" with no mention of bias, variance, calibration or what the judge is forbidden to see. That reads as incomplete to anyone who has shipped one.
7 · Technical area

Security: injection, agency, supply chain

Level 1. Treat all user and retrieved content as data, never instructions. Separate system instructions, screen before and after generation, allowlist actions with typed arguments, least-privilege tools, and escalate rather than guess. Design so that once an agent has read untrusted input it cannot trigger a consequential action. I shipped those guards in production.
Probes and model answers
"OWASP LLM Top 10, which ones matter most in a bank?"

LLM01 prompt injection, LLM02 sensitive information disclosure, LLM06 excessive agency, LLM08 vector and embedding weaknesses (cross-tenant retrieval, poisoned content), LLM05 improper output handling (model text reaching SQL, HTML or a shell), LLM10 unbounded consumption (denial of wallet). The rest: supply chain, data poisoning, system-prompt leakage, misinformation. Assume the system prompt leaks: never put secrets or access logic in it.

"And the agentic list?"

OWASP's 2026 agentic ten: goal hijack, tool misuse, identity and privilege abuse, agentic supply chain, unexpected code execution, memory and context poisoning, insecure inter-agent communication, cascading failures, human-agent trust exploitation, rogue agents. The controls are mostly identity and plumbing: one identity per agent with scoped time-bound credentials, per-tool verb restrictions and pre-execution policy checks, pinned tool definitions, sandboxed execution, signed inter-agent messages, circuit breakers, plain-language risk summaries before approval, a declared manifest of expected tools and a kill switch.

"Design patterns with an actual guarantee?"

From the Beurer-Kellner et al. paper: action-selector (the model only picks from predefined actions, no feedback from tools into the model); plan-then-execute (the plan is fixed before untrusted data is read, so injection can alter arguments but not add actions); LLM map-reduce (isolated sub-agents process each untrusted document with no tools, outputs constrained); dual LLM (a privileged model with tools never reads untrusted text; a quarantined model with no tools does, returning symbolic references the orchestrator dereferences); code-then-execute (the model writes a program that calls tools and quarantined models); context minimization (strip the user prompt before the reply). The principle: untrusted input must never be able to trigger a consequential action. Gertie is action-selector plus plan-then-execute; AgentForge's judge is quarantined from the red team's text.

"Spotlighting? Detectors?"

Spotlighting marks untrusted spans (delimiters, encoding) so the model can tell data from instructions; detectors (classifier models) catch known patterns. Both raise the bar; neither is a guarantee. They sit in front of the structural patterns, not instead of them.

"A real incident you'd cite?"

EchoLeak, mid-2025: a zero-click indirect injection in an enterprise copilot that exfiltrated data through retrieved content. It is the argument for retrieval-side controls and for output handling: model output is untrusted input to whatever consumes it.

"MCP specifically?"

Tool descriptions, schemas and returned data are trusted context by default, so a compromised or malicious server can steer the agent (tool poisoning) and the server executes with its own privileges (confused deputy). Controls: allowlist and pin tools by content hash, OAuth 2.1 with audience binding per server, least-privilege scoped credentials per server, sandboxing, human approval for high-impact actions, logging. A 2026 survey put tool-permission scoping at under a fifth of deployments; that is the gap an FDE closes.

Terms to own

indirect injection
instructions hidden in content the model retrieves or is sent
excessive agency
more tools, permissions or autonomy than the task needs
confused deputy
a privileged component acting on an unprivileged caller's request
denial of wallet
inference-cost abuse through expensive or high-volume requests
audience binding
a token valid only for the server it was issued for
10 + 10OWASP LLM (2025) and Agentic (2026) risks
6 patternswith structural guarantees against injection
~18%of MCP deployments scoping tool permissions (2026 survey)
Trap. "We filter prompts for injection." Filtering is heuristic. Say what the architecture makes impossible.
8 · Technical area

Model risk, validation and the regulators

Level 1. SR 26-2 replaced SR 11-7 in April 2026 and explicitly scopes generative and agentic AI out, so a bank's own governance has to carry them. That is the gap an FDE fills: an inventory entry, risk-tiering by decision proximity and consumer harm, evidence-grounded inputs, output evaluation on named dimensions, monitoring, and a human-accountability trail an examiner can read.
Probes and model answers
"What actually changed in SR 26-2?"

Narrower model definition ("complex" quantitative methods; spreadsheet arithmetic and deterministic rules are out); risk-based proportionality instead of fixed annual validation; validation quality judged by rigour rather than organisational structure; vendor models still in scope even when code and data are withheld; and footnote 3 removing generative and agentic AI from scope while telling banks to govern them anyway. The agencies have said an RFI on AI is coming.

"So what do you build for an LLM system a validator will see?"

Four things at minimum: a registry entry with owner, lifecycle status and versions; ongoing monitoring of drift, hallucination and tool behaviour with alert thresholds; inline policy enforcement, deterministic rules plus model-based scoring on requests and responses; and audit trails capturing enforcement decisions and the policy that applied. Then the evidence that the evals and controls exist and ran.

"How do you risk-tier a GenAI use case?"

Two axes: decision proximity (how close to credit, pricing, adverse action) and consumer-harm potential. Low: grammar and formatting. Moderate: summaries of validation reports. High: summarising monitoring alerts before distribution. Critical: drafting adverse-action language or override rationales, which need traceability to approved codes and mandatory human review. Gate conditions before any of it: no autonomous decision authority, anchored to approved models and policies, bounded role, feasible oversight.

"What does a validator test on an LLM application?"

Conceptual soundness: rationale for the foundation model (a literature review of its known weaknesses), data quality (PII leakage, sampling bias, annotation quality), model specification (prompts, decoding parameters, fine-tuning choices), explainability (context source for each generated fact), fairness testing with guardrail models, benchmarking against a simpler method. Outcome analysis: performance metrics and replication (log the seed and decoding parameters), robustness under perturbation, weakness detection by clustering inputs, hallucination detection (NLI against the source, self-consistency across samples, chain-of-verification), toxicity. Then implementation testing: dev matches prod, user and usage controls, human-in-the-loop controls, change management.

"Which output dimensions do you report?"

Accuracy, citation accuracy, completeness (capture rate), relevance, hallucination (frequency and severity), traceability, latency. Rigour scaled to tier; critical tier gets all seven, continuous monitoring and human review before release.

"NIST, FINRA, the EU?"

NIST AI 600-1 names twelve generative-AI risks; the ones a bank cares about are confabulation, information integrity, information security, data privacy, harmful bias, human-AI configuration and value-chain integration. FINRA's 2026 report has its first GenAI section: technology-neutral rules apply; test agents for privacy, integrity, reliability and accuracy; monitor with output logs; keep AI-enabled communications in books and records; its agent-specific risks are acting without human validation, exceeding intended authority, reasoning that is hard to audit, mishandling sensitive data, missing industry knowledge, and badly designed reward functions. The EU AI Act is relevant for any client with EU customers; credit scoring is high-risk under it.

"How did you handle validators as a developer?"

Validation is adversarial by design and rightly so. I wrote the documentation to their template, logged seeds and parameters so they could replicate, and treated their findings as test cases. Agent-written code only shipped because testing, documentation and governance were human-owned.

Terms to own

footnote 3
SR 26-2's exclusion of generative and agentic AI from scope
decision proximity
how close an output sits to a credit, pricing or adverse-action decision
adverse-action fidelity
a consumer explanation accurately and traceably reflects the factors that drove the decision
NLI check
natural-language inference: does the source entail the generated claim
SelfCheckGPT / CoVe
consistency across samples; chain-of-verification questions
second line
independent risk function that validates and challenges the first line's models
17 Apr 2026SR 26-2 issued (Fed, OCC, FDIC)
4 levelslow · moderate · high · critical
7 dimensionsof output evaluation
12 risksNIST AI 600-1
14.2% → 3.1%hallucination rate in one governed pilot (literature)
Trap. Citing SR 11-7 as current, or claiming SR 26-2 "covers" GenAI. It excludes it, and that exclusion is the whole opportunity.
9 · Technical area

Data, access control and privacy

Level 1. Access control has to travel with retrieval. If the store lets anyone read it, no prompt downstream saves you. Entitlements are enforced at query time with the user's identity, PII is redacted before the model sees it where the task allows, tenants are partitioned in the vector store, and secrets never enter a prompt.
Probes and model answers
"How do you enforce document permissions in RAG, specifically?"

Three options: pre-filter the candidate set by the user's entitlements (metadata filters on the index), post-filter results against an authorisation service, or run retrieval on-behalf-of the user so the data platform's own row and column security applies. Prefer the third where the platform supports it, with pre-filtering for performance. Never rely on the model to decide what the user may see.

"On Databricks?"

An AI Search (Vector Search) index is a Unity Catalog object with object-level grants, but no row filters or column masks on the index itself; row-level security applies to the Delta table behind it. So either filter the index by the user's accessible document set at query time (filter API) or use on-behalf-of-user authorisation in Databricks Apps so Unity Catalog enforces the user's existing permissions, row filters and masks included.

"PII?"

Detect and redact before retrieval and before the model where the task does not need it (Presidio or the platform's PII guardrail); tokenise identifiers so joins still work; log what was redacted; keep raw PII in the system of record, not the vector store.

"Data quality is the existential risk; what do you do about it?"

Lineage from source to chunk, document provenance in metadata, validation at ingestion (schema, dates, duplicates), and evals that fail on stale or contradictory sources. The retrieval eval set includes cases that only pass if the right version is cited.

"The lesson from the RLS incident?"

The anonymous role had full read access and nobody noticed because the application looked fine. In AI systems the equivalent is a vector index with object-level access and no entitlement filter: every similar document is returned regardless of who asks. Test authorisation adversarially, with revocation and session-isolation cases, every release.

Terms to own

RLS / ABAC
row-level security; attribute-based access control
OBO
on-behalf-of: the user's own token is forwarded so the platform enforces their permissions
tenant isolation
one tenant's vectors can never be retrieved for another
lineage
traceable path from source record to the chunk a model cited
Trap. "The system prompt tells the model not to reveal other customers' data." That is not a control.
10 · Technical area

Platforms the team actually uses

Level 1. EY's own stack is Microsoft (Azure AI Foundry for orchestration, Fabric for data) and NVIDIA (NIM model catalogue, NeMo Guardrails) under the EY.ai Agentic Platform, with Databricks on the FS data side, which Mudit leads. My platform work has been Snowflake; the concepts transfer and I would be fluent on Databricks inside the first engagement.
Probes and model answers
"Databricks: what would you use and why?"

Mosaic AI Agent Framework (GA in 2026) to build and register agents in Unity Catalog with versions and lineage; Vector Search (now AI Search) over Delta tables; Model Serving; MLflow 3 for tracing, evaluation and monitoring; Agent Bricks for task-first agents. Unity AI Gateway as the runtime control plane: unified spend visibility, cost attribution by user, team, tool and use case, hard spend caps, smart routing by task complexity, foundation-model access control through Unity Catalog, managed MCP services, unified tracing of model and tool calls, and contextual service policies (beta) that allow, deny or require approval for agent actions with PII, injection and jailbreak protections. That is the control layer a bank asks for, already on the platform.

"Azure AI Foundry?"

Foundry Agent Service is the managed runtime: Responses API, MCP and A2A support, connected multi-agent, prompt and hosted agents; Azure AI Content Safety configurable per agent for input and output filtering; NVIDIA NIM and AgentIQ integrated; AutoGen with NeMo Guardrails for tool-use restrictions. The defence-in-depth pattern is Content Safety plus NeMo Guardrails plus a PII layer like Presidio around the agent.

"NVIDIA pieces?"

NIM: packaged inference microservices, the model catalogue layer. NeMo Guardrails: a programmable rails layer for inputs, outputs, retrieval and tool use. Relevant to me because I run local inference on a GB10 DGX Spark with vLLM and Ollama, so I know what serving a 70B model actually costs and where it breaks.

"Snowflake versus Databricks, honestly?"

Same lakehouse problems with different names: governed tables, lineage, vector retrieval, model serving, a gateway. Snowflake Cortex versus Mosaic AI plus Unity. Where they differ for an FDE is governance of agents at runtime: Unity AI Gateway is further along on budgets, policies and MCP governance. I would come to the technical round with something built on Databricks if that helps.

"Gateways in general?"

One entry point for all model calls: routing by task, cost and policy; budgets enforced before the provider call and attributed per key, team or use case; fallback chains, retries, circuit breakers; prompt and semantic caching; prompt registry and versioning; audit. Databricks' Unity AI Gateway, Azure API Management, or a dedicated gateway. I would not build one from scratch for a bank; I would configure the platform's and own the policy.

Mar 2025EY.ai Agentic Platform with NVIDIA
May 2026EY–Microsoft, $1B+ over five years, FDE delivery model
GA 2026Mosaic AI Agent Framework; Unity AI Gateway spend controls
Trap. Claiming Databricks fluency. Mudit leads EY's FS Databricks business and will know in one question. Name the gap, size it, offer evidence.
11 · Technical area

Gateway, cost and latency

Level 1. Route by task: frontier models for judgment, small or local models for the rest, with fallback chains and per-request token budgets. Cache stable prefixes, stream to the user, set budgets that stop a request before the provider is called, and report cost per case including operations and people. EY's own number: a four-cent chat assistant becomes a dollar-twenty orchestration once you add tools, planning and sub-agents.
Probes and model answers
"Where does the money go in an agent?"

Input tokens, re-sent every step: the system prompt, tool schemas and the growing transcript. A ten-step agent re-reads its context ten times. Hence prompt caching on the stable prefix, context trimming, and routing the cheap steps to cheap models.

"Prompt caching versus semantic caching?"

Prompt caching reuses the KV state of a repeated prefix on the provider's side: up to 90 percent off cached input tokens on Anthropic, about 50 percent on OpenAI, and faster time-to-first-token, with no quality change. Semantic caching returns a stored answer for a semantically similar query: useful for FAQ-shaped traffic, dangerous for anything personalised or time-sensitive. Start with prompt caching; add semantic caching only for repetitive query classes.

"Routing, what does it save?"

Reported 40 to 60 percent average cost reduction from sending simple queries to small models and reserving large models for complex reasoning, at a held quality threshold. Cascade routing (cheap first, escalate on low confidence) versus direct routing (a classifier picks once). The eval set is what proves the threshold held.

"Latency levers, ranked?"

Streaming (perceived latency), caching, routing to smaller models, prompt trimming, parallel retrieval. Measure time-to-first-token and tokens-per-second separately: TTFT is prefill and caching; throughput is decode and the serving stack. p50, p95 and p99 per workflow, not one average.

"Failures and rate limits?"

Exponential backoff with jitter; a circuit breaker after N failures; multi-provider failover through the gateway; idempotent retries; a dead-letter path. The pattern AgentForge used on its bus.

"Numbers you carry?"

List prices, November 2026, per million tokens in and out: Claude Opus 5 about $5 and $25, Sonnet 5 about $2 and $10, Haiku 5.5 about $0.10 and $0.50; GPT-5.4 about $2.50 and $15; Gemini 3.1 Pro about $2 and $12. Verify before quoting; they move monthly. AgentForge: about $0.018 per published vulnerability report versus $0.30 to $1.00 all-frontier.

50–90%input-token saving on cache hits
40–60%typical saving from routing
4¢ → $1.20EY's chat-to-orchestration cost multiple
$0.018AgentForge cost per report
p50 / p95 / p99latency per workflow
Trap. Quoting a cost per call. Quote cost per case, per outcome, including the human review it still needs.
12 · Technical area

Prompting, RAG, fine-tuning: the decision

Level 1. RAG first, in financial services especially: knowledge changes, and you need citations and an audit trail. Fine-tune for stable behaviour, format, voice or latency once prompting and retrieval have plateaued, and only when a held-out eval proves it helps. A fine-tuned model is a model the bank now has to validate.
Probes and model answers
"The hierarchy?"

Prompting, then RAG, then supervised fine-tuning, then preference optimisation (DPO), then mid-training. Each step only when the eval shows the previous one plateaued.

"LoRA, QLoRA, DPO, GRPO?"

LoRA and QLoRA: parameter-efficient adapters, roughly half a percent of parameters trained, large memory savings, low forgetting risk; the enterprise default for format, style and structured-output adherence. DPO on preference pairs (tens of thousands) when the model is helpful but rambling, unsafe or off-brand. GRPO where a verifiable reward exists and reasoning matters. All require a held-out eval and a documented training set, which is exactly what a validator will ask for.

"Structured outputs and function calling?"

Constrained decoding or schema-enforced outputs for anything a downstream system consumes; validate against the schema anyway and treat the output as untrusted input. Small adapters fix JSON adherence cheaply if prompting cannot.

"When did you choose not to fine-tune?"

Every production system so far. Retrieval plus prompting plus deterministic verification met the bar, and fine-tuning would have added a validation burden and a drift risk without a measured gain.

Trap. Fine-tuning as a reflex for "the model doesn't know our policies." It is for behaviour, not knowledge.
13 · Technical area

Observability, monitoring and the audit trail

Level 1. Trace every step: model calls, tool calls, retrieval, decisions, with tokens, cost and latency on each span. Four monitoring layers: infrastructure (latency percentiles, errors), cost (daily spend alerts), quality (eval scores on a production sample), safety (guardrail triggers, PII frequency, injection attempts). The log is tamper-evident because an examiner will read it.
Probes and model answers
"Standards?"

OpenTelemetry's GenAI semantic conventions: a gen_ai.* namespace for request model, token usage, operation duration, tool calls and agent steps, still experimental but supported by the major vendors; multi-agent conventions in progress. They cover what happened and what it cost, not whether it was good; quality scoring sits on top.

"Tooling?"

MLflow 3 tracing and evaluation on Databricks; Langfuse or LangSmith elsewhere; Unity AI Gateway's unified tracing for model and MCP activity. What matters is one trace id from user request to final action, joinable to the approval record.

"Drift?"

Input drift (query mix changes), output drift (judge scores on the 5 percent sample fall), retrieval drift (document set changes), judge drift (recalibrate against human labels on a schedule). Alert thresholds per workflow, owners named.

"Books and records?"

FINRA expects AI-enabled communications captured in firm records; the trace plus the approval queue is that record. Keep it append-only, hashed, with retention set by the records policy, not by the engineering team.

Trap. Logging prompts and responses without the decision context (which policy applied, who approved). That is a transcript, not an audit trail.
14 · Drills

System design, ten minutes each

Draw the six layers (systems of record, orchestration, gateway, knowledge and data, control layer, evidence), then fill them for the scenario. Always: clarify, walking skeleton, controls, evals, failure modes, rollback, what ships first.

Drill 1 · An agent over internal policy documents

  1. Clarify. Who asks (staff only or customer-facing), how many documents and versions, what decisions the answers inform, which answers must be human-reviewed, data residency.
  2. Skeleton, week one. Manual upload of a hundred policies, page-level chunks, hybrid retrieval, top-five with citations, no generation yet; staff read the sources.
  3. Architecture. Ingestion with versions and effective dates; entitlement filter at retrieval with the user's identity; reranker; generation with citations only from retrieved chunks; no-answer path to a named owner; approval queue for any answer that changes a process.
  4. Controls. Input and output guards, PII redaction, injection screening on documents at ingestion (poisoned-write check), gateway budgets, kill switch.
  5. Evals. 50 to 100 labelled queries, precision@5, faithfulness, citation accuracy, a violation bar of zero wrong-version answers; adversarial authorisation cases.
  6. Failure modes. Stale version cited; cross-department leakage through similarity; vector store down (degrade to keyword search); reranker latency.
  7. Rollback. Index versions; embedding-model change run in parallel for 24 hours before the flip.
  8. Ships first. Search with citations. Generation follows once retrieval passes its bar.

Drill 2 · An AML investigation assistant with write access

  1. Clarify. Which alert types, what "write" means (case notes versus SAR filing), who signs the SAR, regulator expectations, current investigation time per case.
  2. Tiering. Narrative drafting is high; anything that files or closes is critical and human-committed. Read tools and draft tools for the agent; file and close only for a human.
  3. Pattern. Plan-then-execute over a fixed tool list; retrieved transaction and KYC data treated as untrusted; the agent assembles evidence and drafts, the investigator decides.
  4. Controls. One identity per agent, scoped time-bound credentials, per-tool verb restrictions, approval gates with plain-language risk summaries, signed messages if agents hand off, tamper-evident log.
  5. Evals. Golden cases from closed investigations with known outcomes; completeness (did it surface every red flag), citation accuracy, pass^k on consistency; trajectory grading on tool use; red-team cases for injection through transaction memos.
  6. Number to anchor. Reported production results of around 50 percent time reduction per investigation; state the baseline before claiming it.
  7. Ships first. Evidence assembly with citations into the existing case tool. Drafting second. Nothing autonomous.

Drill 3 · Credit memo drafting with adverse-action language

  1. Clarify. Which products, whether the memo feeds a decision or documents one already made, who the approved reason codes come from, fair-lending review.
  2. Tiering. Memo summary from approved model outputs: moderate to high. Adverse-action reasons: critical. Evidence-grounded anchoring: the draft may only cite the approved model's factors and reason codes, never general knowledge.
  3. Architecture. Structured inputs from the decision system (not free text), template-constrained generation, citation of each factor to the model output, traceability check that fails the draft if a reason has no source.
  4. Controls and evals. All seven output dimensions; adverse-action fidelity measured against the model's actual drivers; mandatory human review; fairness testing of language across groups.
  5. Validator conversation. It is not a credit model, but it is an application a validator will review under the bank's GenAI framework: registry entry, tier, evals, monitoring, approval trail.

Drill 4 · Complaint triage across channels

  1. Clarify. Volume, channels, regulatory clocks (CFPB response windows), what "triage" decides (routing, severity, regulatory flag), and who is accountable per queue.
  2. Pattern. Routing workflow: deterministic rules for regulatory flags, a classifier for category and severity, a bounded agent only for the summary; human owns the response.
  3. Receipt. RepIntel: 1.15M of 13.65M CFPB complaints matched to 23 banks, deterministic matching first, a model only on the residue, precision stated.
  4. Evals. Stratified golden set by category; routing accuracy; false-negative rate on regulatory flags as the hard bar; drift monitoring as complaint mix changes.
15 · Drill

Decomposition: an under-specified problem

"A regional bank has three fraud-detection systems from acquisitions, with inconsistent labels. Scope the first 90 days." Interviewers score how you clarify, structure, prioritise and adapt, and whether what you propose would actually get used.

  1. Clarify before solving. Which metric moves: fraud loss, false-positive rate, analyst hours, regulatory findings? Who owns each system, what data exists, how fresh, who labels today.
  2. Name stakeholders and success. Fraud operations, model risk, technology, the acquired teams; success is one agreed label taxonomy and one measurable lift, not a unified platform.
  3. Map inputs. Three feature sets, three label definitions, overlap, volumes, label latency.
  4. Decompose by risk and value. Weeks 1 to 2: label reconciliation on a sample, by hand, with the analysts. Weeks 3 to 6: a shadow scorer over the unified labels, no decisions. Weeks 7 to 12: champion-challenger on one segment, validator engaged from week 3.
  5. Walking skeleton. One pipeline, one segment, one dashboard analysts open daily.
  6. Failure modes. Labels drift between teams; warehouse staleness beyond 24 hours; the acquired system's vendor contract; a validator who was not told until week 10.
  7. Keep talking. Check the plan against what the customer said they needed, out loud, twice.
The line interviewers listen for. "What ships first, what it is worth on its own, and what it unlocks next." Software nobody uses is the dominant failure mode of this job.
16 · Drill

Practical coding, if it comes

Unlikely at EY for this level, cheap to be ready for. Ask about edge cases before writing, narrate continuously, write readable tested code, catch your own bugs aloud, and say which edge cases you would handle given time.

TaskWhat a strong answer includes
Rate limiter (per-user and global)Token bucket or sliding window with the trade-off stated; distributed coordination (Redis) and what happens if it is down; tests at the boundary, bursts, clock skew.
Messy CSV or JSON to a clean datasetAsk the error threshold (skip or halt); handle escaped quotes, missing headers, extra columns, whitespace; schema and type coercion; report which rows failed and why.
HTTP call with retriesExponential backoff with jitter, max attempts, log status and attempt, idempotency, timeout.
Paginated API fetchPage size and total-pages logic, cursor versus offset, partial failure, rate limits.
Small RAG over a folderChunking choice defended, embedding model, in-memory store, top-k plus rerank, citations, and how you would know it works.
Top ten error types from a log fileStreaming parse, counters, timestamps, malformed lines handled, summary output.
Streaming consumer with backpressureBounded queue, drop or block policy, backoff, graceful shutdown, memory limits, monitoring.
Your angle. You ship with Claude Code and Codex daily. If asked to code live, say how you would use them and then write the function yourself; the point of the round is your reasoning.
17 · Drill

Rapid fire: fifty, one line each

Switch on Practice mode, read the question, answer out loud, tap to compare.

Workflow or agent?
Workflow by default; an agent only where the steps cannot be predicted and evals have earned it autonomy.
Why does self-attention limit context?
Compute and memory grow with the square of sequence length; the KV cache grows linearly and eats GPU memory.
Default chunk size?
About 400 to 512 tokens with 10 to 20 percent overlap; page-level for PDFs; then measure.
Why hybrid search?
Lexical catches exact identifiers; vectors catch meaning; together plus a reranker they beat either alone.
What is a cross-encoder reranker?
A model that scores query and passage together; accurate but slow, so it runs on a short candidate list.
Faithfulness?
Every claim in the answer is supported by the retrieved context.
Context precision versus recall?
Were the retrieved chunks relevant, versus did they contain what was needed.
pass^k?
Probability all k trials succeed; the metric for consistency a customer-facing agent needs.
Three LLM-judge biases?
Position, verbosity, self-preference; also format bias and calibration drift.
How do you fix position bias?
Swap AB/BA, aggregate over calls, prefer rubric or binary scoring, calibrate against human labels.
Capability versus regression eval?
Hard tasks with low pass rates as targets, versus near-100 percent suites where any drop means breakage.
How big to start an eval set?
Twenty to fifty tasks from real failures; grow; retire saturated tasks.
Golden-hash regression?
Hash expected outputs so any behavioural change on a golden case fails the build.
Direct versus indirect injection?
Typed by the user, versus hidden in content the model retrieves or receives.
Dual LLM pattern?
A privileged model with tools never reads untrusted text; a quarantined model with no tools does, returning references.
Plan-then-execute?
Fix the tool plan before reading untrusted data so injection can change arguments but not add actions.
Excessive agency?
More tools, permissions or autonomy than the task needs; the OWASP risk least-privilege fixes.
Confused deputy in MCP?
A server executing a manipulated agent's request with the server's own privileges.
Three MCP controls?
Pin tools by hash, OAuth 2.1 with audience binding, least-privilege scoped credentials per server.
What did SR 26-2 do with GenAI?
Scoped it out in footnote 3 and told banks to govern it under their own frameworks.
Two axes for tiering a GenAI use case?
Decision proximity and consumer-harm potential.
Which use case is critical tier?
Drafting adverse-action language or override rationales.
Four controls for an excluded system?
Registry entry, ongoing monitoring with thresholds, inline policy enforcement, audit trails.
Three hallucination detectors a validator accepts?
NLI against the source, self-consistency across samples, chain-of-verification questions, with a human sample to bound the error.
Why log the seed and decoding parameters?
Replication is the first thing a validator asks for.
NIST AI 600-1 risk most relevant to RAG?
Confabulation; then information integrity and data privacy.
FINRA's view of agents in one line?
Technology-neutral rules apply; agents need human-in-the-loop oversight, tracking and controls.
Where is access control enforced in RAG?
At retrieval, with the user's identity; never in the prompt.
Does a Databricks vector index have row filters?
No; the index has object-level grants, so filter by entitlement at query time or run on-behalf-of the user.
What is Unity AI Gateway?
Databricks' runtime governance layer: spend caps, routing, policies, MCP governance, unified tracing.
EY's stack in one breath?
Azure AI Foundry and Fabric with Microsoft, NIM and NeMo Guardrails with NVIDIA, Databricks on FS data.
Prompt caching saves what?
Up to 90 percent on cached input tokens and a faster first token, with no quality change.
When is semantic caching wrong?
Personalised or time-sensitive answers; it returns someone else's answer.
Routing saves how much?
Reported 40 to 60 percent at a held quality bar, proven by the eval set.
TTFT versus throughput?
Prefill time to the first token, versus decode tokens per second; different levers.
Cost per what?
Per case and per outcome, including operations and the human review.
EY's own agent cost multiple?
A four-cent chat becomes a dollar-twenty orchestration.
When to fine-tune?
Behaviour, format, voice or latency after prompting and RAG plateau, proven on a held-out eval.
LoRA in one line?
Train small adapter matrices, about half a percent of parameters, with little forgetting.
DPO in one line?
Optimise directly on preference pairs without a separate reward model.
Structured output rule?
Enforce a schema and still validate; model output is untrusted input to the consumer.
What does OpenTelemetry's GenAI convention cover?
Model, tokens, duration, tool and agent spans; not output quality.
Four monitoring layers?
Infrastructure, cost, quality, safety.
Transcript versus audit trail?
An audit trail includes which policy applied and who approved; a transcript is just the text.
Durable checkpointer, why?
So a paused graph survives restarts and a reviewer can resume days later by thread id.
Max tools per agent?
Under ten, each with under five parameters, named verb-noun.
Single or multi-agent?
Single unless phases need different prompts, tools or trust boundaries.
Why local models for a red team?
Frontier APIs refuse offensive work and that posture can change; local keeps the loop stable and the data in the box.
What ships first, always?
A walking skeleton that proves the data and integration story, then the real logic.
The sentence that ends any design answer?
Start with human approval, expand autonomy only where the evals earn it.
18 · Close

Questions for a technical interviewer

  • "What does a typical first engagement look like for this team: greenfield build, or taking a client proof of concept into production?"
  • "How do you work with a client's model validators on evaluation and governance, and at what point do they join?"
  • "Do FDEs default to EY's stack, Azure AI Foundry and NIM, or the client's, and who decides?"
  • "How much of an FDE's year goes back into reusable platform work versus embedded delivery?"
  • "Is there a shared eval harness or control-layer reference the practice reuses, or does each engagement start from scratch?"
  • "What is the Databricks weighting on current FS engagements?"
Close. "This role is the intersection I have built my career on: production AI that survives regulated review. I'm available at short notice for the next round."
19 · Sources

What this page is built on

Interview formats and scoring. Aced, FDE interview guide 2026 · Harvard FAS, FDE guide · Sundeep Teki, FDE interviews · Palantir decomposition round · Decomposition case study · Glassdoor, EY AI Engineer · Glassdoor, EY Technical Consultant · HireVue format
Fundamentals and production. 30 LLM questions senior engineers ask · AI engineer questions 2026 · Anthropic, Building effective agents · LangGraph interrupts
RAG. Chunking strategies 2026 · Chunking benchmark guide · Reranking explained · Hybrid search and reranking in production · RAG evaluation
Evals. Anthropic, Demystifying evals for AI agents (TL;DR) · τ-bench · LLM-judge bias mitigation · Judge as release gate: reliability without validity · Trajectory judges
Security. OWASP LLM Top 10 (2025) · OWASP Agentic Top 10 (2026) · Design patterns for securing LLM agents against prompt injections · MCP security risks and incidents · State of MCP security 2026
Model risk and regulation. SR 26-2: what replaced SR 11-7 · An SR 26-2-compatible framework for generative AI risk control (GAICF) · Model risk management for generative AI in financial institutions (Wells Fargo authors) · NIST AI 600-1, twelve risks · FINRA 2026 oversight report · FINRA AI guidance and agents
Data and platforms. Row-level security for a RAG agent on Databricks · RAG access control, three ways · Unity AI Gateway, Data + AI Summit 2026 · Databricks DAIS 2026 announcements · Azure AI Foundry agents · Presidio + Content Safety + NeMo Guardrails stack
Cost, latency, tuning, observability. Prompt vs semantic caching · LLM cost optimisation: routing, caching, batching · LLM API pricing, November 2026 · Enterprise AI gateways · Fine-tuning guide: LoRA, QLoRA, DPO, GRPO · RAG vs fine-tuning decision · OpenTelemetry GenAI semantic conventions · Agents in banking 2026
Prepared 11 Nov 2026 from the sources above and from Johnny's own project documentation (résumé v4, AgentForge, the GT School advisor agent). Prices and vendor feature status change monthly; verify any number before quoting it to an interviewer.