Every answer, three levels deep
Your transcript is the Level 1 layer: thirty seconds per answer, problem then constraint then decision then number. This page is what sits underneath each of those answers when the interviewer says "go deeper." Read the blue line out loud, then open the probes and answer each one before reading the model reply.
- Read The round once. It says what is scored and how 30 minutes constrains you. The practice page from 3 Nov is the ten-minute version; this page is the depth beneath it.
- Do Your systems first. Résumé deep-dives are the most reported EY format and the probes there are the ones you will actually get.
- Then one technical area per sitting. Say the Level 1 line, open each probe, answer out loud, compare.
- Run the design drills on a whiteboard, timed to ten minutes each.
- Finish with rapid fire in Practice mode until none of the fifty needs a pause.
How technical rounds work for this kind of role
Per the recruiter's brief on 10 Nov: a 30-minute HireVue conversation covering technical depth, behavioral, and current plus prior experience, with time for your questions. Requisition is the Manager posting (1708792). Contact is now Elle Gray.
What the research says gets scored
| Signal | How the interviewer detects it | Where you show it |
|---|---|---|
| Ownership | "I decided", "I shipped", named trade-offs you defended | US Bank build-versus-buy; AgentForge cost invariant |
| Production mindset | Testing, observability, failure modes, rollback mentioned unprompted | Every answer ends with the eval or the control |
| Evaluation | "How do you know it works?" answered with a harness, not "it looks right" | 263-case golden set; 80-case regression; dual judge |
| Decomposition | Clarifies before solving, names stakeholders, sequences by risk and value | Design drills; "walking skeleton first" |
| Customer empathy | Explains a system to a non-technical executive without jargon | Board Risk Committee; validators |
| Communication under pressure | Never silent; narrates thinking | Whiteboard out loud |
Formats you may meet, in order of likelihood at EY
- Résumé deep-dive with one or two managers (most reported for EY technical consulting and AI roles): "pick a project and walk me through your thinking." 30 to 45 minutes. Sometimes one simple coding question first.
- Architecture conversation: a client scenario, often lifted from your own résumé. The healthcare-RAG and policy-document designs below are the canonical shape.
- Decomposition case (Palantir origin, now common at AI labs; not reported at EY but cheap to rehearse): an under-specified business problem, 45 to 60 minutes, scored on how you clarify, structure, prioritise, and adapt, not on the answer.
- Practical coding (AI labs; rare at EY for this level): parse messy data, build a small CLI, a rate limiter, a small RAG over a folder. Readable, tested code over clever code.
- Take-home (AI labs): 4 to 8 hours building a RAG or eval harness. Not reported at EY.
What 30 minutes means
- Level 1 answers only, unless probed. Protect the design probe; let Gertie breadth be the thing that gets cut.
- If HireVue turns out to be a recorded one-way format, each answer gets a short prep window and two to three minutes to record. Lead with the Level 1 line, add one receipt, stop. Transcripts are what get scored, so speak in full sentences and name the numbers.
- Every technical answer carries one number and one control. "Zero knowledge-tier violations" is a number. "Access control enforced at retrieval" is a control.
If the interviewer is Somnath Mukherjee
Twelve years at EY through FSO Advisory, White Plains; before that IBM and IBM Research as a data and analytics practitioner, not an LLM researcher. Possibly a Managing Director since mid-2026, which would make this a judgment conversation two ranks above the seat rather than a peer screen; check the live profile. No publications or public code, so do not plan to cite their work. Their one recent public statement, from Sibos 2026: the AI conversation in financial services has moved from experimentation to real business workflows, in payments, trade finance, securities services, client servicing and financial intelligence; the live-agent story there was exception-queue handling. Read that as a hype skeptic: lead with production constraints and outcomes, never with framework names. The full read is on the practice page.
The résumé deep-dive, three levels down
Each system: the Level 1 line, then the probes an engineer would ask next. The honest limits are written in, because the interviewer who finds one you did not name stops trusting the rest.
U.S. Bank: the Monte Carlo loss platform
Probes and model answers
Inputs: loan-level exposure from the warehouse (geography, collateral type, vintage, servicer), S&P market and risk data through Snowflake. Hazard layer: event catalogues replayed against the exposure. Vulnerability: damage functions by building type (36-plus). Financial: loss to collateral, net of insurance, mapped to PD and LGD adjustments, then CECL. Outputs: expected loss, VaR and TVaR, 1-in-100, 1-in-250, 1-in-500 scenarios, concentration cuts. Monte Carlo over the event set; results versioned per run.
Conceptual soundness: why these damage functions, why this event catalogue, sensitivity to the key assumptions. Outcome analysis: back-tests where history exists, benchmark against the vendor's numbers on an overlapping portfolio, stability across seeds and sample sizes. Implementation: production matches development, inputs reconcile to the warehouse, change control. I wrote the documentation to their template before they asked.
Run-to-run variance of the tail metrics against sample size; pick the sample where the 1-in-250 moves less than a tolerance between seeds; document the seed and the tolerance. Replication of outputs is one of the first things a validator asks for, so the seed is logged with every run.
In how it was built: Claude Code agents wrote and tested code, with me reviewing and owning the methodology and the documentation. The platform itself is deterministic simulation, which is why it could pass validation cleanly. The GenAI piece on my résumé is the risk-news monitor: grounded, cited, in weekly use.
Build the evidence trail as a product from day one: the model inventory entry, the change log and the eval artefacts generated by the pipeline rather than assembled afterwards. And under SR 26-2 I would tier it by materiality explicitly and size the validation to that tier.
The risk and regulatory news monitor
Probes and model answers
A rubric per risk category applied by the model with the category definitions in context, plus deterministic rules for supervisors and instruments. Scores are logged with the rationale so a reviewer can disagree with a specific line, not the system.
Generation only over retrieved items; every brief line carries the source link; a brief that cites nothing fails the build. The archive is the evidence trail. Zero hallucination is not a claim I make; a traceable one is.
Batch, weekly, so latency is irrelevant and cost is dominated by input tokens; prompt caching on the stable rubric and summarisation of long items keep it to a few dollars a run.
AgentForge: continuous red-teaming for clinical LLM workflows
Probes and model answers
We already needed Postgres for the coverage map and cost meter; one typed jobs table with kind, payload, state, parent_id, attempts, last_error and dead-letter gave us idempotent agents and a queryable audit trail. Kafka would add operations for a throughput we never approached.
Independence enforced by the bus, not goodwill: the red team never sees the judge's rubric and the judge never sees the mutation chain. Dual judge, two independent model instances, agreement is a boolean. Rubrics are version-locked; the judge is re-evaluated weekly against a frozen golden set, and if accuracy drops the rubric version bumps rather than drifting silently.
Frontier APIs refuse offensive workflows by training, and that refusal posture can change with any vendor update. Local 70B-class models without that tuning keep the attack loop stable, and no PHI-shaped payload leaves the box, which removes a whole category of BAA questions for a hospital CISO.
A cost meter per agent per campaign halts mutation when signal-per-dollar drops below a threshold and forces the orchestrator to redirect. Every job carries attempts and a dead-letter timestamp. Frontier spend is tied to confirmed-exploit count, not mutation volume, so ten times the testing does not mean ten times the bill.
Watchdog: every agent heartbeats every ten seconds; the watchdog samples outputs for schema and behaviour drift and is the only agent allowed to issue restarts. Regression replays use character-shingle Jaccard similarity to flag drift in the target's responses.
Six categories: prompt injection (direct, indirect, multi-turn), data exfiltration and authorisation bypass, state and context corruption, tool misuse (unintended invocation, parameter tampering, recursion), denial of service and cost amplification, identity escalation. Each scored on attack surface, impact, difficulty and existing defences, and kept as a living document.
The judge ran in stub mode for part of the build, documented as such; the target was a mock clinical co-pilot with four deliberate vulnerabilities; the public deploy was phase two. I'd rather say that than have it found.
Gertie: a governed agent for a school
Probes and model answers
Because the steps that must never vary (authorisation, knowledge-tier check, approval) have to be nodes the model cannot skip. A loop lets the model decide the control flow; a graph lets me decide it and let the model fill in the language. Autonomy is risk-tiered per channel, not granted globally.
Input side: injection screening, PII and off-scope detection, length and rate limits. Retrieval side: knowledge-tier enforcement, entitlement checks before any detail read, repeated on every turn, poisoned-write screening. Output side: schema validation, citation presence, no numbers that did not come from a tool, refusal templates that never echo foreign names or IDs. Process side: approval queue, kill switch, max iterations.
Golden cases per route with fixed reference replies, run with repeats so flaky behaviour shows as variance, independent expected-versus-observed assertions, plus adversarial authorisation, revocation and session-isolation cases. An unavailable source must never become a zero. CI checks the suite; the bar was violations, not an average score.
The controls are the enterprise pattern already: governed knowledge base, entitlement-aware retrieval, intent taxonomy with handoff to an accountable owner, evals as the release gate, tracing and cost controls. What changes at scale is the infrastructure around them: managed model access through a gateway, multi-tenant isolation, an identity provider instead of a portal session, a records system for the approval trail.
Diagnosed that the anonymous database role had full read access, verified the fix, and separately root-caused a two-month lead-capture outage to one new required field silently breaking a sync. The lesson I carry into AI work: access control has to travel with retrieval; no prompt downstream saves you if the store lets anyone read it.
The multi-agent delivery pipeline, Cortex and RepIntel
Probes and model answers
Same reason a validator is independent of a developer: an agent grading its own work shares its blind spots. The reviewer sees the diff and the guardrails, not the implementer's reasoning.
Deterministic first: normalised legal names, subsidiaries and brand aliases; fuzzy matching with thresholds second; a model only on the ambiguous residue, with a sample reviewed by hand to put an error bound on the match rate. 1.15M of 13.65M is the matched set, stated with its precision, not "all complaints."
A typed boundary between assistants and tools: memory, health, file access, each scoped. The lesson relevant to banks: the tool descriptions are part of the attack surface, so they are pinned and the servers run with least privilege.
LLM fundamentals, the ones that come up
Probes and model answers
Cost, context consumption and multilingual performance. Byte-pair encoding means a number, a table or code can cost far more tokens than its character count suggests; budgets and chunk sizes are set in tokens, not words.
Cached key and value tensors from earlier tokens so each new token costs linear rather than quadratic work. It is why long contexts eat GPU memory, why output tokens are priced higher than input, and why prompt caching exists: a stable prefix's KV state is reused instead of recomputed.
Quality degrades before the hard limit ("lost in the middle"), latency and cost grow with input, and retrieval gives citations and access control that stuffing the window does not. Big windows change where the threshold is, not whether you retrieve.
Temperature scales logits before softmax: 0 is greedy and near-deterministic for extraction; 0.3 to 0.7 for conversation. Top-p keeps the smallest set of tokens whose cumulative probability passes a threshold, adapting to the model's confidence. For regulated outputs I pin both and log them, because replication is the first thing a validator asks for.
Storing weights, and optionally activations and KV cache, in fewer bits (FP8, INT8, 4-bit) to cut memory and bandwidth. On the DGX Spark I run 70B-class models quantized; the trade is a measured quality delta on my own evals, never assumed.
Both align a pretrained model to preferences. RLHF trains a reward model then optimises against it; DPO optimises directly on preference pairs without a separate reward model, simpler and cheaper. Either explains why production models sometimes over-refuse.
Separate, purpose-built embedding models for retrieval; they are cheaper, faster and better at similarity. The generation model's internal representations are not what you index.
Terms to own
- BPE
- byte-pair encoding, the tokenizer family behind most production models
- GQA / MQA
- grouped or multi-query attention: fewer KV heads, smaller cache
- MoE
- mixture of experts: only some parameters active per token; cheaper inference per quality
- prefill vs decode
- processing the prompt versus generating tokens; TTFT is prefill, tokens-per-second is decode
Retrieval-augmented generation, production grade
Probes and model answers
Default: recursive splitting at roughly 400 to 512 tokens with 10 to 20 percent overlap, page-level for PDFs and paginated policy documents, structure-aware (headings, clauses) for policies and contracts. Parent-child when answers need the surrounding section. Co-optimise with retrieval: recursive pairs with hybrid; semantic chunking pairs with dense-only. Measure where retrieval fails before adding complexity.
BM25 for lexical exactness (policy numbers, account types, product names) and vector search for meaning, run in parallel; take the top hundred or so candidates; cross-encoder rerank (Cohere Rerank or a BGE reranker) down to the five to ten the model sees. Reported end-to-end precision gains of 25 to 40 percent over naive single-stage retrieval. LLM reranking only as an optional third stage for the hardest queries.
Labelled query set of 50 to 100 questions with known relevant chunks; precision@k, recall@k and MRR; a target like precision@5 above 0.8, and if it underperforms look at the embedding model and chunking before touching the prompt. Then generation metrics on top: faithfulness, answer relevance, context precision and recall, the RAGAS set, LLM-judged and calibrated to a human sample.
Retrieval confidence below a threshold, no chunk passing the reranker, or the judge finding a claim without a supporting chunk. The response is a refusal with a route to a named owner, logged. In a bank a confident wrong answer costs more than an escalation.
Documents carry effective dates and versions; the index is rebuilt or patched on change with the version in metadata; answers cite the version; alerts when document age exceeds a threshold. Policy that changed last week must not be answered from last month's chunk.
Agentic RAG: decompose, retrieve per sub-question, synthesise, at higher latency and more model calls. Use it where single-shot retrieval measurably fails, not by default.
Knowledge changes, you need citations and an audit trail, and a fine-tuned model becomes a model the bank has to validate. RAG keeps the knowledge in a governed store the data team already controls.
Terms to own
- faithfulness
- is every claim in the answer supported by retrieved context
- context precision / recall
- were the retrieved chunks relevant; did they contain the answer
- cross-encoder
- reranker that scores query and passage jointly; accurate, slower, so used on a short list
- parent-child chunks
- retrieve small, return the enclosing section
- metadata filter
- pre- or post-filtering retrieval by tenant, entitlement, date, product
Agents, workflows and orchestration
Probes and model answers
Prompt chaining for fixed decomposable steps, trading latency for accuracy. Routing when inputs fall into distinct classes handled differently (triage by intent). Parallelization: sectioning for independent subtasks, voting for confidence. Orchestrator-workers when the subtasks cannot be predicted and a central model delegates. Evaluator-optimizer when there is a clear rubric and iteration helps. A fully autonomous agent only for open-ended problems where steps cannot be hard-coded and you can trust the model's decisions. Anthropic's framing, and it matches how I built Gertie: routing plus chaining, with one bounded agent for drafting.
Single by default. Multi-agent only when the task has distinct phases that benefit from different prompts, tools or trust boundaries, because it multiplies debugging complexity and tokens. AgentForge is multi-agent precisely for the trust boundary: red team and judge must not share context.
Verb-noun names, a one-line purpose, typed parameters with documentation, a defined output format; fewer than five parameters and fewer than ten tools per agent; examples and edge cases in the description; formats close to natural text; poka-yoke so the obvious misuse is impossible. Tools return data, never instructions.
An interrupt node before any state-changing tool call; state persisted to a durable checkpointer (Postgres, not memory) so the thread survives a restart or a reviewer who comes back tomorrow; the API returns immediately with a thread id; resume with the reviewer's decision. A max-iterations guard on every tool loop.
A standard typed boundary for tools and context, so an agent can reach Jira, SharePoint or a bank's APIs through governed servers instead of bespoke glue. It also makes tool descriptions part of the supply chain, so they are pinned by hash, allowlisted, and run with per-tool scopes and OAuth audience binding.
Collaborative assistance (drafts for a human), human-approved automation (prepares an action, a human commits it), autonomous operation (restricted, with a kill switch). Which tier a workflow gets depends on decision proximity and consumer-harm potential. Adverse-action language and override rationales are critical tier, so always human-reviewed.
Context engineering probes
More of one, because the temptation is to fill it. The window is a budget you spend, not a place you put things: quality degrades well before the limit, especially for retrieval landing in the middle, and every token is latency and money on every turn. What earns a place: the system prompt, the tool definitions, the few results that matter, a running summary rather than raw history.
Three different tools, not interchangeable. Clearing drops stale tool results and reasoning out of the history entirely. Compaction summarises the earlier thread into a block and carries it forward when the thread still matters. External memory is for anything that must survive the session. The common mistake is summarising what should have been deleted, paying tokens to carry a summary of something irrelevant.
It is a prefix match: one byte changed anywhere invalidates everything after it. So the architecture is stable things first (frozen system prompt, deterministically ordered tool list) and volatile things after the last cache breakpoint. The failures to look for first: a timestamp or session id in the system prompt, or a dictionary serialised in a different order each run. A CI assertion on cache hit rate catches a prompt edit that bloats the prefix before it ships.
Terms to own
- ReAct
- reason, act, observe loop; the basic agent shape
- checkpointer
- durable store of graph state per thread; enables pause, resume, replay
- interrupt
- a pause point in the graph awaiting external input
- A2A
- agent-to-agent protocol; messages between agents need auth and integrity like any API
- kill switch
- halts an agent's actions and revokes its credentials immediately
Evaluation: the differentiator in AI roles
Probes and model answers
Task (inputs plus success criteria), trial (one attempt; run several), grader, transcript (everything: outputs, tool calls, intermediate state), outcome (the final environment state, not the agent's claim). You evaluate the model and the harness together.
Code-based (unit tests, regex, schema checks): fast, objective, brittle when several answers are valid. Model-based (LLM-as-judge with a rubric or pairwise): handles open-ended output, non-deterministic, costs money. Human: highest quality, used mostly to calibrate the model judge on a stratified sample.
pass@k is the chance of at least one success in k tries, fine when one good answer suffices. pass^k is the chance all k succeed, what a customer-facing agent needs: 75 percent per trial gives only 42 percent pass^3. I report pass^k for anything a human will trust.
Position bias in pairwise comparisons, verbosity bias, self-preference for its own family's outputs, format bias, and calibration drift over time. Mitigations: binary or rubric scoring over free scores, AB/BA swapping with position bias checked below a tolerance, aggregation over several calls, and periodic calibration against a held-out human-labelled set. A judge with high test-retest repeatability can still be badly biased; repeatability is not validity.
Start at 20 to 50 tasks sourced from real failures, grow to a few hundred stratified by difficulty; retire tasks once they saturate above roughly 80 percent; keep capability evals (hard, low pass rate, a target) separate from regression evals (near 100 percent, a drop means breakage). Read the transcripts: a flagship benchmark score jumped from 42 to 95 percent once grading bugs and ambiguous specs were fixed. Evals are infrastructure with an owner.
Trajectory grading: tool choices, argument correctness, recovery from errors, end state. Simulated users for multi-turn (τ-bench style) checking end state, transcript constraints and interaction quality. New judge biases appear: trajectory-length bias, hallucination blindness, argument blindness. Deterministic checks wherever possible, judge only for the rest.
Golden outputs hashed; any behavioural change on a golden case fails the build; AgentForge replays confirmed exploits deterministically on every target deploy with no model in the loop; Gertie runs each golden with repeats so flakiness shows as variance. Cost is a tested invariant too: the all-local test fails if any platform agent calls a billable endpoint.
Terms to own
- golden set
- curated cases with agreed correct outputs; the acceptance test
- capability vs regression
- stretch targets versus guard rails against backsliding
- saturation
- when a task stops discriminating; retire it
- calibration
- agreement of judge scores with human labels; measured, then monitored
- trajectory
- the sequence of tool calls and states an agent took
Security: injection, agency, supply chain
Probes and model answers
LLM01 prompt injection, LLM02 sensitive information disclosure, LLM06 excessive agency, LLM08 vector and embedding weaknesses (cross-tenant retrieval, poisoned content), LLM05 improper output handling (model text reaching SQL, HTML or a shell), LLM10 unbounded consumption (denial of wallet). The rest: supply chain, data poisoning, system-prompt leakage, misinformation. Assume the system prompt leaks: never put secrets or access logic in it.
OWASP's 2026 agentic ten: goal hijack, tool misuse, identity and privilege abuse, agentic supply chain, unexpected code execution, memory and context poisoning, insecure inter-agent communication, cascading failures, human-agent trust exploitation, rogue agents. The controls are mostly identity and plumbing: one identity per agent with scoped time-bound credentials, per-tool verb restrictions and pre-execution policy checks, pinned tool definitions, sandboxed execution, signed inter-agent messages, circuit breakers, plain-language risk summaries before approval, a declared manifest of expected tools and a kill switch.
From the Beurer-Kellner et al. paper: action-selector (the model only picks from predefined actions, no feedback from tools into the model); plan-then-execute (the plan is fixed before untrusted data is read, so injection can alter arguments but not add actions); LLM map-reduce (isolated sub-agents process each untrusted document with no tools, outputs constrained); dual LLM (a privileged model with tools never reads untrusted text; a quarantined model with no tools does, returning symbolic references the orchestrator dereferences); code-then-execute (the model writes a program that calls tools and quarantined models); context minimization (strip the user prompt before the reply). The principle: untrusted input must never be able to trigger a consequential action. Gertie is action-selector plus plan-then-execute; AgentForge's judge is quarantined from the red team's text.
Spotlighting marks untrusted spans (delimiters, encoding) so the model can tell data from instructions; detectors (classifier models) catch known patterns. Both raise the bar; neither is a guarantee. They sit in front of the structural patterns, not instead of them.
EchoLeak, mid-2025: a zero-click indirect injection in an enterprise copilot that exfiltrated data through retrieved content. It is the argument for retrieval-side controls and for output handling: model output is untrusted input to whatever consumes it.
Tool descriptions, schemas and returned data are trusted context by default, so a compromised or malicious server can steer the agent (tool poisoning) and the server executes with its own privileges (confused deputy). Controls: allowlist and pin tools by content hash, OAuth 2.1 with audience binding per server, least-privilege scoped credentials per server, sandboxing, human approval for high-impact actions, logging. A 2026 survey put tool-permission scoping at under a fifth of deployments; that is the gap an FDE closes.
Terms to own
- indirect injection
- instructions hidden in content the model retrieves or is sent
- excessive agency
- more tools, permissions or autonomy than the task needs
- confused deputy
- a privileged component acting on an unprivileged caller's request
- denial of wallet
- inference-cost abuse through expensive or high-volume requests
- audience binding
- a token valid only for the server it was issued for
Model risk, validation and the regulators
Probes and model answers
Narrower model definition ("complex" quantitative methods; spreadsheet arithmetic and deterministic rules are out); risk-based proportionality instead of fixed annual validation; validation quality judged by rigour rather than organisational structure; vendor models still in scope even when code and data are withheld; and footnote 3 removing generative and agentic AI from scope while telling banks to govern them anyway. The agencies have said an RFI on AI is coming.
Four things at minimum: a registry entry with owner, lifecycle status and versions; ongoing monitoring of drift, hallucination and tool behaviour with alert thresholds; inline policy enforcement, deterministic rules plus model-based scoring on requests and responses; and audit trails capturing enforcement decisions and the policy that applied. Then the evidence that the evals and controls exist and ran.
Two axes: decision proximity (how close to credit, pricing, adverse action) and consumer-harm potential. Low: grammar and formatting. Moderate: summaries of validation reports. High: summarising monitoring alerts before distribution. Critical: drafting adverse-action language or override rationales, which need traceability to approved codes and mandatory human review. Gate conditions before any of it: no autonomous decision authority, anchored to approved models and policies, bounded role, feasible oversight.
Conceptual soundness: rationale for the foundation model (a literature review of its known weaknesses), data quality (PII leakage, sampling bias, annotation quality), model specification (prompts, decoding parameters, fine-tuning choices), explainability (context source for each generated fact), fairness testing with guardrail models, benchmarking against a simpler method. Outcome analysis: performance metrics and replication (log the seed and decoding parameters), robustness under perturbation, weakness detection by clustering inputs, hallucination detection (NLI against the source, self-consistency across samples, chain-of-verification), toxicity. Then implementation testing: dev matches prod, user and usage controls, human-in-the-loop controls, change management.
Accuracy, citation accuracy, completeness (capture rate), relevance, hallucination (frequency and severity), traceability, latency. Rigour scaled to tier; critical tier gets all seven, continuous monitoring and human review before release.
NIST AI 600-1 names twelve generative-AI risks; the ones a bank cares about are confabulation, information integrity, information security, data privacy, harmful bias, human-AI configuration and value-chain integration. FINRA's 2026 report has its first GenAI section: technology-neutral rules apply; test agents for privacy, integrity, reliability and accuracy; monitor with output logs; keep AI-enabled communications in books and records; its agent-specific risks are acting without human validation, exceeding intended authority, reasoning that is hard to audit, mishandling sensitive data, missing industry knowledge, and badly designed reward functions. The EU AI Act is relevant for any client with EU customers; credit scoring is high-risk under it.
Validation is adversarial by design and rightly so. I wrote the documentation to their template, logged seeds and parameters so they could replicate, and treated their findings as test cases. Agent-written code only shipped because testing, documentation and governance were human-owned.
Terms to own
- footnote 3
- SR 26-2's exclusion of generative and agentic AI from scope
- decision proximity
- how close an output sits to a credit, pricing or adverse-action decision
- adverse-action fidelity
- a consumer explanation accurately and traceably reflects the factors that drove the decision
- NLI check
- natural-language inference: does the source entail the generated claim
- SelfCheckGPT / CoVe
- consistency across samples; chain-of-verification questions
- second line
- independent risk function that validates and challenges the first line's models
Data, access control and privacy
Probes and model answers
Three options: pre-filter the candidate set by the user's entitlements (metadata filters on the index), post-filter results against an authorisation service, or run retrieval on-behalf-of the user so the data platform's own row and column security applies. Prefer the third where the platform supports it, with pre-filtering for performance. Never rely on the model to decide what the user may see.
An AI Search (Vector Search) index is a Unity Catalog object with object-level grants, but no row filters or column masks on the index itself; row-level security applies to the Delta table behind it. So either filter the index by the user's accessible document set at query time (filter API) or use on-behalf-of-user authorisation in Databricks Apps so Unity Catalog enforces the user's existing permissions, row filters and masks included.
Detect and redact before retrieval and before the model where the task does not need it (Presidio or the platform's PII guardrail); tokenise identifiers so joins still work; log what was redacted; keep raw PII in the system of record, not the vector store.
Lineage from source to chunk, document provenance in metadata, validation at ingestion (schema, dates, duplicates), and evals that fail on stale or contradictory sources. The retrieval eval set includes cases that only pass if the right version is cited.
The anonymous role had full read access and nobody noticed because the application looked fine. In AI systems the equivalent is a vector index with object-level access and no entitlement filter: every similar document is returned regardless of who asks. Test authorisation adversarially, with revocation and session-isolation cases, every release.
Terms to own
- RLS / ABAC
- row-level security; attribute-based access control
- OBO
- on-behalf-of: the user's own token is forwarded so the platform enforces their permissions
- tenant isolation
- one tenant's vectors can never be retrieved for another
- lineage
- traceable path from source record to the chunk a model cited
Platforms the team actually uses
Probes and model answers
Mosaic AI Agent Framework (GA in 2026) to build and register agents in Unity Catalog with versions and lineage; Vector Search (now AI Search) over Delta tables; Model Serving; MLflow 3 for tracing, evaluation and monitoring; Agent Bricks for task-first agents. Unity AI Gateway as the runtime control plane: unified spend visibility, cost attribution by user, team, tool and use case, hard spend caps, smart routing by task complexity, foundation-model access control through Unity Catalog, managed MCP services, unified tracing of model and tool calls, and contextual service policies (beta) that allow, deny or require approval for agent actions with PII, injection and jailbreak protections. That is the control layer a bank asks for, already on the platform.
Foundry Agent Service is the managed runtime: Responses API, MCP and A2A support, connected multi-agent, prompt and hosted agents; Azure AI Content Safety configurable per agent for input and output filtering; NVIDIA NIM and AgentIQ integrated; AutoGen with NeMo Guardrails for tool-use restrictions. The defence-in-depth pattern is Content Safety plus NeMo Guardrails plus a PII layer like Presidio around the agent.
NIM: packaged inference microservices, the model catalogue layer. NeMo Guardrails: a programmable rails layer for inputs, outputs, retrieval and tool use. Relevant to me because I run local inference on a GB10 DGX Spark with vLLM and Ollama, so I know what serving a 70B model actually costs and where it breaks.
Same lakehouse problems with different names: governed tables, lineage, vector retrieval, model serving, a gateway. Snowflake Cortex versus Mosaic AI plus Unity. Where they differ for an FDE is governance of agents at runtime: Unity AI Gateway is further along on budgets, policies and MCP governance. I would come to the technical round with something built on Databricks if that helps.
One entry point for all model calls: routing by task, cost and policy; budgets enforced before the provider call and attributed per key, team or use case; fallback chains, retries, circuit breakers; prompt and semantic caching; prompt registry and versioning; audit. Databricks' Unity AI Gateway, Azure API Management, or a dedicated gateway. I would not build one from scratch for a bank; I would configure the platform's and own the policy.
Gateway, cost and latency
Probes and model answers
Input tokens, re-sent every step: the system prompt, tool schemas and the growing transcript. A ten-step agent re-reads its context ten times. Hence prompt caching on the stable prefix, context trimming, and routing the cheap steps to cheap models.
Prompt caching reuses the KV state of a repeated prefix on the provider's side: up to 90 percent off cached input tokens on Anthropic, about 50 percent on OpenAI, and faster time-to-first-token, with no quality change. Semantic caching returns a stored answer for a semantically similar query: useful for FAQ-shaped traffic, dangerous for anything personalised or time-sensitive. Start with prompt caching; add semantic caching only for repetitive query classes.
Reported 40 to 60 percent average cost reduction from sending simple queries to small models and reserving large models for complex reasoning, at a held quality threshold. Cascade routing (cheap first, escalate on low confidence) versus direct routing (a classifier picks once). The eval set is what proves the threshold held.
Streaming (perceived latency), caching, routing to smaller models, prompt trimming, parallel retrieval. Measure time-to-first-token and tokens-per-second separately: TTFT is prefill and caching; throughput is decode and the serving stack. p50, p95 and p99 per workflow, not one average.
Exponential backoff with jitter; a circuit breaker after N failures; multi-provider failover through the gateway; idempotent retries; a dead-letter path. The pattern AgentForge used on its bus.
List prices, November 2026, per million tokens in and out: Claude Opus 5 about $5 and $25, Sonnet 5 about $2 and $10, Haiku 5.5 about $0.10 and $0.50; GPT-5.4 about $2.50 and $15; Gemini 3.1 Pro about $2 and $12. Verify before quoting; they move monthly. AgentForge: about $0.018 per published vulnerability report versus $0.30 to $1.00 all-frontier.
Prompting, RAG, fine-tuning: the decision
Probes and model answers
Prompting, then RAG, then supervised fine-tuning, then preference optimisation (DPO), then mid-training. Each step only when the eval shows the previous one plateaued.
LoRA and QLoRA: parameter-efficient adapters, roughly half a percent of parameters trained, large memory savings, low forgetting risk; the enterprise default for format, style and structured-output adherence. DPO on preference pairs (tens of thousands) when the model is helpful but rambling, unsafe or off-brand. GRPO where a verifiable reward exists and reasoning matters. All require a held-out eval and a documented training set, which is exactly what a validator will ask for.
Constrained decoding or schema-enforced outputs for anything a downstream system consumes; validate against the schema anyway and treat the output as untrusted input. Small adapters fix JSON adherence cheaply if prompting cannot.
Every production system so far. Retrieval plus prompting plus deterministic verification met the bar, and fine-tuning would have added a validation burden and a drift risk without a measured gain.
Observability, monitoring and the audit trail
Probes and model answers
OpenTelemetry's GenAI semantic conventions: a gen_ai.* namespace for request model, token usage, operation duration, tool calls and agent steps, still experimental but supported by the major vendors; multi-agent conventions in progress. They cover what happened and what it cost, not whether it was good; quality scoring sits on top.
MLflow 3 tracing and evaluation on Databricks; Langfuse or LangSmith elsewhere; Unity AI Gateway's unified tracing for model and MCP activity. What matters is one trace id from user request to final action, joinable to the approval record.
Input drift (query mix changes), output drift (judge scores on the 5 percent sample fall), retrieval drift (document set changes), judge drift (recalibrate against human labels on a schedule). Alert thresholds per workflow, owners named.
FINRA expects AI-enabled communications captured in firm records; the trace plus the approval queue is that record. Keep it append-only, hashed, with retention set by the records policy, not by the engineering team.
System design, ten minutes each
Draw the six layers (systems of record, orchestration, gateway, knowledge and data, control layer, evidence), then fill them for the scenario. Always: clarify, walking skeleton, controls, evals, failure modes, rollback, what ships first.
Drill 1 · An agent over internal policy documents
- Clarify. Who asks (staff only or customer-facing), how many documents and versions, what decisions the answers inform, which answers must be human-reviewed, data residency.
- Skeleton, week one. Manual upload of a hundred policies, page-level chunks, hybrid retrieval, top-five with citations, no generation yet; staff read the sources.
- Architecture. Ingestion with versions and effective dates; entitlement filter at retrieval with the user's identity; reranker; generation with citations only from retrieved chunks; no-answer path to a named owner; approval queue for any answer that changes a process.
- Controls. Input and output guards, PII redaction, injection screening on documents at ingestion (poisoned-write check), gateway budgets, kill switch.
- Evals. 50 to 100 labelled queries, precision@5, faithfulness, citation accuracy, a violation bar of zero wrong-version answers; adversarial authorisation cases.
- Failure modes. Stale version cited; cross-department leakage through similarity; vector store down (degrade to keyword search); reranker latency.
- Rollback. Index versions; embedding-model change run in parallel for 24 hours before the flip.
- Ships first. Search with citations. Generation follows once retrieval passes its bar.
Drill 2 · An AML investigation assistant with write access
- Clarify. Which alert types, what "write" means (case notes versus SAR filing), who signs the SAR, regulator expectations, current investigation time per case.
- Tiering. Narrative drafting is high; anything that files or closes is critical and human-committed. Read tools and draft tools for the agent; file and close only for a human.
- Pattern. Plan-then-execute over a fixed tool list; retrieved transaction and KYC data treated as untrusted; the agent assembles evidence and drafts, the investigator decides.
- Controls. One identity per agent, scoped time-bound credentials, per-tool verb restrictions, approval gates with plain-language risk summaries, signed messages if agents hand off, tamper-evident log.
- Evals. Golden cases from closed investigations with known outcomes; completeness (did it surface every red flag), citation accuracy, pass^k on consistency; trajectory grading on tool use; red-team cases for injection through transaction memos.
- Number to anchor. Reported production results of around 50 percent time reduction per investigation; state the baseline before claiming it.
- Ships first. Evidence assembly with citations into the existing case tool. Drafting second. Nothing autonomous.
Drill 3 · Credit memo drafting with adverse-action language
- Clarify. Which products, whether the memo feeds a decision or documents one already made, who the approved reason codes come from, fair-lending review.
- Tiering. Memo summary from approved model outputs: moderate to high. Adverse-action reasons: critical. Evidence-grounded anchoring: the draft may only cite the approved model's factors and reason codes, never general knowledge.
- Architecture. Structured inputs from the decision system (not free text), template-constrained generation, citation of each factor to the model output, traceability check that fails the draft if a reason has no source.
- Controls and evals. All seven output dimensions; adverse-action fidelity measured against the model's actual drivers; mandatory human review; fairness testing of language across groups.
- Validator conversation. It is not a credit model, but it is an application a validator will review under the bank's GenAI framework: registry entry, tier, evals, monitoring, approval trail.
Drill 4 · Complaint triage across channels
- Clarify. Volume, channels, regulatory clocks (CFPB response windows), what "triage" decides (routing, severity, regulatory flag), and who is accountable per queue.
- Pattern. Routing workflow: deterministic rules for regulatory flags, a classifier for category and severity, a bounded agent only for the summary; human owns the response.
- Receipt. RepIntel: 1.15M of 13.65M CFPB complaints matched to 23 banks, deterministic matching first, a model only on the residue, precision stated.
- Evals. Stratified golden set by category; routing accuracy; false-negative rate on regulatory flags as the hard bar; drift monitoring as complaint mix changes.
Decomposition: an under-specified problem
"A regional bank has three fraud-detection systems from acquisitions, with inconsistent labels. Scope the first 90 days." Interviewers score how you clarify, structure, prioritise and adapt, and whether what you propose would actually get used.
- Clarify before solving. Which metric moves: fraud loss, false-positive rate, analyst hours, regulatory findings? Who owns each system, what data exists, how fresh, who labels today.
- Name stakeholders and success. Fraud operations, model risk, technology, the acquired teams; success is one agreed label taxonomy and one measurable lift, not a unified platform.
- Map inputs. Three feature sets, three label definitions, overlap, volumes, label latency.
- Decompose by risk and value. Weeks 1 to 2: label reconciliation on a sample, by hand, with the analysts. Weeks 3 to 6: a shadow scorer over the unified labels, no decisions. Weeks 7 to 12: champion-challenger on one segment, validator engaged from week 3.
- Walking skeleton. One pipeline, one segment, one dashboard analysts open daily.
- Failure modes. Labels drift between teams; warehouse staleness beyond 24 hours; the acquired system's vendor contract; a validator who was not told until week 10.
- Keep talking. Check the plan against what the customer said they needed, out loud, twice.
Practical coding, if it comes
Unlikely at EY for this level, cheap to be ready for. Ask about edge cases before writing, narrate continuously, write readable tested code, catch your own bugs aloud, and say which edge cases you would handle given time.
| Task | What a strong answer includes |
|---|---|
| Rate limiter (per-user and global) | Token bucket or sliding window with the trade-off stated; distributed coordination (Redis) and what happens if it is down; tests at the boundary, bursts, clock skew. |
| Messy CSV or JSON to a clean dataset | Ask the error threshold (skip or halt); handle escaped quotes, missing headers, extra columns, whitespace; schema and type coercion; report which rows failed and why. |
| HTTP call with retries | Exponential backoff with jitter, max attempts, log status and attempt, idempotency, timeout. |
| Paginated API fetch | Page size and total-pages logic, cursor versus offset, partial failure, rate limits. |
| Small RAG over a folder | Chunking choice defended, embedding model, in-memory store, top-k plus rerank, citations, and how you would know it works. |
| Top ten error types from a log file | Streaming parse, counters, timestamps, malformed lines handled, summary output. |
| Streaming consumer with backpressure | Bounded queue, drop or block policy, backoff, graceful shutdown, memory limits, monitoring. |
Rapid fire: fifty, one line each
Switch on Practice mode, read the question, answer out loud, tap to compare.
Questions for a technical interviewer
- "What does a typical first engagement look like for this team: greenfield build, or taking a client proof of concept into production?"
- "How do you work with a client's model validators on evaluation and governance, and at what point do they join?"
- "Do FDEs default to EY's stack, Azure AI Foundry and NIM, or the client's, and who decides?"
- "How much of an FDE's year goes back into reusable platform work versus embedded delivery?"
- "Is there a shared eval harness or control-layer reference the practice reuses, or does each engagement start from scratch?"
- "What is the Databricks weighting on current FS engagements?"
What this page is built on
Fundamentals and production. 30 LLM questions senior engineers ask · AI engineer questions 2026 · Anthropic, Building effective agents · LangGraph interrupts
RAG. Chunking strategies 2026 · Chunking benchmark guide · Reranking explained · Hybrid search and reranking in production · RAG evaluation
Evals. Anthropic, Demystifying evals for AI agents (TL;DR) · τ-bench · LLM-judge bias mitigation · Judge as release gate: reliability without validity · Trajectory judges
Security. OWASP LLM Top 10 (2025) · OWASP Agentic Top 10 (2026) · Design patterns for securing LLM agents against prompt injections · MCP security risks and incidents · State of MCP security 2026
Model risk and regulation. SR 26-2: what replaced SR 11-7 · An SR 26-2-compatible framework for generative AI risk control (GAICF) · Model risk management for generative AI in financial institutions (Wells Fargo authors) · NIST AI 600-1, twelve risks · FINRA 2026 oversight report · FINRA AI guidance and agents
Data and platforms. Row-level security for a RAG agent on Databricks · RAG access control, three ways · Unity AI Gateway, Data + AI Summit 2026 · Databricks DAIS 2026 announcements · Azure AI Foundry agents · Presidio + Content Safety + NeMo Guardrails stack
Cost, latency, tuning, observability. Prompt vs semantic caching · LLM cost optimisation: routing, caching, batching · LLM API pricing, November 2026 · Enterprise AI gateways · Fine-tuning guide: LoRA, QLoRA, DPO, GRPO · RAG vs fine-tuning decision · OpenTelemetry GenAI semantic conventions · Agents in banking 2026