Customer Scenario Interviews

A strong answer demonstrates: owning the miss without excuses ("we missed it, here's why" — the upstream feed change is context, not a defense), separating what Maria needs right now (a message she can take to her VP) from the engineering work (the feed fix), and giving her a concrete, checkable next update time. The best candidates ask what her VP actually needs — the full dashboard, or one number for a meeting?

The common failure mode: defending the timeline ("we couldn't have known about the feed change") or promising a new date on the spot without knowing whether it's achievable. Both tell the interviewer this candidate will keep a customer calm for one meeting and create a second angry meeting next week.

The follow-up the interviewer should ask: "Maria says that's not good enough — her VP needs something by end of day. What do you offer, and what do you refuse to offer?"

A strong answer demonstrates: not arguing with the room. The candidate validates the complaint ("if it's slower, we built the wrong thing — help me see where"), asks to watch one real task end-to-end, and treats "we liked the spreadsheet" as data about the workflow, not ignorance. Bonus: acknowledging that adoption is part of the deliverable — a tool nobody uses is a failed tool, no matter how correct its validation logic is.

The common failure mode: explaining why the spreadsheet was bad ("but it had no validation!"). Correcting the customer in front of their team is the fastest way to lose the room, and interviewers watch for the instinct to do it.

The follow-up the interviewer should ask: "You shadow Dev and find the form really is slower for one common task, but catches errors the spreadsheet missed. How do you decide what to change?"

A strong answer demonstrates: treating Lisa as an ally, not an obstacle. The candidate separates the demo (a presentation, which can often run on anonymized or synthetic data with no real data flow) from the launch (the real thing, which waits for the review). They ask Lisa what a review-safe demo looks like, escalate the review request through the proper channel, and — the tell — they note the process failure: the review should have been requested weeks ago, and they'd change the launch checklist so this can't happen again.

The common failure mode: pressuring Lisa to "make an exception" or going around her to the sponsor. Interviewers read this as a candidate who will trade the customer's compliance posture for their own demo — exactly the instinct the enterprise security review exists to filter out.

The follow-up the interviewer should ask: "Your sponsor says the demo must use real customer data or it's meaningless. What's your answer?"

A strong answer demonstrates: not replying-all in anger, and not being defensive in the reply. The candidate acknowledges Tom's frustration, takes the contradiction problem seriously (it's real and it's theirs to solve), and proposes a reset: a short working session to reconcile the conflicting requirements with all parties in the room, with a written summary afterward so "explained three times" can't happen a fourth time. Strong candidates also tell their own manager proactively before the manager asks.

The common failure mode: litigating the email ("we did understand, his team contradicted each other") — either to Tom or to the manager. Interviewers are checking whether the candidate can absorb an unfair public criticism and still move the work forward.

The follow-up the interviewer should ask: "In the reset meeting, two of Tom's people give you directly contradictory requirements again, in front of him. What do you do in the room?"

A strong answer demonstrates: the candidate owns it immediately and completely — no "well, they didn't document it." They lay out the response in order: contain (stop the automation, assess the blast radius of the two bad records, help draft the correction), then learn (the undocumented checks existed because the team didn't trust the old process — that's a discovery failure on the FDE's part), then prevent (shadow-mode runs and sign-off gates before automating human steps in the future). The regulator angle must be named explicitly, not minimized.

The common failure mode: blaming the customer for the undocumented workaround, or treating it as purely a technical bug ("I'll add the checks to the script"). Interviewers are grading ownership and whether the candidate updates their process, not just their code.

The follow-up the interviewer should ask: "The customer now wants every future automation change approved by their team in writing before it ships. Reasonable? How do you respond?"

A strong answer demonstrates: the candidate refuses to start building from a four-word brief. Before: gather whatever exists (sample reports, who consumes them, what decisions they drive). During: ask for one concrete recent pain ("tell me about the last time a report failed you"), identify the actual users and their decisions, and push for a measurable definition of "better" (faster? fewer errors? one fewer manual step?). After: a one-page written summary of what was agreed — problem, users, success metric — sent back for confirmation. The written artifact is the tell; it proves the candidate knows alignment evaporates without it.

The common failure mode: jumping to a solution ("I'd build a dashboard with drill-downs") or asking questions with no structure — twenty random questions that leave the stakeholder more confused than before.

The follow-up the interviewer should ask: "The stakeholder can't give you a single concrete example of a bad report. Now what?"

A strong answer demonstrates: the candidate recognizes this as a risk, not a compliment. "You're the expert" means every later decision can be disowned. Strong candidates change tactics: stop asking open questions and start bringing concrete options ("Option A gives you X by Friday, Option B gives you Y next month — which matters more?"), make the stakeholder choose in writing, and escalate the disengagement itself as a project risk to their own manager early. They also check the hypothesis: maybe the stakeholder is the wrong person and the real users are two levels down.

The common failure mode: accepting the flattery and building on assumptions, or complaining about the stakeholder. The interviewer is checking whether the candidate protects themselves and the project with written decisions instead of hoping it works out.

The follow-up the interviewer should ask: "You bring two options and the stakeholder picks one, then blames you when it doesn't work. What did you miss?"

A strong answer demonstrates: the candidate refuses to pick a side or average the two requests. They reframe: Maria has a signal problem (too much noise), Tom has a coverage problem (fear of missing something). Those can both be solved — severity tiers, routing rules, digest vs. real-time channels — but only if the candidate gets both stakeholders to agree on what each alert costs (attention) and what missing one costs (incidents). The tell: proposing a joint 30-minute session with both of them rather than shuttling between them.

The common failure mode: building a compromise nobody asked for (a settings page with fifty toggles) or escalating immediately ("let my manager decide"). Interviewers want to see the candidate hold the tension themselves.

The follow-up the interviewer should ask: "In the joint session, Tom says any alert Maria's team ignores is a liability for him. How do you get to an agreement?"

A strong answer demonstrates: the candidate meets the customer where they are while smuggling in measurement. They propose a tiny, time-boxed slice ("give me two weeks and ten real cases"), define the metric themselves as a starting draft ("here's how I'll measure whether this helped — tell me what's wrong with it"), and make the customer's "we'll know it" concrete by asking them to judge the ten cases blind. Strong candidates name the failure mode honestly: without a metric, every demo becomes an argument about taste.

The common failure mode: either lecturing the customer about evals or happily building with no success criteria. The interviewer is checking whether the candidate can introduce rigor without making the customer feel managed.

The follow-up the interviewer should ask: "The ten cases come back split — five look great, five are arguable, and the customer disagrees with your scoring on three. What now?"

A strong answer demonstrates: the candidate spots the pattern early and addresses it as a process problem, not a series of individual requests. They propose a visible backlog with costs attached ("each addition moves the date by roughly X — here's the list, you rank them"), tie every addition back to the original agreed goal, and — the key move — have the scope conversation with the sponsor outside the standup, where "just one more column" can't hide behind the rhythm of the meeting. They also own their part: they let it happen three weeks without flagging it.

The common failure mode: saying yes to everything and then missing the date, or saying no bluntly in the standup and damaging the relationship. Interviewers watch for whether the candidate makes the tradeoff visible to the person who owns it.

The follow-up the interviewer should ask: "The sponsor says all ten additions are must-haves and the date can't move. What's your response?"

A strong answer demonstrates: a two-track response. Track one, immediate: can the pipeline be made resilient to the rename (a field-mapping layer, a fallback to the old names) before 10 a.m., and if not, what's the honest status for the launch meeting? Track two, structural: a renamed field shouldn't be able to kill a launch — the candidate proposes schema validation at ingestion with explicit contracts, so the next change fails safe instead of failing loud at the worst moment. The tell: the candidate tells the sponsor the real status before the 10 a.m. meeting, not during it.

The common failure mode: either hacking a silent fix with no validation ("just map the fields and ship it") or waiting for the vendor with no plan B. Interviewers are grading whether the candidate protects the launch without hiding the fragility.

The follow-up the interviewer should ask: "You get it working by 9:30, but you're not confident in the data. Do you launch?"

A strong answer demonstrates: composure and a pivot, in that order. The candidate doesn't debug live for twenty minutes while the room watches — they acknowledge it plainly ("that's not supposed to happen, and I owe you an explanation of why"), switch to a fallback (screenshots, a recorded walkthrough, the staging environment on their own machine), and use the remaining time to talk through what the demo would have shown. Afterward: a real root-cause note to the stakeholder, not silence. The best candidates add the prevention: demos run on the customer's hardware in rehearsal, or they don't run at all.

The common failure mode: live-debugging in front of the audience, blaming the laptop ("it worked on mine"), or pretending the crash didn't matter. All three tell the interviewer the candidate has never carried a demo that mattered.

The follow-up the interviewer should ask: "The stakeholder later asks: 'Can we trust this in production if it can't survive a demo?' How do you answer?"

A strong answer demonstrates: the candidate treats this as a design assumption that failed, not just an ops problem. Immediate: reduce demand (batching, caching, backoff with jitter — the standard rate-limit toolkit from the APIs stage) to buy time. Structural: the limit should have been verified during design, so the candidate proposes a spike to measure the real ceiling and reframes the feature around it. And the commercial angle must be named: the "higher tier" is a procurement decision for the customer, so the candidate lays out the options and costs honestly instead of quietly building something that depends on a contract that doesn't exist.

The common failure mode: either hammering the API harder (retries without backoff — the classic way to turn a limit into a ban) or assuming the contract will materialize. Interviewers watch for whether the candidate separates the technical fix from the commercial dependency.

The follow-up the interviewer should ask: "Caching would fix the volume, but the customer needs real-time data. Now what?"

A strong answer demonstrates: the candidate doesn't minimize it ("it's just a demo") and doesn't over-apologize into a spiral. They name what happened accurately — the system produced an unsupported claim, which is the known failure mode of retrieval-backed generation — show how they'd diagnose it (was the right document retrieved? did the model ignore it? was the source itself stale?), and reframe the pilot's purpose: this is why you run a pilot with evals before production. Strong candidates propose the concrete guardrail honestly: citation-backed answers, confidence thresholds, and human review on high-stakes outputs. They also tell the stakeholder what changes before the next demo.

The common failure mode: blaming the model ("the LLM hallucinated, nothing we can do") or promising it won't happen again. Both dodge the real question, which is whether the candidate has a system for catching wrong answers, not just hope.

The follow-up the interviewer should ask: "The stakeholder asks whether they should trust any AI answer at all now. What's your honest answer?"

A strong answer demonstrates: the candidate refuses the false choice between "migrate garbage" and "miss the date." They propose triage: quarantine the bad records into a visible, countable bucket (with reasons attached to each), migrate the clean majority on schedule, and give the customer a concrete cleanup backlog with an owner and a deadline — making "later" a scheduled thing instead of a wish. The tell: the candidate quantifies the dirt ("about 8% of records fail validation, here's the breakdown") because unmeasured data problems get dismissed and measured ones get staffed.

The common failure mode: either migrating everything silently (the customer's "just do it" becomes your production incident) or blocking the whole migration on perfect data. Interviewers are grading whether the candidate can hold quality and schedule at the same time by making the tradeoff explicit.

The follow-up the interviewer should ask: "The customer won't assign anyone to own the quarantined records. Who owns them?"

A strong answer demonstrates: the candidate names the stakes plainly — this isn't a UX tradeoff, it's removing a fraud control, and "next quarter" is not a control. They refuse the specific ask (not the customer, not the relationship — the ask), and they come with alternatives: a risk-tiered flow where low-risk accounts move fast and high-risk ones still get reviewed, or a time-boxed experiment with explicit fraud-rate monitoring and a kill switch. The tell: the candidate puts the refusal and the reasoning in writing, because verbal "we'll fix it later" agreements are how accountability evaporates.

The common failure mode: shipping it with a vague concern noted somewhere, or framing the refusal as personal discomfort ("I'm not comfortable with that") rather than a professional judgment about the customer's risk. Interviewers want to hear the candidate protect the customer from the customer's own short-term pressure.

The follow-up the interviewer should ask: "The sponsor says you're being difficult and asks your manager to replace you on the account. What do you do?"

A strong answer demonstrates: the candidate knows "just delete the file" doesn't undo exposure — chat systems keep history, caches, and backups. They treat it as an incident: contain (get the file removed through the proper channel, confirm who could have accessed it), assess (was the data real? whose? how widely was it visible?), and report (tell the customer's security contact and their own lead — this is the customer's data and the customer's incident process governs). They also fix the cause: the contractor needs a safe way to share data samples, like redacted fixtures or a governed sandbox. Deleting quietly and moving on is the failure the scenario is designed to catch.

The common failure mode: deleting the file and considering the matter closed, or panicking and blasting the discovery to a wider audience. Interviewers are checking for incident discipline: contain, assess, report through the right channel, fix the process.

The follow-up the interviewer should ask: "Your lead says reporting this will cause a huge fuss and suggests handling it quietly between you two. Now what?"

A strong answer demonstrates: the candidate doesn't get to declare the customer's policy outdated — that's the customer's governance decision, not the vendor engineer's. They pause the build, route the request to whoever owns the policy (with the requester, not behind their back), and propose what they can build meanwhile: scoped exports per tenant, which solves the likely real need without violating the policy. The tell: the candidate distinguishes "the customer asked for it" from "the customer is authorized to ask for it" — a distinction the guardrails stage of this curriculum exists to teach.

The common failure mode: building it because the customer insisted, or refusing with a lecture about compliance. Interviewers want the middle path: don't build the violating thing, do route the decision to its rightful owner, and keep delivering value on the non-violating version.

The follow-up the interviewer should ask: "The policy owner updates the policy to allow the export, in writing. Do you build it now?"

A strong answer demonstrates: a flat refusal to alter the measurement, stated calmly and professionally. The candidate explains why: the eval set and the scoring are the measuring stick — changing them to flatter the result destroys the one thing that lets the team know whether the product actually improved, and it misleads the customer into a launch decision based on fiction. They redirect: report the numbers honestly, lead with the failing slices as the roadmap ("here's exactly what we'd fix before launch"), and let the customer decide with true information. Strong candidates also note they'd document the request, because being asked to massage numbers is itself something a paper trail should exist for.

The common failure mode: complying, or refusing theatrically ("I could never do that!") without explaining the professional reasoning. Interviewers are grading whether the candidate understands why measurement integrity matters — it's what makes every future decision possible — not just whether they know the rule.

The follow-up the interviewer should ask: "The manager says it's just internal reporting, the customer will never see it. Does that change your answer?"

A strong answer demonstrates: the candidate doesn't treat this as a loyalty test — it's a trust test, and the trust at stake belongs to the customer, who just signed an extension based on fiction. Strong candidates start privately (talk to the teammate first, give them the chance to correct it themselves — there may be context you don't have), but they don't end privately: if the teammate won't correct it, the candidate escalates through the proper channel. They name the downstream harm clearly: every future commitment the team makes will be priced against results that never existed. The tell: the candidate is uncomfortable and does it anyway — interviewers watch for whether discomfort changes the answer.

The common failure mode: staying silent for the sake of team harmony, or going straight to public accusation without giving the teammate a chance to explain. The interviewer is checking for the hardest combination in professional life: loyalty to people and honesty to the customer, in the right order.

The follow-up the interviewer should ask: "You talk to the teammate and they say the outputs were 'representative of what the system will do' and it's standard practice. Your response?"