← Better Programmer course
08
Advanced
5 hours

Capstone

Lead one realistic product change from outcome and system design through rollout and review.

Learning outcome

Produce and present a practical RFC for a safe product change, connecting user value to data, reliability, delivery, observability, and rollback decisions.

Jump to a chapter
  1. 1. Define the User and Business Outcome
  2. 2. Map the Request and Data Flow
  3. 3. Choose Storage and Consistency
  4. 4. Design Performance and Reliability
  5. 5. Slice Delivery and Tests
  6. 6. Roll Out with Observability and Rollback
  7. 7. Write and Present the Final RFC
  8. Applied exercise
  9. Knowledge check

Chapter 1

Define the User and Business Outcome

Begin with the result the change must create, not with a preferred implementation.

The capstone follows one change for an online shop: let a customer cancel a paid order within ten minutes, start a refund, and return reserved items to stock. Today customers contact support, wait for an agent, and may miss the warehouse cutoff. The desired user outcome is a clear, trustworthy cancellation without support. The business outcome is fewer avoidable contacts and fewer orders packed after a cancellation request.

Use the product-leadership practice of separating outcomes from output. A cancellation button is output; fewer support contacts and a high rate of eligible self-service cancellations are outcomes. State who is affected, the current problem, the expected behavior, and what is outside scope. For this first release, changing an order or cancelling after packing begins is outside scope. That boundary prevents a useful change from becoming an undefined rewrite of fulfillment.

Turn the outcome into measurable success and safety criteria before discussing storage or queues. Measure the share of eligible requests completed without support, the time until the user sees a final state, and support contacts about cancellation. Guard against duplicate refunds, stock errors, unauthorized cancellation, and increased checkout failures. Record assumptions, such as the warehouse exposing a reliable packing-started state, and assign an owner to verify each one.

Key insight

A strong design starts with a bounded, measurable outcome and explicit safety conditions.

How to apply this

  • Describe the user problem and business result separately from the proposed interface.
  • State scope, non-goals, assumptions, and the decision owner before designing.
  • Pair success measures with safety measures that must not get worse.
Scenario — The team starts with a cancellation service

Situation

Engineers propose a new service and queue before agreeing when cancellation is allowed or how success will be measured.

Decision

Pause implementation and write the eligible user journey, non-goals, outcome measures, and safety rules first; architecture can then serve a shared target.

Chapter 2

Map the Request and Data Flow

Follow the cancellation from the customer's action through every existing boundary.

Return to the system-foundations method of tracing one request end to end. The browser sends an authenticated HTTPS request to the existing order API. The application checks that the customer owns the order, reads its current state, and records an accepted cancellation. A worker later calls the payment provider for the refund and the inventory component to release stock. The user reads the order API again to see whether cancellation is pending, completed, or needs help.

Draw each component and label each request, response, durable write, and queued message. Mark trust boundaries where authentication and authorization are checked. Mark the transaction boundary around local order changes, and show that calls to the payment provider are outside that database transaction. Add a request identifier to the initial request and carry the order and cancellation identifiers through logs and worker activity so one user action can be traced across asynchronous work.

Walk the map once for success and once for each important interruption. The browser may retry after losing the response, the worker may stop after the payment provider accepted a refund, or the inventory release may fail while the refund succeeds. This review exposes where repeated requests must be safe and where the interface needs an honest pending state. It also identifies system owners who must review the proposal rather than discovering the change during rollout.

Key insight

An end-to-end map makes boundaries, owners, durable facts, and failure gaps visible before code hides them.

How to apply this

  • Label requests, responses, writes, messages, identifiers, and ownership on one diagram.
  • Trace success, retry, timeout, and partial-failure paths across the same map.
  • Keep external calls outside the local transaction and show the resulting pending states.
Scenario — A refund occurs but the request times out

Situation

The payment provider accepts a refund, but the worker loses the response and cannot tell whether it succeeded.

Decision

Use a stable cancellation identifier when calling the provider and check the recorded provider result before retrying, so a repeated attempt cannot create a second refund.

Chapter 3

Choose Storage and Consistency

Model cancellation facts and protect the rules that must remain true under concurrency.

Apply the data-storage concepts to facts, not screens. Keep the existing order as the source of truth and add a cancellation record with a stable identifier, order identifier, requester, reason, state, refund reference, timestamps, and failure details safe for operators. A uniqueness rule allows only one active cancellation per order. The decision log should explain why the existing relational database is preferred over a new store: the data is related to orders, requires constraints, and fits known access patterns.

State the invariants and transaction boundary. Only the owning customer or an authorized operator may request cancellation; an order cannot become cancellable after packing starts; and one paid amount can be refunded at most once. In one short transaction, re-read the current order state, create the cancellation record, move the order to cancellation pending, and record work for later processing. Concurrent requests then produce one accepted result rather than two independent refunds.

Choose consistency per operation as taught in Data and Storage. Eligibility and authorization use a current authoritative read because stale state could cancel an already packed order. After acceptance, the customer's next read should show cancellation pending, providing read-your-writes behavior. The refund and stock release may complete later, so the interface names that delay instead of claiming the whole process is complete. Index the customer order-history and worker lookup paths only after writing their real query shapes.

Key insight

Storage and consistency choices should enforce named product invariants and make delayed work visible.

How to apply this

  • Model cancellation as durable facts with stable identity and constrained relationships.
  • Use one transaction for local invariant-related changes, not for external network calls.
  • Require current reads for unsafe decisions and expose honest pending states for later work.
Scenario — Two devices cancel the same order

Situation

A customer taps Cancel on a phone while a family member submits the same request on a laptop.

Decision

Protect one cancellation per order with a database uniqueness rule and return the existing cancellation state to the second request.

Chapter 4

Design Performance and Reliability

Set user-visible targets and make every asynchronous step safe to repeat and recover.

Use performance targets that match the journey. The cancellation request should acknowledge a valid request quickly at the 95th percentile, while the separate completion target measures time from acceptance to final state. Expected volume is modest, so a new cache, service, or shard is not justified. Measure API latency, queue wait, worker time, provider latency, and completion throughput to find a real bottleneck if the target is missed.

Apply the reliability-async practices to the worker path. The queued work contains stable order and cancellation identifiers, not a full mutable order copy. A worker may receive the same work more than once, so it reads durable state and makes each step safe to repeat. Retries use increasing delays for temporary provider failures and stop after a defined limit. Work that still fails remains visible for operator review rather than disappearing or retrying forever.

Set timeouts and failure behavior at each dependency boundary. If the payment provider is slow, the worker times out and retries safely while the order remains pending; it does not tell the user cancellation failed prematurely. If inventory release fails after refund, record that partial state, alert the owning team, and retry only the unfinished step. Limit worker concurrency so recovery traffic cannot overwhelm the provider, and test process loss between every durable update and external call.

Key insight

Reliable asynchronous design assumes delay, repetition, and interruption, then preserves enough durable state to recover safely.

How to apply this

  • Measure request acknowledgement and end-to-end completion as separate latency goals.
  • Make repeated delivery safe through stable identifiers and durable step state.
  • Bound timeouts, retries, and concurrency, and keep exhausted work visible to operators.
Scenario — The payment provider slows during a promotion

Situation

Cancellation jobs wait on the provider, retries increase, and the queue begins to grow.

Decision

Keep the user-visible state pending, enforce provider timeouts and bounded retry delays, limit concurrency, and alert on queue age before adding workers blindly.

Chapter 5

Slice Delivery and Tests

Break the change into reversible steps and test the riskiest claims at the right boundaries.

Use the technical-excellence practice of making changes small enough to review and reverse. First add the cancellation data model and inactive code paths. Next record and display cancellation state for internal test orders without calling external systems. Then connect the refund step, connect inventory release, and finally expose the customer button behind a server-controlled feature flag. Each slice leaves the system valid and produces evidence for the next decision.

Build the test plan from risks rather than from code structure alone. Unit tests cover eligibility rules and state changes. Database integration tests prove the transaction, uniqueness rule, and concurrent-request behavior. Contract tests cover request and response shapes with the payment and inventory boundaries. End-to-end tests cover acceptance, pending display, completion, authorization, and ineligible orders. Load tests verify the acknowledgement and completion targets at expected peak traffic.

Add focused failure tests for lost responses, duplicate queued work, worker restarts, provider timeouts, exhausted retries, and a partial refund-without-stock-release state. Use provider test environments where possible, but do not mistake them for production behavior. Review test data and log output so payment details and personal data are protected. Assign an owner and pass condition to every test category, and record accepted gaps in the decision log.

Key insight

Small delivery slices and risk-based tests turn design assumptions into evidence while preserving a safe stopping point.

How to apply this

  • Sequence database, internal behavior, dependencies, and user exposure as independently safe slices.
  • Match unit, integration, contract, end-to-end, load, and failure tests to specific risks.
  • Give every test a pass condition, owner, environment, and treatment for sensitive data.
Scenario — The team wants one large release

Situation

Schema, refund calls, stock release, and the customer interface are combined because the feature is useful only when complete.

Decision

Deploy inactive foundations and internal paths first, then enable each external effect separately so failures are easier to locate and exposure can stop safely.

Chapter 6

Roll Out with Observability and Rollback

Increase exposure only when evidence shows the change is useful and safe.

A rollout plan connects each exposure step to checks and a decision owner. Begin with team test orders, then a small group of support agents, then one percent of eligible customers, and increase in planned stages. At every stage compare completion, support contacts, and safety measures with the baseline. Leave enough time to observe delayed refunds and stock corrections before increasing traffic.

Create a journey dashboard rather than watching only server health. Show eligible attempts, accepted requests, final outcomes by state, 95th-percentile acknowledgement and completion time, queue age and depth, retry counts, payment and inventory errors, duplicate-refund protection, and support contacts. Alerts should name an actionable condition, such as oldest queued cancellation exceeding the completion target or any confirmed duplicate refund, and route to the owner with a short response guide.

Rollback is more than redeploying old code. The first action is a tested server-side switch that hides the button and rejects new customer requests while workers safely finish or pause accepted work. Keep backward-compatible schema and reads during the rollout, and do not remove records needed to recover pending cancellations. Define who can stop the rollout, exact triggers, how pending work is inspected, how users and support are informed, and what evidence permits resuming.

Key insight

A safe rollout pairs gradual exposure with journey-level evidence and a rollback that preserves already accepted work.

How to apply this

  • Define stages, observation windows, success checks, stop triggers, and one decision owner.
  • Dashboard the complete user journey and alert only on conditions with a clear response.
  • Separate stopping new requests from safely resolving durable work already in progress.
Scenario — Queue age rises at one-percent exposure

Situation

API latency remains healthy, but accepted cancellations are taking longer than the promised completion time.

Decision

Stop increasing exposure, disable new customer requests if the rollback threshold is crossed, and inspect queue wait, worker throughput, and provider latency while preserving accepted work.

Chapter 7

Write and Present the Final RFC

Turn the reasoning into a document that enables a clear, reviewable decision.

The final request for comments, or RFC, tells one connected story: the current customer problem, measurable outcome, constraints, chosen design, alternatives, delivery, and operation. Keep the end-to-end diagram near the design text and link each major choice to the decision log. Use terms already established in earlier modules, such as transaction, consistency, percentile, queue, retry, feature flag, and rollback, and briefly restate the relevant consequence instead of adding unexplained vocabulary.

Make review efficient by separating facts, assumptions, decisions, and open questions. For each decision, record the options considered, evidence, tradeoff, owner, and date. Ask reviewers by role: product confirms outcome and eligibility, support confirms user and operator flows, security checks access and sensitive data, payment and inventory owners check boundaries, and operations checks dashboards, alerts, and rollback. Resolve blocking comments in the document so the history remains useful.

Present the RFC as a decision, not a tour of every paragraph. Open with the user problem and success measure, walk the request and data-flow diagram, explain the two or three highest-risk choices, then show delivery stages, safety signals, and rollback. End with the decision needed, remaining owners, and next checkpoint. Presentation notes should include likely questions, concise answers, and places where new evidence would change the recommendation.

Key insight

A useful RFC makes the outcome, evidence, tradeoffs, operating plan, and requested decision easy to inspect.

How to apply this

  • Connect outcome, architecture, tests, rollout, and operations in one traceable argument.
  • Record alternatives and tradeoffs so reviewers can challenge the real decision.
  • Present risks and decision points first, with detailed material available for follow-up.
Scenario — Reviewers debate details but make no decision

Situation

A long design document mixes confirmed facts with assumptions and never states what approval is requested.

Decision

Revise the opening to state the outcome, recommendation, key tradeoffs, open questions, reviewers, and exact decision deadline, then use the diagram to guide discussion.

Applied exercise

Build and present a production-ready RFC package

Complete the self-service order-cancellation proposal, or adapt the same journey to a real product change with equal data, dependency, and rollout risk.

  1. 01Write the user problem, business outcome, scope, non-goals, assumptions, success measures, safety measures, and named decision owner.
  2. 02Draw an end-to-end diagram showing client and API requests, authorization, durable writes, transaction boundaries, queued work, payment and inventory calls, identifiers, and failure paths.
  3. 03Create a decision log for the data model, invariants, consistency, asynchronous processing, performance targets, alternatives, tradeoffs, evidence, owners, and unresolved questions.
  4. 04Write a risk-based test plan covering unit, database integration, contract, end-to-end, concurrency, load, retry, duplicate-work, timeout, restart, and partial-failure cases, each with a pass condition and owner.
  5. 05Define delivery slices and a staged rollout plan with audiences, exposure levels, observation windows, success checks, stop triggers, approvers, and communication to support and affected users.
  6. 06Specify journey dashboards and actionable alerts for outcomes, latency percentiles, queue health, retries, dependency failures, duplicate protection, stock correction, and support contacts.
  7. 07Write and rehearse a rollback plan that stops new requests, handles accepted and partially completed work, preserves compatible data, names operators, and defines verification and recovery steps.
  8. 08Prepare presentation notes that frame the decision, explain the diagram and highest risks, answer likely reviewer questions, state what evidence would change the recommendation, and end with owners and next actions.

Deliverable

A practical RFC package containing the final RFC, an end-to-end diagram, decision log, detailed test plan, staged rollout plan, dashboard and alert specification, tested rollback procedure, and concise presentation notes.

Knowledge check

Check your understanding before finishing

Answer from the reasoning and trade-offs, not keyword memory. After submitting, review the explanation for every question.

Q1What is the strongest starting point for the self-service cancellation RFC?
Q2Why should the payment-provider call remain outside the local database transaction?
Q3Which test most directly addresses two devices cancelling the same order at once?
Q4During a small rollout, API latency is healthy but the oldest cancellation job exceeds the completion target. What should the team do?
Q5What makes the final RFC ready for a decision?