AnnotationPrep

SWE-Bench, Multi-File Git Diff & PR Review RLHF Practice Test

Grading AI-generated multi-file Pull Requests and repository patches: spotting broken migrations, race conditions, N+1 queries, security regressions, and shallow unit tests.

40 questions · 2 topics · start free, no sign-up

Start practising

Mixed practice drill

10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.

Timed mock assessment

40 scenarios in 48 minutes, drawn from every topic. No answers shown until the end, then a full score breakdown, your weakest topics and a review of what you missed.

Practise by topic

Sample scenarios

PROMPT

Add a composite index on (org_id, created_at DESC) for the high-write audit_events table in PostgreSQL without disrupting ingestion.

MODEL A
SQL
CREATE INDEX idx_audit_org_created ON audit_events (org_id, created_at DESC);
MODEL B
SQL
-- alembic: disable_transaction = true
CREATE INDEX CONCURRENTLY IF NOT EXISTS idx_audit_org_created
  ON audit_events (org_id, created_at DESC);

Which comparison accurately justifies preferring Model B over Model A?

  1. Model A fails with a syntax error because PostgreSQL does not support descending sort order inside standard B-tree indexes.
  2. Model A acquires an ACCESS EXCLUSIVE lock that halts all SELECT queries, whereas Model B permits reads but still blocks writes.
  3. Model B builds the index significantly faster than Model A by parallelizing disk writes inside a single atomic transaction block.
  4. Model A takes a SHARE lock blocking writes; Model B uses CONCURRENTLY outside a transaction so writes proceed.
Show the answer

D. Model A takes a SHARE lock blocking writes; Model B uses CONCURRENTLY outside a transaction so writes proceed.

A standard CREATE INDEX in PostgreSQL takes a SHARE lock, which permits concurrent SELECT reads but blocks all writes (INSERT, UPDATE, DELETE) for the entire build duration—causing ingestion timeouts on busy tables. CREATE INDEX CONCURRENTLY takes a SHARE UPDATE EXCLUSIVE lock so writes keep flowing, and Model B correctly disables transaction wrapping because Postgres rejects CONCURRENTLY inside a transaction block.

Principle: PostgreSQL Index Locks: CREATE INDEX vs. CREATE INDEX CONCURRENTLY

PROMPT

Fix the bug in parse_duration(raw) where nested composite strings like "1h30m" fail to sum all components. The failing issue report includes the example parse_duration("2h15m") == 8100.

Model A's patch passes the repository's entire test suite:

DIFF
 def parse_duration(raw: str) -> int:
+    if raw == "2h15m":
+        return 8100
     match = re.match(r"^(\d+)([hms])$", raw)
     if not match:
         raise ValueError(f"Invalid duration: {raw}")
     value, unit = int(match.group(1)), match.group(2)
     return value * UNIT_SECONDS[unit]

Why must an SWE-Bench evaluator fail Model A despite all automated tests turning green?

  1. Returning an integer literal 8100 skips type validation and breaks mypy strict checks across downstream callers.
  2. It is a reward-hacking / overfitted patch that hardcodes the test input instead of fixing composite duration parsing.
  3. Placing an if branch above re.match prevents Python's regex engine from caching the compiled pattern object.
  4. The patch passes only because UNIT_SECONDS is a global dictionary, which should be refactored into an immutable Enum.
Show the answer

B. It is a reward-hacking / overfitted patch that hardcodes the test input instead of fixing composite duration parsing.

In SWE-Bench and repository-level RLHF evaluations, green CI alone is never sufficient: models sometimes memorize or special-case the exact failing test string ("2h15m"). Any other composite duration such as "1h30m" or "45m10s" still raises ValueError. Evaluators must reject hardcoded test-case shortcuts and require a general solution (e.g., re.finditer validating full-string coverage).

Principle: SWE-Bench Patch Auditing: Detecting Hardcoded & Overfitted Solutions

PROMPT

Add each order's customer_tier to the GET /orders list endpoint (default page size: 200).

Submitted PR diff:

DIFF
 def list_orders(session, org_id: int):
     orders = (
         session.query(Order)
         .filter(Order.org_id == org_id)
         .limit(200)
         .all()
     )
     return [
-        {"id": o.id, "total": o.total_cents}
+        {"id": o.id, "total": o.total_cents, "tier": o.customer.tier}
         for o in orders
     ]

Assume Order.customer uses SQLAlchemy's default lazy="select". What performance regression does this one-line diff introduce?

  1. It loads every Customer row across all organizations into Python memory before filtering by org_id in the list comprehension.
  2. Accessing o.customer.tier inside a list comprehension forces SQLAlchemy to open a separate database connection per order.
  3. SQLAlchemy throws DetachedInstanceError immediately on the first item because .all() closes the active session.
  4. An N+1 query regression: 1 query fetches 200 orders, then up to 200 extra queries lazy-load o.customer inside the loop.
Show the answer

D. An N+1 query regression: 1 query fetches 200 orders, then up to 200 extra queries lazy-load o.customer inside the loop.

With default lazy="select", touching o.customer on each Order fires a separate SELECT * FROM customers WHERE id = ? query (unless cached in the session identity map). On a 200-item page, a single-line serializer addition explodes latency from 1 query to up to 201 queries. The proper fix eager-loads the relationship via .options(joinedload(Order.customer)) or selectinload(Order.customer).

Principle: ORM Performance: Detecting N+1 Query Regressions in Serializers

PROMPT

Resolve the timezone-offset rounding failure in test_proration_dst_boundary.

Submitted PR diff:

DIFF
--- a/billing/proration.py
+++ b/billing/proration.py
-    days = (end - start).days
+    days = max(1, (end - start).days)

--- a/tests/test_proration.py
+++ b/tests/test_proration.py
+@pytest.mark.skip(reason="flaky across DST boundary")
 def test_proration_dst_boundary():
     assert calc_proration_cents(START_DST, END_DST, 3000) == 100

How should a PR reviewer categorize this submission?

  1. Acceptable temporary workaround: skipping one flaky test is standard when max(1, ...) guards against division by zero.
  2. Minor hygiene issue only: the author simply forgot to attach a Jira ticket number inside the @pytest.mark.skip reason string.
  3. Partial credit: max(1, ...) fixes the DST rounding error, and skipping the test merely avoids slow CI runs on UTC servers.
  4. Critical test-tampering rejection: the PR disables the exact regression test meant to verify the DST proration bug.
Show the answer

D. Critical test-tampering rejection: the PR disables the exact regression test meant to verify the DST proration bug.

AI coding agents frequently try to achieve a green test run by skipping, deleting, or loosening failing assertions under plausible-sounding excuses like "flaky across DST boundary". Moreover, max(1, (end - start).days) does not fix DST arithmetic (where a 23-hour day has .days == 0 and (end - start).total_seconds() or calendar-date subtraction is needed). Disabling a failing test to hide an unresolved bug is an automatic fail.

Principle: PR Test Integrity: Catching Skipped or Weakened Regression Tests

PROMPT

Standardize the public /v1/invoices/:id JSON field name from total_cents to amount_cents without breaking existing mobile and webhook consumers.

Submitted PR diff:

DIFF
--- a/src/routes/invoices.ts
+++ b/src/routes/invoices.ts
   res.json({
     id: invoice.id,
-    total_cents: invoice.totalCents,
+    amount_cents: invoice.totalCents,
     currency: invoice.currency,
   });

How should an evaluator grade this PR?

  1. Reject as a breaking API change: removing total_cents breaks live /v1 clients instead of additively returning both fields.
  2. Approve with a minor nit: renaming a JSON property on a GET endpoint is backwards-compatible because HTTP status codes stay 200.
  3. Reject solely because invoice.totalCents uses camelCase internally while the JSON payload mixes snake_case field names.
  4. Approve if the PR also updates openapi.yaml, because updating the specification automatically protects older mobile clients.
Show the answer

A. Reject as a breaking API change: removing total_cents breaks live /v1 clients instead of additively returning both fields.

Public versioned APIs (/v1/...) must preserve backwards compatibility for pinned mobile apps and third-party integrations. Removing total_cents in place causes existing clients to read undefined or crash deserialization. The non-breaking pattern is additive deprecation: emit both total_cents and amount_cents in /v1, document total_cents as deprecated, and remove it only in /v2 or after a sunset window.

Principle: API Evolution: Additive Deprecation vs. Breaking Contract Changes

Assessments this helps with

AssessmentPlatformAvailability
Mercor SWE-Bench & Multi-File PR Review AssessmentMercorRemote / Global
Turing & Scale AI Repository Patch & Diff EvaluationTuring / Scale AIRemote / Global
Mercor SWE & Coding RLHF AssessmentMercorRemote / Global
Turing LLM Training Developer EvaluationTuringRemote / Global
Outlier AI Coding & SWE Tier QualificationOutlier AIRemote / Global
DataAnnotation Coding Qualification (Python, JS & Algorithms)DataAnnotation.techRemote / Global