AnnotationPrep

Coding Assessment Practice: Debugging LLM Code & Algorithm Verification

Finding subtle logic bugs, off-by-one errors, security flaws and missed edge cases in Python, JavaScript and SQL code generated by AI models.

30 questions · 3 topics · start free, no sign-up

Start practising

Mixed practice drill

10 scenarios, graded as you go, with the exact grading principle for every answer. Start free, no sign-up.

Timed mock assessment

30 scenarios in 36 minutes, drawn from every topic. No answers shown until the end, then a full score breakdown, your weakest topics and a review of what you missed.

Practise by topic

Sample scenarios

An LLM writes a JavaScript function to sort numbers:

```js
function sortNums(arr) {
return arr.sort();
}
```

What happens when you call `sortNums([10, 2, 1])`?

  1. Returns `[1, 2, 10]`.
  2. Returns `[1, 10, 2]` because `Array.prototype.sort()` converts elements to strings and sorts lexicographically by default, AND mutates `arr` in-place.
  3. Throws a `TypeError` because numbers need a comparator.
  4. Returns `[10, 2, 1]` unchanged.
Show the answer

B. Returns `[1, 10, 2]` because `Array.prototype.sort()` converts elements to strings and sorts lexicographically by default, AND mutates `arr` in-place.

Without a comparator `(a, b) => a - b`, JS `.sort()` sorts by UTF-16 code units (`'10' < '2'`). It also mutates the input array in-place; use `[...arr].sort((a, b) => a - b)`.

Principle: JavaScript Array.prototype.sort Default Behavior

An LLM writes a Python helper to accumulate items per user:

```python
def add_tag(tag, tags=[]):
tags.append(tag)
return tags
```

Why must you flag this code in a coding RLHF evaluation?

  1. `list.append()` returns `None` instead of the list.
  2. Mutable default argument bug: the same `tags` list is shared across all calls that omit `tags`.
  3. Python does not allow two parameters with similar names.
  4. `tags` must be a tuple to use `.append()`.
Show the answer

B. Mutable default argument bug: the same `tags` list is shared across all calls that omit `tags`.

Default parameter values in Python are evaluated once at function definition time. Using `tags=[]` causes state to leak between independent calls. The golden fix uses `tags=None` and initializes `if tags is None: tags = []`.

Principle: Python Best Practices: Mutable Default Arguments

An LLM writes a SQL query to find all users who have NEVER placed an order:

```sql
SELECT u.id, u.email
FROM users u
LEFT JOIN orders o ON u.id = o.user_id
WHERE o.status = 'paid' AND o.id IS NULL;
```

Why does this return 0 rows?

  1. `LEFT JOIN` does not allow `IS NULL`.
  2. Putting `o.status = 'paid'` in the `WHERE` clause turns the `LEFT JOIN` into an inner join—any unmatched row has `o.status = NULL`, which fails `o.status = 'paid'`.
  3. `WHERE` must come before `FROM`.
  4. `o.id` can never be `NULL`.
Show the answer

B. Putting `o.status = 'paid'` in the `WHERE` clause turns the `LEFT JOIN` into an inner join—any unmatched row has `o.status = NULL`, which fails `o.status = 'paid'`.

Filter conditions on the right-hand table of a `LEFT JOIN` must go in the `ON` clause (`AND o.status = 'paid'`), not in `WHERE`, otherwise unmatched rows (where `o.id IS NULL`) are filtered out.

Principle: SQL Joins: LEFT JOIN Filtering in WHERE vs ON

An LLM writes an async loop:

```js
async function fetchAll(ids) {
const out = [];
ids.forEach(async (id) => {
const data = await fetchItem(id);
out.push(data);
});
return out;
}
```

What does `await fetchAll([1, 2, 3])` return?

  1. An array of 3 fetched items in order.
  2. `[]` (an empty array) because `Array.prototype.forEach` does not await async callbacks.
  3. A `Promise` that rejects.
  4. `[undefined, undefined, undefined]`.
Show the answer

B. `[]` (an empty array) because `Array.prototype.forEach` does not await async callbacks.

`forEach` ignores the Promises returned by the async callback and finishes synchronously, so `return out` runs before any `fetchItem` resolves. Use `Promise.all(ids.map(fetchItem))` or a `for...of` loop.

Principle: JavaScript Async: forEach Does Not Await

An LLM attempts to remove all even numbers from a list in-place:

```python
nums = [2, 4, 5, 6, 8, 9]
for n in nums:
if n % 2 == 0:
nums.remove(n)
print(nums)
```

What does this print, and what is the bug?

  1. Prints `[5, 9]`; no bug.
  2. Prints `[4, 5, 8, 9]`; mutating a list while iterating over it skips elements as indices shift.
  3. Raises `RuntimeError: list changed size during iteration`.
  4. Prints `[2, 4, 6, 8]`; `%` checks for odd numbers.
Show the answer

B. Prints `[4, 5, 8, 9]`; mutating a list while iterating over it skips elements as indices shift.

Unlike dictionaries/sets, mutating a Python `list` during a `for` loop does not raise `RuntimeError`—instead, removing `2` shifts `4` to index 0, so the iterator advances to index 1 (`5`) and skips `4` (and later skips `8`). Use a list comprehension: `[n for n in nums if n % 2 != 0]`.

Principle: Python Iteration: Mutating Lists In-Place

Assessments this helps with

AssessmentPlatformAvailability
DataAnnotation Coding Qualification (Python, JS & Algorithms) readyDataAnnotation.techRemote / Global
Outlier AI Coding & SWE Tier Qualification readyOutlier AIRemote / Global
Mercor SWE & Coding RLHF Assessment readyMercorRemote / Global
Alignerr Python & TypeScript Coding Assessment readyAlignerr (Labelbox)Remote / Global
Turing LLM Training Developer Evaluation readyTuringRemote / Global
micro1 AI Coding & Technical Screening Test readymicro1Remote / Global