Calibrate classifier probabilities

hardMLacceptance —~35 min

Implement calibration_error(probs, labels, bins) — the expected calibration error of a binary classifier. Split [0, 1] into bins equal-width bins and assign each probability to bin min(int(p·bins), bins - 1) (so p = 1.0 lands in the last bin). For every non-empty bin add (count / n)·|mean probability - fraction of positive labels|.

The consensus model in ai quants reports calibrated probabilities because a confident wrong number is worse than an honest vague one. Return the error rounded to 4 decimal places.

example 1

input

probs=[0.1, 0.4, 0.35, 0.8, 0.9, 0.6], labels=[0, 0, 1, 1, 1, 0], bins=5

output

0.3417

example 2

input

probs=[0.1, 0.5, 1.0, 0.95, 0.3], labels=[0, 1, 1, 1, 0], bins=5

output

0.19

note

A probability of exactly 1.0 is in the data — where does it bin?

constraints

1 ≤ len(probs) == len(labels) ≤ 10_000
0 ≤ p ≤ 1
labels ∈ {0, 1}
2 ≤ bins ≤ 20

teacher · 2 hints · the first is free

↗ read the theory

the chapter this problem is cut from

▶ open in the workbench

run the same math on a real tape

●solution.py· utf-8 · spaces:4

def calibration_error(probs: list, labels: list, bins: int) -> float:
    """Expected calibration error over equal-width bins, rounded to 4 dp."""
    n = len(probs)
    sums = [0.0] * bins
    pos = [0] * bins
    cnt = [0] * bins
    for p, y in zip(probs, labels):
        # TODO: p == 1.0 maps to index `bins`, one past the end — clamp it
        b = int(p * bins)
        sums[b] += p
        pos[b] += y
        cnt[b] += 1
    err = 0.0
    for b in range(bins):
        if cnt[b]:
            err += cnt[b] / n * abs(sums[b] / cnt[b] - pos[b] / cnt[b])
    return round(err, 4)

ln 1, col 1tab indent · esc leave editor

python 3.13 · pyodide

+ 6 hidden on submit

input · keyword arguments to calibration_error()

probs=[0.1, 0.4, 0.35, 0.8, 0.9, 0.6], labels=[0, 0, 1, 1, 1, 0], bins=5

expected

0.3417

the judge calls calibration_error() with exactly these inputs · outputs are compared to 6 decimal places

● standing bypython 3.13 in your browser · pyodide 0.28.3 · judged on the server to 6 dp⌘↵ run · ⌘⇧↵ submit

paper only · educational · not advice