Phone Number Generator: Do Your Tests Catch Broken Fixtures?

Check the assertion, not just the generated input

A generator can return hundreds of phone-shaped values while its tests check only that a string is present. That leaves the expected country, exact output and safety fields untested. Changing the input distribution will not repair an assertion that ignores those fields.

In source-code mutation testing, a tool changes implementation code and reruns the tests. A failing test detects a change; a passing suite leaves it surviving. The small experiment below uses hand-selected output changes instead. It is a quick check of the test oracle, not a substitute for a tool such as Stryker or a report about this site's production test coverage.

An oracle is the rule that decides whether a result is correct. For one deterministic display fixture, keep an independently reviewed expected record. Do not obtain the expected value by calling the generator you are testing: the same defect could then produce both the actual and expected values.

Our five-change experiment

We fixed the baseline at +12025550100, region US, a source URL, fictional_only: true and delivery_allowed: false. Its 555-0100 line number is inside the limited NANPA fictional reservation. Both the baseline and the alternate 555-0101 slot below are display fixtures; neither is a live messaging destination.

The weak check tests only length and region. The exact contract check compares this particular deterministic record with its reviewed expected record. We ran the downloadable Python script locally on October 7, 2026 with Python 3.13.15; the baseline passed both checks. Then we applied one change per independent copy:

These are five selected cases, not a statistical sample. The three survivors demonstrate missing assertions in this weak check. Detecting all five with the exact contract proves sensitivity to those five changes only; it does not prove every implementation defect is covered.

Reproduce the result without an SMS provider

Save phone-fixture-mutation-check.py and run python phone-fixture-mutation-check.py. It uses only Python's standard library, contains no network client and sends no SMS. Run without Python's -O option so its baseline assertions stay enabled.

"""Offline, hand-selected output mutations; no SMS or network code."""
from copy import deepcopy
import json

EXPECTED = {
    "phone": "+12025550100",
    "region": "US",
    "source": "https://www.nanpa.com/numbering/555-line-numbers",
    "fictional_only": True,
    "delivery_allowed": False,
}

MUTATIONS = {
    "remove_plus": {"phone": "12025550100"},
    "wrong_region": {"region": "GB"},
    "empty_source": {"source": ""},
    "enable_delivery": {"delivery_allowed": True},
    "change_fixed_slot": {"phone": "+12025550101"},
}

def weak_check(row):
    return len(row.get("phone", "")) == 12 and row.get("region") == "US"

def contract_check(row):
    # Independent, reviewed expected values, not the generator under test.
    return row == EXPECTED

def classify(check, row):
    try:
        return "SURVIVED" if check(row) else "KILLED"
    except Exception as error:
        return "HARNESS_ERROR:" + type(error).__name__

def main():
    assert weak_check(EXPECTED) and contract_check(EXPECTED)
    results = []
    for name, change in MUTATIONS.items():
        row = deepcopy(EXPECTED)
        row.update(change)
        results.append({"mutation": name, "weak": classify(weak_check, row),
                        "contract": classify(contract_check, row)})
    print("BASELINE: PASS")
    print(json.dumps(results, indent=2))
    assert all(row["contract"] == "KILLED" for row in results)
    assert sum(row["weak"] == "SURVIVED" for row in results) == 3

if __name__ == "__main__":
    main()

The output begins BASELINE: PASS. The JSON contains five rows: two weak-check kills, three weak-check survivors and five contract-check kills. An unexpected exception is reported as HARNESS_ERROR; do not present a broken runner as evidence that the intended assertion detected a defect.

The script changes data in memory and returns the original reviewed baseline to each case. It does not alter a production dataset, parser or numbering policy. Exact equality is intentional for this one fixed fixture. For randomized output, assert the reviewed invariants of the chosen pool rather than requiring every generated result to equal this one number.

Turn a survivor into a useful regression

1. Identify the changed field and the contract it should satisfy. “The result looks like a phone” is too broad for a missing source or a changed delivery flag.
2. Add an assertion from an independent expected rule. Pin exact values for deterministic fixtures and policy fields; document whether optional presentation fields may vary.
3. Run the unchanged implementation first. A baseline failure means the experiment cannot yet distinguish a regression from existing breakage.
4. Reapply the one controlled change and confirm the intended assertion fails. Record the case name and failing assertion, not just a red build.
5. Restore the baseline and keep the new assertion in ordinary regression tests. Run broader source-code mutation analysis separately when its cost is justified.

Keep transport disabled even when you intentionally flip delivery_allowed in a test record. The mutation runner must not call a messaging adapter at all. A boolean guard being tested is not the only barrier between a fixture and a real delivery system.

Stryker's report states distinguish failed tests, survivors, missing coverage and runner or compile errors. When you use that tool, retain its reported categories rather than relabeling them to match this simplified data experiment. Review an apparently equivalent change before adding an assertion solely to raise a score.

Questions developers ask

Does a phone number generator need more random values to fix a surviving change?

Sometimes new inputs matter, but not when the check ignores the changed field. In this experiment every value is fixed: adding the source and delivery-policy assertions detects defects that length and region cannot see.

Is changing a fixture record the same as full mutation testing?

No. This experiment checks the sensitivity of a few assertions to selected output changes. Full source-code mutation testing modifies implementation behavior and classifies results across a configured test suite. Do not publish this five-case demonstration as the generator's mutation score.

Should these checks run on every commit?

Keep the small deterministic contract regressions with ordinary unit tests. Schedule broader mutation analysis according to the changed code and runtime budget. A public software-testing discussion raises the cost of running mutation analysis in a pipeline; it is a question signal, not a benchmark or a rule for every team.

Can a deliberately changed test number receive a real verification code?

This example never attempts delivery. Its known phone values stay within the documented fictional block, and the runner has no messaging code. If you mutate arbitrary digits in your own tests, do not assume the resulting number is unassigned or safe to contact.

Sources and next checks

• Stryker: what mutation testing measures
• Stryker: mutant states and metrics
• NANPA: the limited fictional 555 line-number block
• Original reproducible experiment — baseline, changes and check definitions.
• Replay a randomized parser failure when the input, rather than the assertion, needs investigation.
• Phone number testing checklist for the broader release matrix.