Turning our OpenAPI schema into a bug finder

Buttondown already had good API tests. Those are mostly example based tests. We covered happy paths, permissions, validation errors, creates, updates, snapshots, and regressions from production bugs.

What we did not have was a systematic way to test requests we had never thought to write. For example, one generated request contained this publication date

{"publish_date": "0422-09-03T16:55:45.947974-16:55" }

The value made it through request parsing, reached Postgres, and turned bad client input into a 500 response. The API should have rejected it at the request boundary.

None of our example-based tests contained a value like it. A generated test did. To find cases like this systematically, we introduced property-based API testing with Schemathesis.

APIs do not only receive the requests we remember to write. They receive missing fields, strange dates, empty strings, large numbers, old clients, future clients, and integrations that read the docs differently than we intended.

Moving to property-based testing

Property-based tests approach the problem from the other direction. Instead of specifying every input, we state an invariant and let a testing engine search generated inputs for a counterexample.

Schemathesis applies this approach to HTTP APIs. It reads an OpenAPI document and generates requests from the operations, parameters, and schemas it finds there. It runs on top of Hypothesis, Python’s property-based testing engine.

Buttondown’s REST API is built with Django Ninja. We use Pydantic schemas to define request and response shapes, and Django Ninja uses those schemas to generate our OpenAPI document.

That document already supported several parts of Buttondown:

flowchart LR
    Schemas["Django Ninja routes
Pydantic schemas"] -->|generate| OpenAPI["openapi.json"]
    OpenAPI --> Docs["API documentation"]
    OpenAPI --> Types["TypeScript types"]
    OpenAPI --> SDKs["SDKs & integrations"]
    OpenAPI --> Tests["Schemathesis tests"]

Example-based tests show that an endpoint handles the scenarios we know to test. Schemathesis probes the scenarios we did not think to write down.

We started with one rule client-controlled input must not produce a 500.

That matters more as APIs become interfaces for humans, integrations, internal tools, automation, and agents. Agents can make POST and PATCH requests too. If they send a wrong field, the API should reject it cleanly, not fall through to a server error.

Schemathesis gave us a practical way to ask that question at scale. It reads OpenAPI, generates requests, sends them to the app, validates responses, and lets Hypothesis shrink failures into smaller reproductions.

sequenceDiagram
    participant OpenAPI as openapi.json
    participant Schemathesis as Schemathesis
    participant Django as Django application

    OpenAPI->>Schemathesis: Generate test cases
    Schemathesis->>Django: Request
    Django-->>Schemathesis: Response
    Schemathesis->>Schemathesis: Validate response and reject 5xx failures

Schemathesis does not know our business logic. That's what makes it so useful. It exposes three useful mismatches:

1. Requests the schema allows but the implementation rejects
2. Inputs that produce a 500 error
3. Responses that don't match the scheme

We started by connecting two things we already had: the generated OpenAPI document and the Django app.

SCHEMA_PATH = Path(__file__).resolve().parents[2] / "assets" / "autogen" / "openapi.json"

with open(SCHEMA_PATH) as f:
    raw = json.load(f)

schema = from_dict(schema=raw, config=config)
schema.app = get_wsgi_application()

The OpenAPI document tells Schemathesis what requests to generate. The Django application gives Schemathesis an in-process interface to send those requests to. That lets the tests exercise the real application without starting a server or making network requests.

Without auth, Schemathesis mostly proves that protected endpoints return 401. That is correct, but not useful. We wanted generated requests to reach the route code behind the auth layer.

So our auth hook injects the same header a real API client sends.

@schemathesis.auth(refresh_interval=None)
class TokenAuth:
    def get(self, case, context):
        return active_api_key["api_key"]

    def set(self, case, data, context):
        case.headers = case.headers or {}
        case.headers["Authorization"] = f"Token {data}"
        case.headers["Accept-Encoding"] = "identity"

After that, most tests were small. For collection routes, Schemathesis can generate a request, call the app, and validate the response against the OpenAPI schema.

@schema.include(path="/attachments").parametrize()
def test_attachments_collection(case, newsletter, mock_s3_storage):
    response = case.call()
    case.validate_response(response)

We did not add fuzz testing all at once.

We started with subscriber tag routes, fixed the schema gaps they exposed, regenerated OpenAPI, and repeated the loop. Once that felt stable, we added more routes.

That rollout mattered. Fuzz testing can be noisy if you try to cover everything at once.

Smoke mode is fast enough for regular development. Deep mode runs more examples when API contracts change or we want broader coverage.

Fuzz testing changed how we think about API request schemas.

A field is no longer just str. We ask what strings are actually valid. Sometimes that means adding a pattern, a length bound, or a stricter type.

The same applies to IDs. If an endpoint accepts an ID, we define which format it accepts: UUID, TypeID, or both. read more about our TypeID migration.

Errors are part of the contract too. If a route can return an error, the OpenAPI schema should describe its shape and HTTP status code.

Fuzz tests also exposed edge cases that example-based tests missed: invalid inputs that return 500s, similar filters behaving differently, responses drifting from the schema, and unusual values reaching code paths that were not designed for them.

Going back to our first example of the email publish dates:

{"publish_date": "0422-09-03T16:55:45.947974-16:55"}

The fix separated malformed-date validation from the product rule that allows intentional backdating. Now an invalid value wouldn't hit Postgres and cause a 500.

Another example was metadata. Normal metadata worked, but some valid-looking values were unsafe for our stack: null bytes, surrogate strings, and integers outside signed 64-bit bounds. Tightening validation prevented serialization and database errors from surfacing as 500s.

The lesson was not “metadata was broken” or “publish dates were broken.” Normal examples had hidden the rough edges. Fuzz testing made those edges visible.

With Schemathesis, our OpenAPI schema now helps test our API. A field is not just a type annotation. An error response is not just an implementation detail. The OpenAPI spec is not just documentation.

It is a promise about how our API should behave. Schemathesis gives us a practical way to test that promise under pressure.