How we check every link in your email

One of the scariest parts about hitting "send" on an email is the feeling that there's no takesies-backsies. As soon as it starts hitting people's inboxes, you are left without a mechanism to fix any mistakes. A broken link in a blog post is a minor embarrassment; a broken link in an email to ten thousand subscribers is a small catastrophe.

This is why Buttondown checks every link in your email before you send it, and why the machinery behind that check is more involved than you might expect.

Our link checking infrastructure makes the most sense viewed through the lens of its development: just like in so many things, we started naively and grew increasingly robust over time.

1. First, the simplest version: extract every URL from the email body, make an HTTP request, see if you get a 200 back.
2. Then you discover some sites aggressively block automated requests (LinkedIn, Amazon, Codepen, the list is literally infinite). We maintain a denylist of domains we know will lie to us, and we spoof a browser User-Agent for everything else.
3. Then you realize you're sending requests for obviously malformed URLs, so you move some of the checking logic clientside to do a better job filtering out bad inputs.
4. We start with a HEAD request to politely and performantly check things, but many servers return 404s and 405s for HEADs even though a GET for the same URL returns a 200 — so we added fallback logic and a mild backoff.
5. Then we added aggressive timeouts for servers that chew up the clock.
6. Then, we realized that for good and bad senders alike we needed to check outbound links against a handful of external services (Google Web Risk, SURBL, Spamhaus, etc.) We don't want to block the actual client-side link checking on this, but we do want to block sending. So we introduced a standalone link model that can hold onto some state and ensure that no emails go out without us verifying it won't trigger a flag in some database we care about.
7. Finally, we cache the hell out of it. Many of our authors send emails with dozens of links, and it's important to keep the performance loop tight even for them. So we cache both client-side with slightly varying logic depending on prior performance of both the author and the link itself, as well as server-side, which lets us dedupe link checking across newsletters.

All in all, it looks something like this:

---
config:
  layout: elk
  elk:
    mergeEdges: true
    nodePlacementStrategy: LINEAR_SEGMENTS
---

flowchart-elk
    A["Extract URLs from email body"] --> B{"Malformed?"}

    B -- "Yes" --> WARN
    B -- "No" --> D{"Cached?"}

    D -- "Hit (safe)" --> SAFE
    D -- "Hit (warn)" --> WARN
    D -- "Hit (block)" --> BLOCK
    D -- "Miss" --> F["HEAD https://foo.com"]

    F --> G{"2xx?"}
    G -- "Yes" --> L
    G -- "No (404/405)" --> I["GET https://foo.com"]
    I --> J{"2xx?"}
    J -- "Yes" --> L
    J -- "No / Timeout" --> WARN



    L["Check external services (Google Web Risk, SURBL, Spamhaus)"] --> M{"Flagged?"}
    M -- "No" --> SAFE
    M -- "Yes" --> BLOCK

    SAFE["Safe to send"] --> P
    WARN["Broken link warning"] --> P
    BLOCK["Block send"] --> P

    P[("Cache result (client + server)")]

    subgraph Client-Side
        A
        B
        D
    end

    subgraph Server-Side
        F
        G
        I
        J
        L
        M
    end

    classDef reject fill:#dc2626,stroke:#991b1b,color:#fff
    classDef success fill:#16a34a,stroke:#166534,color:#fff
    classDef warn fill:#ca8a04,stroke:#854d0e,color:#fff
    classDef process fill:#2563eb,stroke:#1e40af,color:#fff
    classDef cache fill:#9333ea,stroke:#6b21a8,color:#fff

    class BLOCK reject
    class SAFE success
    class WARN warn
    class F,I,L process
    class P cache