A small infrastructure update: the API has been rewritten. It's about 2× faster on every measurement we ran, handles many more simultaneous requests gracefully, and fixes a notable bug, but otherwise behaves identically to before.

If you use the website or API directly for scoring: change nothing. Same URL, same parameters, same responses. But if you see new issues that you weren't getting before, email peter.organisciak@du.edu.

I also added an updated API reference page with sample code:

What changed

The Open Scoring site and scoring API have lived in a separate Python/Flask serviced for years. A few months ago the site was rewritten in Javascript, to run more flexibly and handle concurrency better. Now, the API has also been ported to that codebase. In the process, I also found a bug where scoring two uncached responses concurrently would give an undiagnosed API error - this may solve some issues I've heard about but haven't been able to reproduce.

API Scoring Speed Tip

If scoring many items with the API (old or new), try to score in batch or keep the connection open. The first connection to our server adds latency, so if every response you score is a brand new call to our server, it slows everything down, particularly if you're further away from Denver. For example, if you're using Python to connect to the API, you can keep the session alive with requests.Session(), so your computer only does the handshake with our server one time.

Footnote

The numbers! These were measured on my laptop, the relative ratios should hold on the server.

Single-request latency (One request, one row to score, no caching.)

Cache hits (Most calls in practice hit the cache because the same prompt/response/model/language tuple has been scored before.)

Concurrent load

The old API at 5 concurrent clients about 3× slower and would degrade with more connections.