Interactive drawing assessments; Ocsai is back

Two updates to Ocsai over the past week. First: we're back up! Second: we've added figural originality scoring. I want to cover those in reverse.

Ocsai Drawings

I added Ocsai-D to Open Creativity Scoring, an interactive tool for figural creativity scoring. This is the work presented in the Acar, Organisciak, and Dumas pre-print (2024). It culminates a good deal of effort over two years, so we're excited to share it with others!

The new page is intended for play: give it a try! While you can upload images to score, drawing and iterating in the browser a) is fun, and b) helps internalize the various strengths and limitations of figural scoring.

Details. In the article, we evaluate various scoring options (CLIP-based feature classifiers, vision transformer classifiers, and semantic distance) across datasets from two tests: TTCT-Figural, as well as the MTCI data reported on in Patterson et al. (2023). Both tests ask you to complete/embellish an abstract doodle prompt. The specific model posted on OCS is trained on the latter data, using a specific architecture of vision transformers called BEiT. It is an 'applied' version of the models reported in the paper, trained with more data (whereas for evaluation more data is held out for evaluation). The evaluated models achieved a correlation with human judges of r=.85. The models themselves are posted to Hugging Face for research use.

Vision Transformers, like BEiT, work by splitting up an image into patches. These patches are each given a token identifier, and each is treated like a word would be in large language model training. Training is then done very similarly to language models, where the architecture masks random patches during training, with an objective of learning to understand the corrupted image patches from context of the rest of the image. Here is a simplified example:

Ocsai Back Up

Ocsai was down last week, primarily due to a jolt in usage that punctuated two things: 1) we're not protected against over-use by malicious agents, like bots that crawl the web and send requests to any server that will take them, and 2) we don't communicate any expected scale of use.

Both of these were corollaries to intentional design choices: wanting to make Open Creativity Scoring valuable and easily usable for as many people as possible. However, short-term access needs to be balanced out with long-term access. The way I struck this balance was implementing rate-limits, with optional API keys to raise the limit if you hope to score more.

Currently the limits are 1500 items per hour, 10k/day, and 30k a week - but I expect to adjust with that as I get a sense of use. A key approximately doubles those limits, though can be adjusted per-key. The limits are only on Ocsai, not semantic scoring, which runs on CPU and only has trivial costs associated with it.

For most people, the site and API will work as always, easily accessible in the browser. If basic bots find the server, they'll be capped in how many resources they can naively use up. Finally, the existence of rate-limits will hopefully communicate the expected scales of use, in a way that we previously didn't. We designed the system to score thousands or tens of thousands of responses; hundreds of thousands or millions would need a conversation on better options.

On privacy: nothing is tracked other than a ticker for how many items you've scored. Nothing else is saved (e.g. what items you score).

Rquesting / Using an API Key

If you're a scholar, student, or practitioner looking to use Ocsai more than the default rate limits allow, send me an email (peter.organisciak@du.edu) and I can make you a key. Remember that it's not necessary for most people - you can score thousands of responses without it.

How do you use the key?

In the Open Creativity Scoring site, there's now a 'key' field. Paste it there!

If using the API, you need to add the following to the header of your request: X-API-KEY: your_api_key. That will differ from language to language, but is generally a simple process. Here's an example with the Python requests library:

from requests import post
headers = {"X-API-KEY": your_api_key}
params = { ... } # the regular API arguments
post('https://openscoring.du.edu/llm',
     data=params,
     headers=headers)

If you're accessing the API via code, the response headers provide information on the most granular rate limit (i.e. per-hour). e.g.

x-ratelimit-limit: 1500
x-ratelimit-remaining: 1499 
x-ratelimit-reset: 1714753675