The approach of fall means a lot of things: kids going back to school, possibly some relief from summer heat and usually worse traffic.

For me, it also means new collegiate athletic seasons, and that means more roster data. For a few years now I’ve been collecting and publishing roster information for multiple college sports, and what used to be pretty onerous scraper maintenance work is now made easier by Large Language Models.

My focus in this work has been on women’s sports, partly because I’ve involved my students, many of whom are young men, and partly because women’s sports gets less attention than it deserves.

My initial foray into roster data came in women’s college basketball (I’ve been a season ticket holder at Maryland for 10 years now), and I quickly learned that assembling a collection of every NCAA player is, well, a lot. There are hundreds of teams, all of whom control their own sites. The NCAA eventually has a list of all of the players, but that list often is missing details such as hometown and previous school.

The good news is that most colleges don’t build bespoke roster sites. Instead, they use software from one of a few vendors, the main one being Sidearm Sports. Here’s an example, picked totally at random.

My original work on this consisted of writing a scraper for Sidearm sites and seeing how many teams it worked for, which turned out to be quite a lot? But the remainder were annoyingly different enough that eventually I had practically unmanageable scraper code that became hard to read, let alone maintain.

Once again - and I know this is a theme here - this is where a large language model really helps, because you can tell a model to have better habits than you will enforce on your own coding (at least, that’s my experience). Chasing down edge-case scraping issues also is, I would argue, not the best use of my time, or anyone’s really.

As a result, I’ve published women’s college basketball roster data even before the season has started, which is the earliest I’ve done it in the five years I’ve been tracking rosters. Along the way I’ve developed better methods for finding issues I need to fix, such as information being in the wrong column, and instead of a single Markdown notebook with some analysis, I’ve built an entire website:

The site isn’t just for a single season; it has players since the 2020-21 season, and for all three NCAA divisions. You can explore the geography of players, including where international players hail from and which states produce the most Division I players this season (Maryland ranks third, fwiw).

The other benefit of a more systemic approach to this work is that I can re-purpose the codebase to scrape other sports, which means we’ll have more women’s college roster data to come.