SAIL: Agents, LLMs, Transformers, Attention, Data
Welcome to Sensemaking, AI, and Learning (SAIL). My goal with the newsletter is to share, for mainly the higher education sector, trends that matter in AI. Sometimes I take a month off between emails it seems. Things have been more than a bit hectic in the new AI-focused university work with Paul LeBlanc that I mentioned previously (more on this soon-ish - like in a few months).
When things are accelerating as rapidly as they have been over the last few months, there is an intricate balance between condition 1 ingesting information (learning, tracking trends, building things) and condition 2 outputting and sharing those experiences (i.e. writing this newsletter, blog posts, or academic articles). I've been living in condition 1 for the last few months. To really get a read and sense of what's happening, the read/write relationship is key. I'm hoping to return more to the writing space going forward.
AI & Education
Great article on AI for upskilling and workforce development. Anyone building agents, though, is playing in a space that is completely up in the air, but with massive plays by big tech. Have a look at Google's announcements on roughly everything agents this week. Or Andrew Ng's comments on agents a few weeks ago at AI Ascent.
Texas, always pushing the boundary in humanizing technology, is focusing on AI grading (automated scoring). Instead of hiring the usual 6,000 graders, only 2,000 were hired this year.
Privacy is always a concern when working with learner data. We're hoping to somewhat mitigate that, along with of course more rapid testing and model building, with a focus on synthetic data (Gretel). Here's an article from last year on the value of synthetic data that is worth a read.
Two absolutely must watch videos: But what is a GPT? and Attention in Transformers. Two critical core concepts in today's generative AI landscape. Beautifully done videos.
National AI Policies. Excellent. Via Stephen Downes
Let's build the GPT tokenizer. This technical video is well worth the 2 hour length. It's a master class in how to communicate clearly.
Could chatbot use increase loneliness with students? Tough one - loneliness is absolutely epidemic in our society. Chatbots might do both: worse for some, better for others.
AI & Tech News
Data is key to AI development (well, one of three, actually, the other two being algorithms and computation). Big tech is investing heavily in getting data to train models. If only there was a consortium of higher education institutions sharing data to help the sector build AI capabilities...
The market for AI is huge. So say the VC firms. Who of course have no possible motivation to push hype.
Top 100 Gen AI Apps. And another top 100 AI companies.
After an explosion of AI investments, and companies hitting billion dollar valuations after only a few months, we're seeing a correction. What does it look like when AI companies die? Well, in one case, it looks like Microsoft buying talent. In another it looks like floundering while playing with leadership changes. DeepMind founders now lead AI at Google and at Microsoft.
It's all about the chips. Nvidia is dominant here. But others are starting to come along. Note Meta. Intel's Gaudi 3. Google is accelerating their chip focus. Nvidia is not sitting back, with numerous announcements last month, including an industry leading new chip/architecture: Blackwell.
Lots of new (small) language models and periodic threats to GPT-4's dominance. DBRX (trained for only $10m), Mistral, Command R, Claude 3, Gemini (with an impressive 1.5m context window), etc. OpenAI announced an updated GPT-4 yesterday to reclaim the top spot on the leader board.