You can build a GPT-2-style model from scratch
Dear reader,
Thank you for following our work. This list is reserved for major technology releases and selected posts, rather than every article we publish. This one is worth sharing.
We rebuilt a GPT-2-style language model from scratch in R, without any deep-learning frameworks or pretrained weights.
At the centre of the Transformer is one equation:
If that looks opaque, the post works through it from first principles and shows how it becomes a complete language model. The accompanying open-source repository can be run, modified and inspected directly.
The post completes a short series reproducing three key stages in the rapid 2015–2019 progression towards modern language models: recurrent neural networks as widely used for neural language modelling around 2015, the Transformer introduced in 2017, and GPT-2 in 2019.
Most of our work centres on datasets and discovery algorithms for genomics and multi-omics. We also develop AI models for genomic sequences and protein structures. Demonstrating with an ordinary language model here makes the mathematics and learning process easier to inspect without specialist biological knowledge.
Thank you again for your support.
Best wishes,
Our small team at
Switzerland Omics