How Tokenizers Work

Hi there!

Ever since I wrote my debugger post, I've always felt a little... cheap. I mean, I was writing a post about how to use a compiler without knowing how a compiler works! I wanted to close that knowledge gap and the best way that I knew how was to try to make a compiler myself.

So a few weeks ago, I carved out some time over the weekend to try to recreate the popular Babel compiler. By the end of it, I was able to get it done! Albeit it was only able to understand the following code and nothing more:

function hello() {
  console.log('hello, world!');
}

You can check out the whole thing on GitHub.

I learned a ton in the process too, and I'm looking to translate some of those learnings to a series of blog posts on compiler internals. I just released the first part on tokenization the other day - "Rebuilding Babel: The Tokenizer". Expect more to come!

That's all for today!

Nanda