Issue 15: Approaching the Problem from a Different Angle
An interesting insight from work this week. Struggling with RAM usage on an ETL tool that builds a graph in memory and tracks the edges along with corresponding data pointers, I tried various methods of storing the edges: an array of arrays, a BTree, and just a flat vector that listed all the edges.
I was about to acquiesce and store the results to files on disk when the epiphany hit: I needed to write the edges to disk in both forward and reverse direction, but I didn't need to actually maintain both directions in memory at the same time. Merging the two edge trackers into one, I set it up to write the "forward" output, reverse the edge directions and re-sort the data, and then write the "reverse" output. It was slightly slower, but the RAM usage was cut in half. Stepping back and approaching the problem from a different angle saved about 100GB of RAM usage in this case.
Not much interesting on the Ergo front recently. I've been splitting my free time between day-job work -- I probably shouldn't, but it's easier to rationalize when you co-own the company -- and on writing all the missing Ergo tests. Writing the tests has found a couple bugs and revealed some non-ergonomic interfaces within the code, but I'm looking forward to getting this done so that I'll have a solid base to build upon. Once again, the examples from Zero to Production in Rust have saved a bunch of time here.
Recommended Reading and Videos
Blockly is a library from Google that presents a visual, block-based interface that can actually generate code. I'm toying with the idea of using this for an Ergo workflow builder.
I don't use React, but the Utopia code editor was a notable new project. What stuck out to me is its seamless sync between an element on the page and the code that generated it. This reminded me of Bret Victor's Inventing on Principle talk which I mentioned in this space a few months ago, but is well worth watching if you haven't seen it.
If you enjoyed this, I'd love if you share it with a friend (sign up here) or just reply to this email with your thoughts. Thanks for reading!