Write Code That Delivers Value to Clients
Does a person help my team deliver software to stakeholders? If they stomp their feet and demand that we spend a week reducing the program’s memory footprint by 10% when the stakeholders don’t care, then the answer is “no”. - Online comment
Before I went down the weird-ass software road I'm walking now, I was a backend dev for eSpark Learning. Core rails app, infrastructure, stuff like that. I did some really smart things, some really stupid things, and learned I'm not the best team player. I'm still good friends with several people who work there. One project I worked on has been on my mind this week. I'm working entirely off memory here and tweaked some things for the sake of the narrative, so this isn't 100% accurate.
2016, several months before the stuff that would lead me to formal methods, I forget what I was officially working on at the time. I had an urgent assignment, though: our data scientists' queries were timing out, I was supposed to look into that. I booted into our "data warehouse" (An AWS server running Postgres), made a couple tweaks, and asked someone to try a query again.
"Is it timing out?"
"It's still running, but we don't know yet."
"When will you know?"
"Tomorrow."
The average query took a day to run. Nobody saw anything wrong with that; that was just how it always was. But this deeply bothered me. We only had a couple terabytes of data, it shouldn't take that long? I decided to look into it, seeing if I could make it take less time.
(One of my big motivations at the time was "internal happiness". I was less interested in serving our customers and more interested in serving the people who served our customers. This wasn't the first time I got mad over internal friction and it would not be the last.)
Eventually, I concluded the problem was our infrastructure. We shouldn't be running on a single Postgres box, we should have a proper data warehouse! We were on AWS, let's try Redshift. With a proper migration and schema we should make it much faster for everyone! I proposed this to management.
"Would this help students? Teachers? Admins? Revenue?"
"No, it'd just make things easier for the data scientists. I think."
The CTO took me aside and explained one of the most important rules of business: do things provide value to the customer. If the stakeholders don't care, then it's not what matters.
But I was a cocky young programmer and decided to push ahead with it anyway. I used all my free time for a month working it out. I dove into the AWS infrastructure, learning how to organize redshift tables and use the schema features. I taught myself how to use window functions and CTEs to optimize our queries. I was going to do this right, and I was going to make this engineered.
Halfway through someone else took one look at the timeouts and fixed it in an hour. The Postgres database was on a large RDS instance. He bumped it to an extra large. Problem solved. The data scientists were happy again.
But I wasn't happy, so I kept working. I built out all the infrastructure and migrated our event logs to it. I spent days tuning the database, setting up sort keys, and running tests. A month or two after I started, I proudly rolled it into production. Through tests and tears and sweat, I'd gotten over a 1000x speedup. Queries that used to take a day finished in under two minutes.
This is where, in most cases, I'd have learned just why this was a bad idea. But not this time. Everybody loved it. The data scientists could suddenly work iteratively. Salespeople could run ad-hoc queries and get useful results. Engineers found the new system stabler and easier to modify. And using a proper data warehouse saved us 60k a year over our ad-hoc one. The accountants certainly liked that. Nobody knew they wanted this until they first had it.
The customer knows most of their needs better than we do. But they don't know all their needs better than we do. They're domain experts, but we're domain experts too. They might not realize that how much happier they'd be if our products are faster, more secure, more stable. They don't see how the internal work we do has downstream effects that make it easier for us to help them better. Because they're not as versed in the bullshit of software engineering as we are, they don't know everything we can do to make things better. And they might not even realize it's better until long after we improve things.
That's not to say that all, or even most, of our ideas are like this. We're still not the customers, and we shouldn't be arrogant to think we know what they need better than they do. This project I did at eSpark was a stunning success. I had other successes like this, where I was convinced that people didn't know what they actually needed. And I had many failures, too, where I knew I could make things better, and made a change, and turned out to be wrong. I'm not saying this is easy! But this is a pendulum. We move between extremes, first thinking we always know better than the customer, then thinking they always know better than us. When we're both imperfect and just trying to figure things out as we go. What we want is not always what we need. Hindsight is 2020.
If you're reading this on the web, you can subscribe here. Updates are once a week. My main website is here.
My new book, Logic for Programmers, is now in early access! Get it here.