The value of math is at the margins
The value of math is at the margins
I used to work at a Bitcoin startup. It ended up being a waste of my time, but I was lured there with promises of working on interesting mathy problems in the cryptography space. What actually happened was that the interesting mathy work didn’t materialize. Instead, we rushed to launch a product, a website, a mobile app, fix bugs, deal with customer feedback, and promote it. I spent almost all of my time on those things, and almost no time on anything mathematical. In hindsight, this was all too obvious. Startups need a thousand basic things before they need anything close to mathematical sophistication.
Google doesn’t seem to be too much different, at least in my position. I work on optimization pipelines that automate the planning of Google’s data centers. It sounds cool, but once the basic system is in place almost all of the work is decidedly unmathematical. Data is dirty, documentation is missing, outages happen, and alignment and communication are hard. Not to mention that mathematical deliberation is slower than hacking. If you take too long, organizational pressures will mean someone else comes up with a patchwork solution that becomes the established normal leaving your work fighting against a general fear of change. Add to that the all-too-common organizational shock, like a re-org, important engineers leaving, or a global pandemic, nuanced mathy work can get deprioritized until death. That’s software and business working as intended, as my managers remind me.
The value of math is at the margins of software work. Marginal here doesn’t mean “not worthwhile.” Instead it usually means “not worthwhile first.” A business can only justify spending resources on mathematical sophistication in a few narrow cases:
- A system already exists that can be compared against.
- Improving the system is hard.
- Improving the system is sufficiently valuable (more than alternative work).
- The details are easily abstracted.
If no system exists, it’s faster and cheaper to make a simple first version and see if it’s good enough (no premature optimization). If improving the system is easy, you just improve it without math. If the system is not valuable to improve, you’re better off spending effort on other parts of the business. And if people without mathematical sophistication require knowledge of the details, those details must be simple enough.
If, for example, program managers need a clear explanation of the workings of a system, then the pressure for a simple explanation works against the incentive for sophisticated mathematics. This happens to me. When I explain, “this system is a cost optimization model,” I often get the response, “so can you add a special case to prefer X?” It takes detailed, repeated explanations to show them why their request is fundamentally incompatible with the system design. And again, the other party in this scenario is more likely to go off and build a hack to hamfist their special case through, instead of developing a cost model to leverage the math, because the latter is more work in the short term. Another example, non-mathematical engineers will struggle to maintain a system created by mathematical engineers who have since left the team. This often results in the system being abandoned, or—for a team blessed with experience—not adopted in the first place because the maintainers see this problem coming.
One exceptional circumstance is when the system design is being informed by mathematical considerations. Auction design is a good example. Here the entire system is designed and scrutinized mathematically, so that the incentives of the various participants are set up so as to make dishonesty a suboptimal strategy. The resulting system often has the benefit of not appearing sophisticated, because the sophistication lies in the proofs of the system properties. For example, Google’s ad system works via a second-price auction, which has desirable mathematical properties but is trivial to implement.
However, I think it’s rare that mathematical folks are consulted at such an early stage of the design process for most products and systems. Google Ads started as a “premium sponsorship,” in 1999, then moved to a self-service (non-auction-based) model in 2000, then finally to an auction-based model in 2002—which, if I understand the history correctly, was mostly copied from a competitor. I doubt mathematical sophistication was on the table until well after Google Ads established itself as Google’s cash cow. And even then, establishing good incentives, while touted as important by leaders, often falls prey to narrow metrics and financial incentives. I’m looking at you, Facebook.
I mention Google, but PageRank (Google’s original, but now defunct, ranking engine) should be the exceptional case. It was primarily a mathematical conception, based on linear algebra, random walks, and an eigenvalue computation. Google didn’t have a precursor ranking algorithm, to my knowledge. But this actually fits within my framework. There were plenty of prior lower-quality ranking engines among other search engines. Improving upon them was both hard and lucrative. And the details of PageRank were easily abstracted away as “rank pages high if they have lots of links from other highly ranked pages.” PageRank would have likely remained a measure of publication influence (its original conception) without these circumstances.
Another exception to “math is at the margins” is when a company is large enough to have an internal group dedicated to mathematical consulting for other teams. In this case, mathematical sophistication sometimes is present at the table during design. This is because, for whatever reason, the group asking for help doesn’t have the knowledge or resources to do it alone. Such situations are special, and necessarily occur in the context of success. What other company can afford to have mathy consultants on staff? They must be large enough, and have enough opportunities, to ensure these highly-paid specialists aren’t idle. In this light, it’s no surprise that, historically, the United States government is the largest employer of mathematicians in the country.
A third exception is in independent consulting. But I know little about that. I imagine you spend a decent amount of time finding clients and convincing them to hire you.
If you’re like me, and you want to do mathematical work professionally, chances are slim that you’ll find such work at a small startup, unless its mission is inherently related to something mathy. These chances go down the more the company is oriented around a VC hypergrowth model, because growth prioritizes hacks to get to market. And it also seems clearer to me these days that within a team, mathematically flavored work comes in bursts, usually few and far in between. This leaves few options.
One: move around often and leave a team as soon as the mathy work dries up. This feels a bit selfish, and to compensate for the guilt (I would feel this way), one needs to be deliberate in providing a system that is simple enough, well-documented, and maintainable by the rest of the team. Two: find a big company and get on the internal “math consultant” team. Positions of that form are quite limited, and almost never advertised well. Three: try your hand at independent consulting.