One Shot Learning #6: Like Tinder, but for data scientists trying to make money.
The data science job market has undergone a marked transformation since I first entered it in 2011. As a new round of job advice hits the Electronic Tubes, I'd like to share my updated thinking, in the hopes that folks find it useful.
Before that, though, I must admit I've made some, uh, "rough" claims in the past about the skills one requires to be an effective data scientist. Young and Naive Alejandro once felt that graduate machine learning classes and Knuthian engineering were requirements for a true data unicorn's skillset. Nowadays, Aging-Like-Fine-Wine Alejandro realizes two things about these claims. First, they reflected the experience of someone privileged enough to attend graduate school in a booming tech market. Second, these claims were wrong.
Why? Labor markets emerge from a simple matching problem: companies need skills and experience at a certain price point, and candidates offer their skills at other price points. Companies try to fill these roles by offering benefits, perks, and equity or cash compensation to people. Conversely, people only supply labor if their needs are met: some only ask for catered dinners and interesting problems; others prefer parental leave and retirement accounts. Not all company require expertise in deep learning or distributed systems!
Nowadays, I believe the data science job market exhibits two fundamental characteristics. First, matches for highly-experienced roles are primarily driven by the data person. Fierce competition from companies with lots of data - like Facebook and the rest of the FAANGs - and packed coffers has led to aggressive demand. We can confirm senior salary numbers quickly in Hacker News posts from 2016 and 2018, or a 2018 Medium post on salary transparency. When discussing total compensation (salary and equity) with senior-level data people, I've heard that packages starting in the $250-300k range are common, and packages over $400k are available to very senior ICs and managers. Meanwhile, supply is tight, since these workers must lead high-return projects reliably, require less supervision, and (hopefully) provide technical and soft mentorship. If you are not a FAANG, you need to recruit these people over a long timeline or a non-financial dimension, like work-life balance or company vision.
While this situation may bode well for experienced candidates, matches at lower levels of experience have been complicated for both companies and workers. The supply side of this equation should not surprise anyone. The academic job market has tightened and cheapened from 2010 (in The Economist) through 2016 (per The Atlantic) and continuing into this year, per The Washington Post.
Bootcamps and academic programs have proliferated in all major metropolitan areas. Consider this list of 20 academic data science programs. It covers all cardinal and intercardinal directions within the continental United States! Take a minute to stand outside sometime this week, surrounded by tiger swallowtails and warmed by the June sun. Stretch out your arms as far as they go, and twirl around in place. Spin spin spin, your fingers slicing the dense, humid air. Then stop. There's a decent chance you're pointing in the direction of two schools with data science departments, each stacked with classrooms looking like this:
The introductory data science course at @Cal is Data 8. The course is so popular that it's in Zellerbach Hall. Fall semester 2018, Day 1. pic.twitter.com/VbHtPnikmw
— Mike Olson (@mikeolson) October 4, 2018
Companies also struggle to cope with this wave of junior-level candidates. Anecdotally, hiring managers are inundated with resumes and coding assignments. Job applicants may inflate or misrepresent their contributions on a resume. This is symptomatic of imperfect information in job matching. Caitlin Hudon's poll on data science engineering skills offers a nice example:
Those hiring data scientists:
— Caitlin Hudon👩🏼💻 (@beeonaposy) May 28, 2019
Is it a good idea for a data scientist to take a position as a data engineer for a few years to build their software engineering skills if their long-term goal is to be seen as a highly technically competent, full-stack DS?
Peep the number of people responding "Show answers." Almost half of the respondents sought more information! The split between "Yes" and "No" also illustrates a split in the expected engineering abilities of candidates, a point noted by Robert Chang back in 2015. Even someone in the comments called this out:
This new world of merging skills is great, but it’s so frustrating to see job descriptions that predicate both A & B skills.
Amen, my guy.
Luckily, hiring managers are increasingly encoding these expectations in job descriptions and titles. If companies need to build and maintain a warehouse, they now hire data engineers, and increasing operational visibility and requires data analysts or decision scientists. Companies who sift through the deluge of resumes effectively and aggressively recruit for their needs will find ambitious candidates ready to deliver value.
On reflection, I see my description of the market paints a bleak picture for junior candidates. As a salve, I'll respond to Caitlin’s question. One of my first roles in 2011 carried responsibilities closer to “junior site reliability engineer.” I mean, I maintained a Graphite cluster. Through that work, I developed foundational engineering skills that I still employ today. On reflection, I actually followed Vicki Boykis's advice to aspiring data scientists:
It’s much easier to come into a data science and tech career through the “back door”, i.e. starting out as a junior developer, or in DevOps, project management, and, perhaps most relevant, as a data analyst, information manager, or similar, than it is to apply point-blank for the same 5 positions that everyone else is applying to. It will take longer, but at the same time as you’re working towards that data science job, you’re learning critical IT skills that will be important to you your entire career.
I still agree, eight years on. Leverage your strengths and find a good match.