Differential Privacy, pipeline mistakes, team inefficiencies
-
This is very exciting: Google has released PipelineDP, an open-source python library that allows developers to aggregate data in a privacy-preserving way, based on their internal differential privacy library. The library provides a straightforward API you can run in Spark – see this short example for a demonstration.
-
A Reddit thread on mistakes to avoid in building data pipelines. Among mentioning tools like airflow, prefect, dbt, etc, there are also pointers to Kimball’s book, which is a good reference for some of the basics of things like dimensional modeling.
-
Another Reddit post on What’s your data engineering stack has one thread totally go off on BI tools, with little love expressed.
-
Ask HN has a thread on Great Blogs by Programmers.
-
An old photo of a very large BBS that will take those of us of a certain age back in time.
-
The work of an engineer will be best if developed iteratively with the product team.
-
Suddenly there are several “the algebra of” posts floating around, but when I read Scott Galloway’s The Algebra of Decisions, I immediately thought of the benefits to engineering decisions. My interpretation of the approach basically boils down to: focus on value, embrace the learnings of failure, hold no ideas too precious, and always seek feedback.
-
This long list of questions to ask your interviewer prompted me to think, which of these is more important to me? How would I answer? I definitely identified some gaps in our thinking about product development.
-
Seth Godin’s short post on The Trough of Inefficiency in medium-sized organizations is accurate. I’ve seen the amazing efficiency of a five-person company of freelancers as well as the inefficiency in a medium-sized team. Great short read to ponder.
-
People who work with me will know which category I fall into for Interrupters and Non-Interrupters.
-
Not totally unrelated, another gem from James Clear:
When working with people, assume good intentions.
When listening to people, interpret their words in a generous way.
You will occasionally get burned and mistreated by always assuming the best in others, but it is a far better way to live than the opposite.”
Enjoy the Super Bowl!