My current company is struggling with some challenges that are pretty typical for a wildly successful startup that's rapidly grown into a medium-sized company. We've got the expected technical debt, organizational design challenges, and a seemingly infinite number of small systems that work great... until they catch fire as we hit new scaling thresholds.

A commonly accepted notion in software engineering leadership is that managers have a much bigger potential for impact on a business than an individual contributor. This is certainly a credible argument, given that a great manager can have a huge impact through building a great team. They're responsible for recruiting the right people, steering the culture, and making the biggest decisions about what risks to take, what opportunities to pursue, etc. Ultimately, they're accountable for what the team delivers.

Over the past 15 years, I've been working on one form or another of internal developer platform. Even long before, while working at small startups, I inevitably ended up building (or curating) some little web framework, a build system, and slapping together scripts to package and deploy our stuff reliably. No one ever told me to do this, it was just obviously necessary.

In these cases, I was building a product for myself and my immediate team members, so it was a pretty tight feedback loop with the customer. I'd put a little extra effort to make things nice for other developers on my team, and also out of a bit of pride in making something that felt elegant.

At work, we're in the process of rebuilding our metrics pipeline, as we've outgrown our old self-managed TIG (Telegraf, InfluxDB, Grafana) solution. We've had this solution in place for many years, and it's served us well. Especially given the increasingly predatory pricing models of observability vendors, it's been extraordinarily cost-effective.

But over the last couple years, as we've grown, we've started to hit the limits of what we can handle with a single, vertically scaled instance of InfluxDB (especially using InfluxDB v1). It was increasingly stressful to keep it running smoothly, and we had to be very vigilant about cardinality, as it's very easy to accidentally introduce a cardinality explosion that can bring down the entire database.

As part of an evaluation of Prometheus compatible monitoring solutions, I found the need to push our use of the OTEL Collector to handle some use cases like creating metrics allowlists, renaming metrics, or adding and modifying labels.

Here's some examples, based on what I learned, of the crazy and powerful things you can do with OTEL collector processors to manipulate metrics.

It was 2004, and I had a huge chip on my shoulder.

For most of my career, I've found that I tend to quickly develop a reputation in whatever company I'm working at. I've never been the best programmer, but I've got some breadth, creativity, and critical thinking skills, and I'm good at synthesis and communication. This has helped me see the big picture in moments where a novel idea was needed, and I was able to connect some talented people to come up with some cool stuff together.

I've been reflecting recently on a really formative period in my career, when I had a chance to be part of a massive experiment in progressive engineering management.

About 3 and a half years before I left Vistaprint, I was asked to join the Engineering leadership team by our (relatively) new VP of Engineering, Erin DeCesare (who is now the CTO of EZCater). She was a particularly bold leader in terms of her progressive management ideas, and was rapidly reshaping the organization with a strong set of values around empowerment and servant leadership.

When learning a new recipe, especially when dabbling in cuisine from different cultures, I find it really important to make sure one is really precise in their understanding the words used in the recipe. I've had a few unfortunate misunderstandings that resulted in... gastronomic disaster.

Similarly, I find that I can't responsibly use the word "DevOps" without testing that the person I'm talking to know which meaning I'm using. Here's some examples of what someone may think I mean when I say "DevOps":

Every once in a while, some new product comes along that solves a problem you didn't know you had, and does it so well that after you've had it, you can't imagine how you ever lived without it.

This is how I've come to feel about Karpenter. I guess you could say that the category it lives in already existed, given it's designed to replace the Kubernetes Cluster Autoscaler, but the effect it's had on my life as an EKS cluster operator and platform engineer makes me feel like the comparison cheapens it.