I haven't written much for the past few months- for a couple reasons.

First, I've become utterly addicted to AI coding tools, and I haven't been able to pull myself away from watching my AI agents absolutely tear through my feature backlog. Suddenly nothing can escape my reach, from long-postponed major refactoring, to minor annoyances, to features that were tricky enough I wasn't sure they'd ever make sense to tackle.

Even my old nemesis, CSS, is no match for me on a dopamine-fueled, manic, AI coding rampage.

Secondly, while I'm learning an insane amount with this stuff, it feels like all my mind-blowing insights have a shelf-life of about 5 minutes. Everyone else in the world seems to be coming to all the same conclusions. The next morning, everything I write just seems painfully obvious.

Please don't tell anyone, but I wrote this (gasp) by hand​

Despite being neck deep in a frantic, explosive bout of creativity, I've occasionally taken some time to go a bit beyond just reviewing the code Claude's writing for me, and to think about it a bit more deeply. A few times I've even caught myself typing- realizing that I'd been at it for a half-hour or so- without noticing how objectively weird it is to be doing that when I have an army of tireless, magical code-writing gremlins at my disposal.

I realize it might appear that I'm wasting time in an anachronistic attempt to hand-craft some artisanal TypeScript- to infuse it with some humanity and love... or some shit like that. Or perhaps that I'm just having trouble letting go of the joy I used to experience from the puzzle-solving aspect. I mean, it might be a little of those things- at least it was at first. But it isn't anymore.

Within only a few months after committing myself fully to getting good at applying AI to engineering, my brain has already been completely rewired. A few months ago, AI-assistance was getting me at most a 20% productivity boost. Now, with better tools, a new mindset, and lots of practice- it's conservatively like 5-10x.

The other day, when I set out the ingredients in preparation for cooking dinner, I was viscerally disappointed when I realized I couldn't spawn subagents to chop the onions, peppers, and carrots.

Don't lie, you know this has happened to you too.

Old habits​

I've been ruminating on why I still occasionally revert to my old habits. I know the code I write by hand isn't any better than what Claude would do, and I'm absolutely sure I could have achieved the same outcomes (certainly faster) by prompting it. But at the speed the agents were moving, something felt off about the how much time I was spending thinking deeply about anything.

And I think, until now, I hadn't appreciated how much programming could be like writing: a tool to facilitate thinking.

Paul Graham once wrote, in a tiny essay on writing:

I think it's far more important to write well than most people realize. Writing doesn't just communicate ideas; it generates them. If you're bad at writing and don't like to do it, you'll miss out on most of the ideas writing would have generated.

This hits home. I abort roughly 2/3 of my attempts to write blog articles, mostly because the process of writing helps me see how fucking stupid some of my own ideas are. Sometimes I start writing out a thought, and find that it's become completely unrecognizable by the time I'm done- because writing forced me to think it through. Seeing my own thoughts in concrete form allows me to feel how they might land on another person, which helps weed out all the bullshit and expose what seems true.

This is valuable.

Programming is also thinking made concrete​

When I'm really in the zone while programming, I float between a meditative, dissociative state, and a more analytical, critical one. It forces me to engage with a problem in abstract terms, and then pop back up and consider the broader effects of the changes I'm making.

This mindset gets to the core of what humans add to the AI-assisted engineering equation. I'm still way better than Claude at knowing when to step back and ask questions like:

• what tradeoffs come along for the ride with this change (especially tradeoffs beyond the knowledge and context window of the LLM)?
• what are the intrinsic relationships/coupling between entities, and how well does this model the corresponding concepts in the real world?
• what will the 2nd order effects of this change be?
• what social/emotional/behavior outcomes am I actually looking for?
• are there assumptions built into my idea that I could test with less risk?

But without any time spent with my mind immersed in types, data structures, or algorithms, I found it was really easy to get swept up in a what felt like a creative frenzy, only to later realize that I was too disconnected from the details of the problem, and my instincts ended up being all wrong.

Code relates to the real world in all kinds of ways- some obvious, and some much more subtle. Human intuitions around something this complex require some effort to nurture.

Sometimes, the inherent difficulty of programming, like writing, is epistemologically valuable. When tools remove all friction, they can also remove the struggle that creates the opportunity for insight.

AI authored patches can anchor your conceptions​

A related effect of relying primarily on agents is that the code diffs they generate can reinforce and calcify your preconceptions about a problem space.

Another relevant quote, from George Orwell's essay, Politics and the English Language:

But if thought corrupts language, language can also corrupt thought.

Orwell refers here to "ready-made phrases" (e.g. cliches or idioms) whose power, by virtue of tradition or sheer catchiness, can infect patterns of thought and leave their victims vulnerable to sloppy thinking- and prone to drawing illogical conclusions.

I feel the echoes of this phenomenon when I read LLM-generated code. While they can certainly generate code that's elegant and idiomatic, it also has the side-effect of anchoring us into a particular conception of how a problem is shaped.

Since LLM output is largely a product of the language we use to prompt it (and the context we give it explicity), it can be easy to accidentally produce output that's a reflection of our own (perhaps flawed) mental model. Plus, because LLMs are trained on all the code in the universe, we're likely to get middle-of-the-bell-curve ideas as outputs, unless we work deliberately to push it towards something more interesting.

Cognitive tools: old and new​

Recognizing this, I've also developed plenty of tricks to use LLMs as a critical, antagonistic thinking partner: one that helps me poke holes in my own ideas and helps me explore divergent paths. This is, for me, a really amazing new tool to amp up my own creativity- and to combat my own cognitive biases.

But I don't think it's the only tool available. We've got tens of thousands of years of human history of creativity and critical thought behind us. Long before AI started spewing out probabilistic responses, we developed all kinds of tools and processes to facilitate innovation- and weed out bad ideas.

Written language has been a huge one- for several thousand years. Programming, which has been around for (depending on your definition) something like 80-180 years, has a lot of similar properties, certainly in communicative power, but also as a method to visualize concepts, force logical reasoning, and root out ideas that don't hold water.

Human thought is still in style​

It can be easy to think of code only in terms of its ostensible purpose- a way to describe processes to be executed by a machine- and forget that it's also an interface designed for humans to be able to express these processes and reason about them. Programs exist in a much broader context than LLMs can be aware of, and humans are still far better at working at that level of abstraction.

I feel like there's some value to me in retaining a connection to code. For several decades, marinating in data structures and algorithms gave my brain time to absorb and synthesize concepts, explore spaces- imaginary and real- and serendipitously stumble upon new perspectives.

Its possible I'll begin to see spec-writing (i.e. "programming in English") as the successor to programming in this way: an activity that forces deep reflection and forges clarity of thought. I'm open to this, and I intend to give it a real go.

But there's something about the precision of programming languages, versus natural languages, in particular, that has a tendency to activate different parts of my brain- which has been genuinely invaluable in helping me solve problems that other cognitive modes didn't- on their own. This isn't something I'm quite ready to give up.

Even if it sometimes feels like an anachronism.

I've been thinking a lot lately about the significant discrepancy I've observed in how effective different people are at using AI for problem solving. I'm certainly not the only one; there seems to be a strong consensus building in my field (software engineering) that AI tools disproportionally augment the effectiveness of experienced people (e.g. those who developed their craft before AI tools became omnipresent), and that providing AI to people early in their career can actually slow them down- and impede their growth.

This certainly fits the observations I've made personally, but raises a lot of questions about what experienced people are doing so differently with AI. I've spent some time intermittently reflecting on my own use of AI, and also paying attention to how some of my younger colleagues are using it, and I think I have a working theory.

Behold: Science!​

Like many people who spend their careers solving technical problems, I employ a version of the scientific method:

1. I start with my existing mental model of how a system works (based on previous observations). Like any scientific model, this is always varying degrees of incomplete/incorrect.
2. I use my model to formulate hypotheses about what changes to the system will result in a solution to the problem I'm working on.
3. I decide which of the hypotheses I should test first. This incorporates several factors:
   • the hypothesis's likelihood of being correct
   • how difficult and time consuming it will be to validate
   • what side-effects or tradeoffs come along for the ride
4. I test the best hypothesis and use the findings to improve my mental model.
5. Problem solved... or... rinse and repeat with my refined model.

This isn't just for problems of a certain granularity; it's fractal, and works at any level. I approach both multi-month and 5 minute problems this way (with varying degrees of rigor, depending on how well caffeinated I am). As I break down big problems into smaller ones, recursively, these tools (mental model, hypotheses, running tests) maintain their value.

The forking road​

I think of problem solving like traveling a road that branches into infinite forks. Each fork has different probabilities of success, different risks, and different tradeoffs. Some have hidden dangers, or they might be really exceptionally long and difficult to travel. On top of that, any of them could be dead ends.

Following any fork- even the dead ends- could reveal valuable information, which you can use to refine your mental model (which in this metaphor would be something like an aerial map). But it costs time and energy to take any one, so you need a methodology to make each decision as efficiently as possible.

If there's a better way than the scientific method, I'm not aware of it.

What experienced problem solvers do differently​

Putting aside AI completely, the biggest advantages experienced problem solvers have in any given domain are:

1. They have a more accurate, well-developed mental model of the problem space, and they invest time in continuously refining that model.
2. They use this model to formulate better hypotheses, prioritize based on chances of success, and estimate the effort required to test them.
3. They design and execute these tests efficiently- they focus on building understanding and reducing uncertainty with the minimum necessary time/effort.
4. They have a general awareness of when it would be more productive to double back instead of plowing forward. After invalidating a hypothesis, they're willing to retrace their steps past multiple previous forks in the road and use their newly refined model to reevaluate a previous decision.

Novice engineers are pretty good at noticing how AI impacts #3 (executing tests on a specific hypothesis). In fact, I don't think they're usually thinking of what they're doing in terms of a scientific method with discrete parts; they're just instinctively taking guesses and charging ahead. The test result is often binary: "problem solved" or "problem not solved".

AI is so freaking powerful that this approach can be enough to solve a lot of kinds of problems. This is especially true if you've got an adequate mental model out of the gate, and you can craft a prompt with sufficient context. If you're dealing with a problem in a well documented space, there's a good chance that an AI can land you a hole-in-one on your first try.

But for an inexperienced problem solver faced with a problem that falls slightly out of this ideal zone, they can easily become lost in our branching road of possibilities. Or, they get stuck on a particular branch and keep trying different variations of a flawed approach.

Avoiding forks entirely​

Seasoned engineers often get a gut feel that they can't be the first person facing a particular problem they've encountered. And it's often true- depending on the question you're asking, the answer (or relevant data that can help you answer it) might be readily available on the internet.

As it turns out, AIs are exceptionally good at rapidly scouring the internet, ingesting huge swaths of text, analyzing and summarizing it. These types of questions can require orders of magnitude less time and effort to answer with AI than by a human doing the research.

Experienced engineers get so much more out of AI in this realm because they have the instinct to ask the question before starting any work- and AI reduces the cost of answering many of these questions dramatically.

Scouting the fork ahead more quickly​

When initial research leaves you with open questions, you're left with investment decisions: which experiments do you want to spend time on? For a software engineer, this often means writing enough working code to try out an idea- which can require hours/days of your scarce and valuable time. This can be a risky bet on a hypothesis- one that may bear no fruit.

With AI, you gain the remarkable ability to execute tests like this with superhuman speed. If you can articulate your experiment in a prompt (and usually guide it through some iterations), a decent AI agent can take care of the mechanics and gruntwork to generate the minimum viable code to answer a whole lot of questions- lickety-split. The fact that AIs tend to write code that's less than production quality is of little consequence when your goal isn't writing production code- it's answering questions.

The AI speed difference in this phase of the process isn't just incremental, it's transformative. An AI-assisted human can evaluate exponentially more hypotheses than a human alone. Experienced engineers are much more likely to utilize AIs in this way, and better equipped to interpret the results of each experiment.

Discovering which forks exist​

The two most common failure modes I observe when inexperienced software engineers get their hands on AI tools:

• they very rapidly execute fully formed solutions that carry very significant (and often unacceptable) tradeoffs they didn't recognize or understand.
• they fail to solve the problem, and get stuck trying to find ways to solve it within the constraints they assume apply, but don't fully understand.

Using our forking road analogy, they had gotten stuck after choosing a path without finding out what other paths existed.

In both cases, I would generally guide my younger colleagues through some divergent thinking. Experienced problem solvers deliberately invest time in idea generation, exploration, research, and brainstorming- to expand the realm of possible solutions they can consider- before they begin converging on a specific solution.

Divergence is a core element of human creativity- and applies especially to difficult and novel problems which often require diversity of perspective and fresh approaches to solve.

Using AI to promote divergence​

The first step in divergent thinking usually involves expanding the scope of your existing mental model by surveying preexisting knowledge. For software engineers, it might mean reading up on some relevant computer science concepts, product documentation, or open-source code- and letting this open up possibilities beyond the constraints you previously perceived.

This is where I think AI is drastically underutilized by inexperienced users- for exploration, learning, and idea generation in a broad problem space. AI is incredibly useful when used in this way. In our forking road analogy, it's like having the ability to fly a drone to scout ahead so you can see all the forks available within a few hours' walk, eliminate the ones that obviously result in dead ends, and discover new ones that look very promising.

AI's current flaws, like hallucination and sycophancy, are a lot less problematic when you're using it to augment divergent thinking- since any idea generated in this phase of problem solving (including by humans) can be flawed. It's part of the game- you take early ideas with a grain of salt.

Build your mental model, then go nuts​

Lately I've been putting deliberate effort into using AI in this way: to build my own understanding of problem spaces. I've been using AI agents as research assistants that I send off on missions to educate me on some topic, at some level of abstraction that I'm interested in. This is usually fairly broad and shallow to start, and from there I can explore the space interactively by peppering them with follow-up questions.

This is a process I used to do with Google searches, O'Reilly books, and a whole lot of reading in bed. It's a process of synthesis- sifting through large amounts of information to build a mental model, so there's a large amount of waste involved, by definition.

If it had been an option, I would have much rather spent an hour talking with an expert on a topic and grilling them with questions. Having to grind through hundreds of pages of source materials on my own, when 95% of the content wasn't relevant, or wasn't at the level of abstraction I was aiming for, wasn't an efficient way to learn.

Current LLMs are incredibly good at this mode of exploration, especially if you keep your skeptic hat on all the while. Applying a little Socratic method to any aspect of their findings that triggers my spidey-sense often helps them self-correct. It helps that I have years of experience engaging in this same process with humans- who, I have noted, are also sometimes wrong.

Having an "expert" give me a broad overview of a problem space also makes me a whole lot more efficient at doing my own manual research. Armed with concepts, vocabulary, and starter sources, I can get right in the weeds and begin reading primary sources on a topic I'm a complete beginner in.

Once my mental model is sufficiently filled out, I find I'm way more effective at the subsequent phases of problem solving. I've developed some vocabulary around key concepts and their relationships, and my intuitions allow me to formulate much better prompts to guide AIs through code generation. I'm also much better at recognizing subsequent hallucinations or other bad ideas.

AIs don't grant problem solving skills​

Many non-engineers hold the misconception that software engineers spend most of their time and effort writing code: the kind of code that LLMs are getting increasingly good at writing every day. It is true that we spend a good amount of time staring at code on a screen, but obscures the truth that our time is absolutely dominated by more general problem solving: designing, debugging, troubleshooting, and analyzing systems.

Engineers who understand how to problem solve using the scientific method, who value mental models, and who practice deliberate divergent thinking, have always outperformed those who don't. Augmenting them with powerful AI tools is going to drastically accentuate this discrepancy.

A couple years ago, my daughter, Sadie, began a gradual transformation that eventually led to her being able to honestly describe herself as "bird obsessed". Everywhere we go, I notice her staring up at trees, rooftops, and sometimes vaguely up and into the distance. When she was younger, she used to get lost in daydreams, and I thought this was just a remnant of that, but I've come to understand this is actually a product of intense focus and fascination- with our feathered neighbors.

I'm not sure exactly where this all started, but it was most likely rooted in her interest in photography. I'm told this is common: photographers become birders, and birders become photographers- it's a pretty natural progression.

We took a trip to New York about a year ago, and my wife and I were a bit surprised (and a little disturbed) that at least three quarters of Sadie's photos were of pigeons. Like most city dwellers, I tend to think of pigeons as dirty sky vermin, but to her, they were unendingly curious and delightful. They apparently have intricate social lives, with backstories, drama, conflict, family ties, and complex rituals. She told us about how these feral birds are actually the descendants of escaped domestic pigeons, who themselves were originally bred from wild "rock doves", who are cliff-dwelling foragers.

Last Thanksgiving, she got to know one of our family friends: an octogenarian, former biology professor and amateur wildlife photographer who lives in South Florida. He invited her to join a text chain with himself and some of his friends, where he shares a recent photo every day. As it turns out, she finds this absolutely splendid, and despite being at least sixty years younger than any of the other participants, enjoys both the photos (which are admittedly quite beautiful) as well as the banter (observing the way a group of eighty-something former professors struggle with SMS can be entertaining in itself).

While her trajectory was probably already predetermined, seeing these pictures definitely accelerated things. It also helped that we gave her some money for her birthday towards a new camera, which she supplemented from her own earnings as a camp counselor to buy a pretty nice entry-level Canon and lens... which she now carries everywhere.

Fast forward to today: Sadie has become shockingly knowledgeable about birds. She recognizes virtually every bird in our local area by their calls, and can rattle off facts about Cedar Waxwings, Chimney Swifts, Carolina Wrens, and a hundred other birds I would have never known about, absent our dinner conversations.

As Sadie's dad, I'm certainly thrilled that she's found something she's so excited about, especially given that it gets her outside and away from her phone and social media. There's certainly worse things a teenager could be into.

But what's been most surprising is how much her interest in birds has impacted my life as well.

As a person who types for a living, I have a tendency to forget to exercise. Having a high-energy dog has helped a bit, since it forces me to go for a walk at least a couple times a day. I do enjoy walks, and occasionally go on a longer hike. It wasn't until birding that Sadie started to get interested in going on more of them with me.

Of course, I love just having an excuse to spend some time with her. Teenage girls don't exactly have a reputation for enjoying time with their dads, so I'm savoring time I know is going to slip away fast. But beyond just that, I've noticed some changes in the way I think about my relationship with the outdoors.

When I hike, I tend to get lost in my own head- often working through whatever programming or management challenges I'm facing at work. This has generally been useful to me- an afternoon dog walk usually helps me to step back and rethink whatever I'm working on, and I usually end up finding better approaches than I would of if I'd just kept plowing through.

But since I've been doing birding-oriented walks with Sadie, I've found my attention focused in new ways. I'm seeing more of what's happening all around me: absorbing the movement of the trees, the wind, or little flutters in the brush. My full brainpower is engaged processing data from my peripheral vision. When we hear a bird call, we fall silent, trying to pinpoint the source of the sound, and stand quietly for a few moments.

All this use of my attention feels a lot like what I've tried (and failed) to achieve in the past through meditation. I always had trouble maintaining the patience and persistence that it takes to interrupt my thought patterns- where I get stuck repeatedly reliving the past and anticipating the future.

It's been much easier to achieve this kind of mindful focus when we go birding together- especially when she's radiating excitement about a new bird. I can't help but absorb some of her enthusiasm, and it helps to sustain me and keep me present.

What's more: I find that this attention to my surroundings is sticking- even when she's not with me. Previously, odds are I'd be visualizing data structures, replaying the latest difficult conversation I've had with a coworker, or being angry about a government causing needless suffering.

Instead, I'm directing more of my attention to ferns, wildflowers, rock formations, or the flow of water in the creek that borders one of my favorite trails. I'm noticing the sensation of the wind on my skin, or the feeling of endorphins from the exercise.

I recently started have these little moments of spontaneous gratitude- being momentarily overwhelmed by how much beauty there is to be found in my little pocket of Eastern Massachusetts.

I have friends and loved ones who experience real, chronic anxiety and depression, and I recognize how lucky I am that my head isn't generally an unpleasant place to be. I do often enjoy getting lost in an engineering problem while I walk or drive, and thinking through recent social challenges often allow me to temper my initial emotional reactions- which I think make me a better colleague, friend, dad, and husband. Even some of the time I spend worrying about politics is useful as motivation to get me off my ass and do something.

But I have had trouble, especially over the last few years, worrying a lot about things I can't control- in ways that aren't helpful to me, or anyone else- ways that are more paralyzing than motivating. It's given me a bit more understanding of what debilitating anxiety might be like.

Birding with Sadie has, surprisingly, given me a new perspective on how I can choose to use my attention. I still spend some of my walks in my imaginary idea world, but I'm finding more and more often, I can take a few deep breaths, reset, and listen for bird calls.

I chat with a lot of younger engineers about their future, career goals, and what kind of work makes them happy. By far, the most frequent conversation I have is about engineering management- what it involves, how to get into it, and whether it's a good fit for them.

I've also worked with people who've been thrown into management without any real guidance or preparation. This can be a really hard situation, because you absolutely need to be able to ask for help, but it can be terrifying as a new manager to admit that you're struggling.

I thought I'd spend some time summarizing the conversations I often have with prospective managers of what I think life as an effective engineering manager looks like. I'll start with the high level purpose, and then we'll get into the details of what day-to-day activities look like.

The whole point is business outcomes​

Let's strip away a bunch of complexity and make this really simple. There's one fundamental purpose of an engineering manager: to deliver business outcomes.

Wait, just that one thing? Yeah, though I understand your confusion. It's easy to get distracted by all the things that managers have to do to make this happen:

• recruiting and building teams
• coordinating priorities and roadmaps
• nurturing culture
• owning technology
• managing stakeholders
• growing and mentoring team members
• ...and a million other things

Yes, these are examples of activities that a manager generally has to do to drive business outcomes, but don't conflate the mechanism with the underlying purpose of their role.

Managers are given a lot of power over people, processes, budgets, and other precious resources, and in exchange, they're accountable for what their teams achieve. The management hierarchy depends on this accountability to carry out their strategy across the organization as a whole, and thus managers are essential as points of leverage for senior leadership.

Some cautionary tales​

The Machiavellian vibe of this point sometimes elicits doubt, so let me put this in even more stark terms:

An engineering manager whose team consistently delivers successful business outcomes can get away with a lot.

Here's some examples, from my personal experience, that illustrate the extremes of this dynamic:

That guy's a jerk, but he builds a mean product​

I once worked (adjacently) with a manager who I personally (quietly) believed to be an asshole, but who had been successfully running a team that was doing some admittedly great product innovation. He was a darling of the executive team, and was given extraordinary leeway to run his team in some quirky and unconventional ways.

I later found out one of the "quirky and unconventional" things he was doing was emotionally abusing the shit out of his team. This had apparently been going on for a while, and while he did eventually get fired, it was only after multiple, excellent engineers left the company (after having the courage to lodge complaints with HR, even in the face of explicit threats of retaliation). Even after all this, he was only actually held accountable once all this started to impact his team's delivery.

A "healthy" team isn't the only goal​

Around the same time, I worked with another manager who was very bright and competent, a deeply caring, empathetic leader, and all-around lovely person. She was very experienced in building teams with a great, positive, supportive, collaborative culture of engineering excellence. They had many of the hallmarks of a high-performing team: motivated, engaged engineers who felt empowered to do their best work, and who really enjoyed working together.

But despite her talent for nurturing teams' health and culture, she wasn't particularly connected to our customers' needs, or the technical details of the systems her team was building. This misalignment of focus built up over time and led to some underwhelming results. She eventually lost the trust of her stakeholders, and was managed out as her team got re-orged.

Business isn't about harmony​

If you're thinking both these examples are really fucked up- yeah. While I wish businesses were motivated primarily by creating a great working environment, nurturing a culture of respect, creativity, and collaboration- they're not. They're motivated by economic success- and everything else they do is in service to that.

Luckily for us, thought leaders in our industry have discovered that, at least in knowledge work, value delivery strongly correlates with great cultures and empowered, motivated teams. Businesses that build great cultures do so with the belief that it makes them more competitive- and they're often right.

It's really critical as a manager to recognize the nature of this dynamic. Successful managers use the framing of business outcomes to make decisions, articulate their strategy, and execute against their plans.

OK, so what does a manager actually do?​

So now that we understand the primacy of business outcomes, let's do a quick, 100 level overview of the mechanisms and activities that an engineering manager generally has to do to deliver them.

Strategy and alignment​

Successful engineering managers act as a bridge between their teams and senior leadership, translating the organization's strategy into tactical, incremental goals for their team. This happens through a few different channels:

• Cascading high-level strategy through the management hierarchy
• Helping the team formulate a roadmap that aligns with the strategy
• Facilitating alignment with product management and other business stakeholders
• Facilitating alignment with other engineering and supporting teams (UX, InfoSec, etc)

Managing up​

This isn't a one-way flow of information. Effective managers also influence the decisions of their senior leadership, usually because they have visibility into details of the work on the ground- constraints, opportunities, trade-offs- that senior leadership doesn't.

Being able to effectively identify and articulate these constraints in terms that the business understands is often what distinguishes successful managers from the rest. And really effective managers go beyond surfacing impediments- they show up with solutions, and they're able to sell them to senior leadership and other stakeholders.

Facilitating alignment with external teams​

External teams are likely have different reporting lines, priorities, processes, and cultures than the team(s) you manage. You don't have any direct power over them- but you depend on each other to get anything done. How do you manage this?

Successful managers build and nurture trusting, positive relationships outside of the team: with senior management, stakeholders, other managers within Engineering and across other functions. There's no trick to this: it requires a lot of empathy, integrity, follow-through, and attention to detail- consistently- over a long period of time. The reputation you build is a huge factor in your ability to succeed.

Team building​

Counter to the instincts of many new managers, building a high-performing team isn't about exerting control over people, rather, it's an exercise in system design. As a manager, your job isn't making all decisions and handing them off for execution (a.k.a. "micromanagement" or "command and control"). Your job is to make sure good decisions get made.

You do this by defining processes, culture, policies, and other organizational structures that come together, along with the people you hire, to form a self-sustaining system. You need to continuously optimize this so that team members feel connected to the mission, understand their freedoms and constraints, and are empowered to make decisions.

Driving continuous improvement through visibility and accountability​

Because there's no "one size fits all" formula for building a system as complex as an engineering team, managers need to guide the team through a process of continuous improvement. As a manager, you'd help to set the wheels in motion- establishing the processes and culture to enable the team to repeatedly self-reflect on their work, identify areas for improvement, and implement them.

As the steward of this process, you need to establish two important controls:

• visibility into your team's operations
• accountability for the team's outcomes, and individuals' contributions to these outcomes

Some examples of visibility mechanisms:

• attending team ceremonies (e.g. stand-ups, planning, retrospectives, operations reviews, etc)
• holding regular 1:1s with team members
• meeting with stakeholders to get feedback on the team
• soliciting direct feedback or through formal performance reviews
• tracking work through management systems (tickets, kanban boards, CI/CD systems, etc), including metrics derived from them (e.g. cycle time, deployment frequency, change failure rate, etc)

While accountability mechanisms are all ultimately derived from the manager's ability to promote/fire, in practice, you utilize this power very, very rarely. 99.9% of the time, accountability is driven through:

• Providing feedback to team members (e.g. praise, encouragement, guidance, or critical feedback)
• Clarifying expectations by updating processes, policies, etc.
• Mediating when misalignment/conflict arises (within the team or with external stakeholders)
• Direct involvement/interventions in various team decisions

In cases of persistent underperformance, you have an unfortunate, but critical, responsibility: ramp up the directness of feedback, engage HR, formulate performance improvement plans, and occasionally manage someone out.

On the other hand, if you're lucky, you'll have people on the team who are growing quickly, having a big impact, and just overall killing it. In most companies, you'll need to do some work to compile documentation recommending a promotion. The process can be a pain, but man, it is super rewarding to be able to tell someone you manage that they've earned a promotion.

Advocacy and removing blockers​

As a manager, you're in a unique position to help your teams identify impediments and obstacles- things that are slowing them down or preventing them from finding the best solutions to problems- and help knock them down. Does your team need some new SaaS tool? Is UX constantly late delivering mockups? Are you blocked waiting on InfoSec to complete a security review? It's your job to make sure these obstacles get removed.

Note that I said that it's your job to make sure these obstacles get removed, not that you have to do it yourself. A strong team will feel empowered to reach outside their normal boundaries and work to address external blockers themselves. A really successful team will build their own cross-organizational relationships, reputations, and influence- expanding their ability to solve problems autonomously.

Recruiting​

Recruiting is one of the most impactful things a manager does, because the choice of people to add to the team will have huge and lasting effects, often well beyond the manager's tenure.

I don't think I'm the only person who absolutely hates the process of recruiting. It's often grinding, tedious, and stressful. Making the decision to hire someone after one interview can feel a lot like getting married to someone you just met at speed dating. And if your instincts tell you to pass on someone who you're on the fence about, you've just committed yourself to many additional hours of reviewing resumes, interviewing, and debrief sessions.

Hiring the wrong person can cause enormous, long-term problems for you and your team. It can be very, very hard to manage someone out- not just in terms of time, but in disruption to you and your team, not to mention your own emotional energy. Having to fire someone absolutely sucks.

But damn, when you hire the right people, it can be utterly game changing for you and your team.

Career management​

Finally, as a manager, you're generally expected to help coach your team members, helping them grow their skills and progress their careers. This is a little different than the type of mentoring that happens in the context of the day-to-day work; it's usually framed in terms of longer term goals. While this depends on their personal goals, it's most often framed in terms of helping them grow towards a promotion.

This is usually an exercise in assisted self-reflection, goal setting, and articulating the skills and behaviors that objectively define the next level of seniority.

It also might entail advocating for someone externally, helping to open doors for them, encouraging others to take chances on them, and put your own reputation up as collateral.

No seriously, what does a manager do all day?​

Fine, it's meetings. Always meetings. So many meetings.​

What does this mean in terms of day-to-day work? I'll be honest- it means a metric shit ton of meetings:

• ceremonies: stand-ups, planning, retrospectives, scrum-of-scrums
• 1/1s: career coaching, status check-ins
• brainstorming/design meetings, working sessions
• performance management meetings
• recruiting: interviews, debriefs
• Ad hoc conversations
• ...and a head-exploding amount of other meetings

To a new manager, especially one who was recently a high-performing individual contributor, this can seem wasteful- since meetings don't solve actual customer problems. But as organizations grow, managers are increasingly necessary as conduits of information; they maintain alignment and combat entropy. They make sure that teams aren't just working effectively, but that they're working on the right things.

You absolutely need to be careful about how you use meetings- they're a very expensive way to use people's time. But if you're using them carefully, preparing for them in advance, running them thoughtfully and efficiently, and you're actually achieving alignment and clarity, then it's not a waste of time.

Also writing, too​

You'll likely also spend a lot of time creating and consuming written communication and documentation. Some organizations have stronger cultures of knowledge management than others, but at the very least you'll be authoring/editing work management artifacts (e.g. tickets, designs), writing up meeting agendas/notes, sending out status updates, and juggling a whole lot of ad hoc conversations (e.g. in Slack, etc).

Writing is certainly important for all of these cases, but it's also really important as a leader to put additional effort into using writing as a tool to refine and clarify your thinking, especially when your ideas will carry disproportionate weight in the organization.

Staying connected to the practice​

Despite the pressures to constantly keep everyone aligned and make sure outcomes are being achieved, you need carve out time keep yourself connected to the actual practice of engineering: customer needs, your teams' backlogs, your tech portfolio, industry trends, best practices, etc. This may take the form of doing some side projects, reviewing code, or even just listening to a ton of podcasts.

I chose to optimize my career for happiness​

If you've read other stuff I've written, you may know that I left management a few years back, and went back to being an individual contributor. This was actually a pretty hard decision for me, but one I'm really glad I made in retrospect.

Life as a manager is characterized by several things I actively sought to minimize as an IC: meetings, interruptions, and multitasking. As an IC, you generally seek to keep your work-in-progress to a minimum, and focus on getting a series of small, incremental things completely done. I never found a way to do this as a manager, since you have to be fairly reactive and attentive to the needs of everyone else around you. You're also responsible to handle lots of requests generated by external teams that don't track (or care) how much WIP they're generating for you.

I found managing much more draining from a social/emotional perspective, which was difficult given my tendency towards introversion. After a few years, I found myself having a much harder time getting out of bed in the morning- in a way I never did when I was writing software every day.

But that's me- some people absolutely thrive in this kind of dynamic, unpredictable environment, and find life in front of a text editor to be much more tedious.

If you're an IC faced with the opportunity to move into management, and you're not sure what to do, I'd encourage you to not only try it, but really dedicate yourself to some deliberate study of the craft for a while. Even if you decide it's not for you, the experience will make you a much more effective IC.

Since I've been primarily working in platform and infrastructure recently, a lot the "code" I've been writing is domain-specific configuration languages (e.g. terraform, helm charts, OTel collector config, Kyverno policies, etc). Despite so many glowing testimonies of massive wins with generative AI dev tools, the results from my first few attempts to use them were a bit underwhelming. My guess is that config code usually doesn't get open sourced, so the models just don't get a lot of good examples to train on.

This has left me hankering to do something more ambitious, and probably in a general purpose language, where I can get a better sense of what the tools can do (beyond fairly mundane auto-completion).

Then recently, I had a few days where a lot of my colleagues were off on vacation (school vacation in Massachusetts), and it occurred to me I'd been quietly stewing on a problem that might be a great candidate for my own little hackathon.

Aggravation is the best inspiration​

The problem I'd been pondering:

Load balancer metrics are incredibly valuable, and CloudWatch- the only way to get them in AWS- is a spectacular pain in the ass to use. As a result, load balancer metrics have been seriously underutilized. I wanted to find a way to improve the developer experience for load balancer metrics, and make it possible to scaffold out great Grafana dashboard panels and alert rules for them.

Why are load balancer metrics so important?​

Our applications are already instrumented with OpenTelemetry for traces and metrics, and are emitting structured logs. We've already got visibility into request counts, latency distributions, error rates, as well as all kinds of other telemetry for their application-specific operations. With all this rich data, why would we also need metrics from our load balancers?

Regardless of what your own application self-reports, the thing that ultimately matters most is how your service is perceived by your users (e.g. actual customers, or other services calling your API as a client). Load balancers are the actual interface users interact with, and there's a number of cases where if you were only to depend on self-reported telemetry from pods, you'd be profoundly misled about your service's reliability.

You don't mind a few thousand 502s on every deployment, do you?​

For example: We use the aws-load-balancer-controller to create AWS load balancers from Kubernetes Ingresses and Services. There's a not-so-well-documented footgun with this controller, in which a pod will continue to receive requests from the load balancer for up to 20 seconds after it receives a SIGTERM signal (which Kubernetes uses to tell it to shut down). This is because, after the aws-load-balancer-controller notifies the target group that a pod is terminating, the load balancer's target groups take a few seconds to update, and during this time they continue sending traffic to the pod's IP- which no longer exists. This results in 502 or 504 responses to the client.

Whomp whomp.

Note that in this case, there's no pod to report an error- the only evidence of the problem is the load balancer's own metrics- which is how we discovered this problem shortly after we started using the aws-load-balancer-controller a few years back.

There's plenty of other edge cases with Kubernetes and load balancers where requests will fail at the load balancer, and never make it to a pod- and it's really important to have visibility into these.

CloudWatch metrics are so freaking hard to use​

AWS load balancer metrics are available through CloudWatch (either through AWS's console or APIs), and we use Grafana for our dashboards and alerts, so we've been using Grafana's CloudWatch datasource to pull them in. This has been a generally frustrating developer experience.

Probably the biggest single problem is that CloudWatch queries for load balancer metrics require you supply a "Dimension" parameter called LoadBalancer, which is a portion of the load balancer's ARN (e.g. app/scrumulator/2ab15bf8abef2f4c). This contains an ID that's randomly generated by AWS- it's not something you can set. So when you create an ingress using the aws-load-balancer-controller, you have to wait for the controller to create the load balancer, and only then can you use AWS console or APIs to get the ARN.

Now, if you want to build a Grafana dashboard with this, you need to hard-code this LoadBalancer ID into your dashboard's configuration. And since we don't know the ID until after the load balancer is created, we can't scaffold out a dashboard or alert rules until the service is live in production. Yuck.

And since you probably want your dashboards parameterized so they can display metrics for multiple environments, you'd need to create a specially-formatted Grafana variable to map the environment to load balancer ID. This is possible, but isn't particularly straightforward.

Don't worry- it gets worse. The CloudWatch datasource for Grafana has 2 completely different types of queries: "Metrics query" and "Metrics Insights", as well as 2 different editing modes "Builder" and "Code". This creates 4 different modes for writing queries, each with their own quirks and sharp edges- including how they work with Grafana variables. If you're lucky enough to find documentation or examples for one of these modes, it likely won't work at all with the others.

Oh, and did you want to be able to use a Grafana variable to switch the AWS region? Well that requires a totally different mechanism to parameterize, since the datasource itself is configured with a static region.

While all these problems can be worked around, the overall complexity, fragility, and inflexibility of this mess was causing teams to avoid using their load balancer metrics. Not good.

The elevator pitch​

Here was my idea:

I'd build a Kubernetes controller that would discover all the load balancers owned by Ingresses/Services in our clusters, scrape their CloudWatch metrics, and convert them into Prometheus metrics.

The big win here would be that I could control what labels to put on the Prometheus metrics- so I could create a load_balancer_name label with the load balancer name tag, which is deterministic, and specified in the Ingress. This would make building panels/alerts just as easy for load balancer metrics as it is for application metrics.

We could also add labels like job, namespace, env, region, and cluster, that matched the labels of the service that owned the load balancer. This would allow our developers to select metrics the same way they do for their application's own OTel metrics.

For example, let's say I created a Kubernetes Ingress and used an annotation from the aws-load-balancer-controller to set the name:

annotations:
  alb.ingress.kubernetes.io/load-balancer-name: scrumulator

Then, to get 5xx errors per minute, a fully parameterized PromQL query would look like:

alb_http_code_elb_5xx_count_sum{
    load_balancer_name="scrumulator",
    env="$env",
    region="$region"
}

This would be ridiculously easy to use in comparison to doing it with CloudWatch, plus I'd be able to create simple templates to scaffold out Grafana panels and alert rules for these metrics.

Baby's first Kubernetes controller​

I'd never built a Kubernetes controller before, so I spent a few minutes digesting the Operator SDK documentation to get an idea of how it could work. The Operator SDK has a scaffolding tool, so I used that to create the shell of a basic controller that watches for changes to Ingresses in a cluster, added some dumb logging, and deployed it to a test cluster. After a little fiddling with RBAC and IAM integration, I had it logging each modification to any Ingress in the cluster.

Scraping from CloudWatch​

One of the biggest sources of uncertainty was around cost implications of scraping CloudWatch metrics, so I spent some extra time up-front making sure I understood the pricing model, and how to use the APIs in the optimal way (e.g. batching requests at the right size, keeping total returned datapoints under the right limits).

I spent the next few hours jumping between CloudWatch documentation and cajoling Github Copilot (alternating between a few different models) to write me some Go to grab metrics via the AWS CloudWatch SDK. It actually did a pretty great job on this. It needed a little feedback along the way, but for the most part, I did this almost entirely via prompts, and managed to avoid the temptation to just tweak the code by hand.

Exposing Prometheus metrics​

After I got it logging CloudWatch data points for a sample app's load balancer, I started looking into how to expose these as Prometheus metrics. I quickly discovered that the Prometheus Go SDK's basic interfaces aren't designed for one of my fundamental requirements: setting the timestamp of the metrics to match the time of the CloudWatch datapoint.

I spent a bit of time figuring out how to implement the Prometheus SDK's Collector interface, which allow's specifying the timestamp of a datapoint explicitly.

I got it working roughly before I realized that this had some fundamental limitations:

• With a Prometheus client, the pull model introduces an additional time interval to data that's already delayed by a couple minutes by the CloudWatch API. It also adds unpredictability around the actual delay due to the alignment of the Prometheus scrape interval with my CloudWatch scrape interval.
• The Prometheus text exposition format doesn't have a way to specify multiple datapoints (with different timestamps) for the same metric/labels combination. This means I couldn't "backfill" metrics with older timestamps once I had a newer one.

Switching to OpenTelemetry metrics​

I'm pretty familiar with the OpenTelemetry metrics model, so I figured I'd see if pushing metric datapoints via OTLP would overcome the limitations of the Prometheus client's pull model.

Again, I found I couldn't use the OTel Go SDK's primary interface, since it also doesn't allow setting the timestamp of a metric explicitly. Once again, I dropped down to the lower-level OTLP APIs, and it didn't take long before I was shipping metrics to our Grafana-hosted Prometheus backend.

I built a dashboard that allowed me to compare the underlying CloudWatch metrics with the scraped Prometheus metrics, which helped me work through a bunch of edge cases and quirks of CloudWatch APIs.

Rubber, meet road​

At this point, I was fairly certain this approach was going to work, so I spent another hour or two cleaning it up before I started showing it to some folks to see if they thought it was useful... which they did, because CloudWatch is objectively terrible.

Skipping ahead a few more days, I'd polished it up, added support for Services and NLBs, done a bunch of refactoring and testing, added some self-telemetry, and deployed it to a couple pre-production clusters. After some more refinement, it went out to all our clusters in production.

Within another couple days, we have dozens of Grafana dashboards and alert rules using these new metrics. Within the first day of using the metrics, I found an error with the readiness gates on one of our internal services that was causing 5XXs on pod terminations.

Great success!

Reflections on using gen AI for something non-trivial​

While the LLMs I used weren't much help in writing code against lower-level Prometheus or OTel APIs, it was tremendously helpful in speeding me up a bunch of other ways.

Writing code against widely used APIs​

Writing basic data manipulation code for things like the CloudWatch API response schema isn't exactly hard, but it can be time consuming. With Copilot, I barely had to look at the schema documentation at all; the LLM got this almost entirely right.

Implementing algorithms and data structures​

Once I had a well written prompt, the LLMs did really great at writing the boring plumbing code to manage metadata (e.g. mapping CloudWatch metric names and labels to Prometheus versions). It initially wrote some mildly inefficient code, and needed a little high-level feedback to optimize some data structures, but ultimately did the bulk of this work for me- with the bulk of the prompts being closer to business requirements than implementation details.

Suggesting high-level architectural approaches​

I don't have a ton of Go experience, so it took me a while to land on a concurrency model that worked well for this use case (something like an actor model). Once I realized I didn't like my first approach, and came up with a good high-level description of what I wanted, the LLM was able to produce the bones of an actor model implementation that ended up being exactly what I wanted.

Refactoring​

While iterating through various concurrency models, and also with the switch to the OTel SDK, the tooling was incredibly useful for refactoring. This is where, even without integration with the Go language server, Copilot's agent mode saved a huge amount of time on tasks that would otherwise have been pretty mechanical and tedious.

Building features- with the right prompt granularity​

The optimal granularity I've found for prompts is to describe relatively small, but complete features- like a single, focused user story. When I asked the models to do more than that at a time, I ended up wasting a lot of cycles trying to understand what it did, and it was more likely to diverge from my intent.

When I requested relatively bite-sized features, the changes were much more obvious, easier to verify, and more often then not- correct.

Courage boost​

I think the biggest advantage gen AI gave me here was the courage to try something that might have been a bit too ambitious otherwise. I certainly could have built this controller without it, but honestly it probably would have doubled or tripled the effort. I would have been much less likely to attempt this within a time-box of a few days.

Next up​

This experience has got me pumped to double down and get more ambitious. I've got a number of things I want to try next:

• I'd like to see if it's possible to use LLMs to generate integration tests for a legacy system that has so far resisted efforts to be refactored. I'm wondering if an LLM would be capable of finding all the subtle, implicit behaviors that aren't actually visible in the API, but still function like bubblegum holding the larger system together. If we could use it to generate really comprehensive, working tests, it might then give us the confidence to be a lot more aggressive about rearchitecting it.
• There's a number of OSS projects I've wanted to contribute to, but the effort required to understand their architecture has deterred me. I'd like to try using LLMs as a tool to accelerate my understanding of their structure; not as much as a tool to write code, but just to comprehend it.

Anyone interested in a slightly used k8s controller?​

If anyone is interested in using the controller I built, I'm considering open sourcing it, and I'd love to hear from you. Please reach out!

I spend quite a bit of time helping developers figure out how to make sense of telemetry data they've collected. Often this is showing them to navigate our observability tooling, how to think about visualizing data, or just tossing some saucy query tricks into the mix.

Sometimes though, I find that their telemetry is... ill-conceived. I occasionally find metrics being written like they were a point-in-time event, missing context in logs because someone was worried about adding too much cardinality, or log lines with variable and static context needlessly mashed together. Alas.

This kind of thing makes me wish I'd had a chance catch them earlier, when they were just beginning to consider instrumentation choices.

Given how often this happens, I've built up some insights on some of the most common misconceptions about the different observability signals, what makes them different from one another, and why you'd choose a specific one for a given situation.

The basics: observability signals (a.k.a. the "three pillars")​

Let's review: what do we mean by "observability signals"? We're talking about the venerable "three pillars" of observability: logs, metrics, and traces.

• Logs are streams of text that usually represent individual events. Logs are most powerful when they're structured (e.g. JSON or otherwise parsable); they can be arbitrarily "wide" and can have many, high-cardinality fields (no limit on the number of possible values per field).
• Metrics are series of numeric measurements over time, labeled with key/value pairs. Metrics represent aggregations of data (e.g. sums, counts, percentiles) and not individual events. Time-series databases store metrics in an extremely compact and cost-effective way, with the tradeoff that they require relatively low cardinality value labels (a low number of unique combinations of labels) to operate efficiently.
• Traces are a special kind of logs that are rigidly structured to provide context about a set of related operations that occur across a (usually distributed) system. Individual events in traces are called "spans", which represent operations with a start/end time, and are linked together in a nested structure via correlating IDs. Spans can also contain arbitrarily wide, high-cardinality fields.

Enough with the theory already​

Blah, blah, blah. I'm dying for a real-world example. How about you?

Introducing PetFace​

Let's say we've got a startup idea for a mobile app: it allows users to select pictures of their pets, upload a video, and then some fancy AI processes them and substitutes the faces of anyone in the video with the pets' faces. Hilarity ensues.

Let's call this app "PetFace".

The architecture of PetFace​

For the supporting backend of this app, let's build a little API server which does the following:

• receives pet pictures and a video from the mobile app via an API
• runs the fancy generative AI to replace human faces with pet faces
• writes the processed video to S3
• updates a database record to indicate the video processing is complete

The app would poll the API server to check the status of the video processing.

Making PetFace observable​

Let's start with some nice, structured logs​

OK, let's say we've got the MVP of the app done, and now we want to set up some observability.

99% of the time, the first thing to do is just sprinkle in some logging. Here's an example of some nice, structured log events:

<ts> 
  level=info
  event=HttpRequest
  message="POST /v1/video"
  status=201
  request_id=abcdefg 
  pet_pics_count=3 
  duration_ms=7830 
  user_id=12345 
  instance=api-3
<ts> 
  level=info
  event=VideoReceived
  message="video uploaded successfully" 
  request_id=abcdefg 
  pet_pics_count=3 
  duration_ms=7830 
  user_id=12345 
  instance=api-3
<ts> 
  level=info
  event=VideoProcessed
  message="video processed successfully" 
  request_id=abcdefg
  duration_ms=13204
  user_id=12345 
  instance=api-3

We'll also need some additional event types (e.g. "VideoSaved", "DatabaseUpdated") and probably also a message type for errors:

<ts> 
  level=error
  event=VideoProcessingError
  message="video processing failed: too many pet photos" 
  request_id=abcdefg 
  user_id=12345 
  instance=api-3 
  error=ETOOMANYPETS

As we build, test, and troubleshoot the PetFace backend, these log messages will be invaluable for understanding what's going on for any individual request.

Deriving metrics from logs​

As PetFace's usage begins to grow, scanning raw log lines with our eyeballs to try to determine the health of the backend isn't going to cut it. To reason about large amounts of data, we're going to want to use the magic of math to extract aggregate, numeric time-series we can visualize.

In this case, we may want to start drawing graphs to represent:

• How many requests are we getting per second?
• What are the average, 95th percentile, and 99th percentile of video processing time?
• What is the percentage of requests that result in an error?

Luckily, our logs already contain all the data we need in structured fields. If you've got a log query language that supports grouping and some aggregation functions, you can generate time-series from your logs.

Here's an example: Here we're using LogQL (the query language for Loki) to get requests per second with LogQL. First, here's a basic log selector expression that gets the request logs for PetFace in production:

# get production logs for PetFace
{service_name="PetFace",env="production"}
  # parses the key/value pairs
  | logfmt 
  # select only HTTP request logs
  | event="HttpRequest" 

Now we can convert it into a single metric series by wrapping it in rate() and sum() functions to calculate requests per second over time:

sum(
  rate(
    {service_name="PetFace",env="production"} 
      | logfmt 
      | event="HttpRequest" 
      [$__auto]
  )
)

Voila! Now we can use our visual cortex (a.k.a. the GPU inside our brain) to comprehend tons more data than we could possibly hope to by scanning through log lines! We can spot trends, anomalous blips, and build an intuitive sense of what healthy vs unhealthy looks like for our app.

Logs get expensive at scale​

PetFace is blowing up! We're scaling our backend dramatically, and our bill for logs storage is killing us. Logs are super powerful and flexible, but they cost a lot; not just for storage, but also for our code to generate, serialize, buffer, write to stdout, and ultimately harvest and transmit to our observability backend.

But we're using the logs to generate metrics that have become really, really valuable. We've using them to drive alerts to tell us when something has gone off the rails, to project our growth, and to tune our system. We can't live without them.

Do metrics aggregation directly in the app​

What if, instead of generating the metrics from the logs, we just do the metrics aggregation directly in our app? This would not only save the cost of the logs storage, but also remove all the overhead of generating the logs from the app.

For example: to track request rate per second, we could just have an integer counter that we increment on each request, and then send the sum total to a timeseries database at a regular interval (e.g. 1 minute).

Timeseries databases are incredibly cheap to run compared to log aggregators. If we are collecting metrics directly, we can decrease the log level (i.e. change a setting to only emit log lines with level=warning or above), and save a ton of money on log storage, without losing these critical metrics.

How to use metrics completely wrong​

Let's say we've hired a bright-eyed young college intern at PetFace, and have asked them to implement a new feature: a button which allows a user to share their pet-faced videos with their friends by email.

This intern is super talented, and builds the feature in no time. The final requirement is to add some observability to track adoption of the new feature, so they add a new line of code which gets run once a user clicks the "send" button.

The intern figures that this metric will be most useful if it contains the full context of the email sending function:

def send_video_to_friend(user_id, video_id, friend_email):
    # build content for an email to the user's friend
    email_body = build_email_body(user_id, video_id)

    # Send the email
    result = send_email(email_body, friend_email)

    metrics.increment_counter(
      name="petface_friend_email_sent", 
      labels={
        "user_id": user_id,
        "video_id": video_id,
        "friend_email": friend_email,
        "success": result.success
      }
    )

I've mentioned previously that metric labels should be low cardinality. In this example, we've introduced very high cardinality: there's going to be a new, unique metric series for every single combination of user_id, video_id, and friend_email, which all have unbounded possible values.

Egads! That's effectively infinite cardinality, which will rapidly bring any time-series database (e.g. Prometheus, InfluxDB) to its knees- or cause your observability vendor bill to explode.

Metrics defined in code are for answering questions about aggregate quantities, not individual events. Here's some questions we could answer effectively with metrics:

• How many video emails are being sent per minute?
• What percentage of email sending attempts fail?

To answer these, we don't need user_id, video_id, or friend_email; we only need to count the number of sent emails, including the label success, which has only 2 values: true or false. This results in only 2 series. That's wicked cheap!

Since video sharing emails are a brand new feature, we may not have any idea what questions we want to answer up front. We may want to a series of more ad-hoc, open-ended questions like:

• Which users send the most emails? Does it correlate to their subscription tier? Country of origin? Length of the video?
• Which users are associated with the highest send failure rates? What could be causing that?

This would have been an excellent usage of a log line. You can include all kinds of context in the log line, and then use your observability backend tools to slice and dice the data to answer a much broader range of questions compared to the metric.

Here's the same code, after a review from a seasoned mentor:

def send_video_to_friend(user_id, video_id, friend_email):
    # build content for an email to the user's friend
    email_body = build_email_body(user_id, video_id)

    # Send the email
    result = send_email(email_body, friend_email)

    metrics.increment_counter(
      name="petface_friend_email_sent",
      labels={
        "success": result.success
      }
    )

    logger.info(
      event="FriendEmailSent",
      message="friend email sent",
      labels={
        "user_id": user_id,
        "video_id": video_id,
        "friend_email": friend_email,
        "success": result.success
      }
    )

Did you forget about traces?​

Up to this point, PetFace was essentially a single, horizontally scaling backend API server. While traces can be used in a monolithic app (e.g. to visualize key functions in the call stack), their utility is more limited.

Let's imagine PetFace has gone absolutely gangbusters, had a successful IPO, and now has 15 different dev teams working on dozens of different supporting microservices: advertising placements, data collection for model training, social networking features, etc. We've even got yoga classes and catered lunches two days a week!

At this scale, system failures are a lot harder to diagnose with logs or metrics. There's all kinds of complex, second-order effects, backpressure, and subtle failure modes across multiple services in the system.

For example, when the video processing microservice's p95 performance suddenly takes a nosedive, everyone starts scrambling around trying to figure out the root cause, but it's hard to get a broad view of the whole system when each service's logs and metrics are separate.

This is where tracing absolutely shines.

Let's take the example above, but imagine we have our system instrumented with OpenTelemetry tracing, pointing to a backend like Honeycomb or Grafana Tempo. These tools allow someone to quickly identify a few example traces representing the slow video processing requests, drill down into the spans in the trace- across all microservices transitively involved in the operation. Spans are visualized as a tree, where the width of each span represents duration, and child operations are nested recursively underneath.

Behold, the power of the correlation ID!

Now you can easily see that, 3 services deep, there's a slow PostgreSQL request from the ad placement service. Oh crap- it looks like there's lock contention in the database during high load!

Wow- traces are amazing. I mentioned before that the spans that make up traces are basically just a special, rich type of log, and that metrics can be derived from logs. So if we have tracing... why would we want to use any other signal?

The 3 signals are complementary​

There's a few reasons why the 3 different signals are complementary:

• Traces have roughly the same unit economics that logs do- they can get very expensive at scale. To make traces more affordable, many companies will do random sampling of traces (e.g. collect 1 out of every 5 traces), which will reduce the cost significantly, but also make the traces a lot less useful for forensic use cases (e.g. diagnosing a particular bad request). Sampling is also complex, and can be error prone.
• There's a bunch of use cases that traces don't do a good job of handling today (e.g. long running operations, async operations, capturing state from a crash, representing background processes)
• Traces are a lot newer and less mature than logs and metrics; the tooling ecosystems are more limited.

Cheat sheet​

I'll leave you with some quick heuristics to help you match signals to use cases:

When to use logs​

• When you're building a new application. Start with structured logs, and log liberally.
• Forever after. Logs are always useful to help ask arbitrary, novel questions of your system.
• When you need the ability to retroactively troubleshoot specific, individual events.
• When you have strict requirements for auditing, security, or regulatory compliance.
• For edge cases that traces don't handle well: capturing crash data, initialization and background processes, correlation of related async tasks, etc

When to use metrics​

• To track important numeric indicators of system or business health over time (e.g. request counts, error counts, CPU temperature, WiFi signal strength, request duration distributions) that you probably want to alert on (e.g. request rate drops, error rate jumps)
• When your log volume starts to get expensive, and your system is mature/stable enough to reduce the verbosity/frequency of your logs, and you can fill in the resulting gaps with some key time-series

When to use traces​

• When your system grows more complex, especially when you break it up into multiple services (or APIs, databases, etc)
• To provide a deep understanding the behavior and topology of your (usually distributed) system, including the ability to ask ad-hoc, arbitrary questions
• To help understand the performance characteristics of your system, and especially to find the root cause of performance problems

General tips​

• If you've already got sufficient visibility into a specific operation from one signal, and another signal would be duplicative, you may want to drop the one that's less rich (e.g. maybe drop a log line if you're already capturing an un-sampled span).
• Don't prematurely optimize for cost; lean on richer signals (traces, logs) and avoid sampling until it becomes an actual problem. The ability to ask ad-hoc questions of your system is very important, especially early in a product's lifecycle.

Now that OpenTelemetry has gained such significant traction, it's starting to attract a lot of attention beyond the hardcore observability community. While most of what I read about OTel is pretty positive, it also draws its fair share of shade.

Honestly, I get it. As a big user of OTel, I've spent plenty of hours rage debugging OTTL filters in OTel collectors, desperately searching for SDK examples that actually work, or pulling my hair out trying to figure out which version of a protobuf schema changed and broke telemetry from my Swift clients. And like a lot of the haters, I also get frustrated that the overall level of complexity in the specifications leaks into the implementation details of every part of the project.

All that said, I'm incredibly thankful that OTel exists. While it's still in its awkward teenage years, it's already changing the industry dramatically. Despite the challenges that currently exist, I think OpenTelemetry is the only reasonable path forward for observability.

So, like, I totally already know what OpenTelemetry is, but just give me a little refresher​

OpenTelemetry is an umbrella project with the broad and audacious mission to provide a set of open tools and standards for building telemetry pipelines.

The before times​

Before OTel, the tools available were mostly proprietary and vendor-specific. This has some big downsides:

• Black boxes: Telemetry agents/libraries/SDKs (the thing you install into your app to collect metrics, logs, etc) were closed source, not modifiable or verifiable by users. If they didn't work exactly as you need, you could really only hope to persuade your vendor to add a feature for you.
• Vendor lock-in: Since most telemetry agents are vendor-specific, switching between them can be very labor intensive, which gives vendors enormous leverage to develop predatory, rent-seeking pricing strategies

OK, so what actually is OpenTelemetry?​

OpenTelemetry provides an alternative ecosystem, starting with some open specifications:

• A standard telemetry protocol (OTLP) that defines the low-level shape of telemetry data (including traces, metrics, and logs) and how it gets transmitted
• A lightweight, standardized API that can be used to add instrumentation to your code
• SDKs that implement the API and configure your applications to export telemetry data
• Semantic conventions that define the meaning of fields in telemetry data

And then, more practically for users, there's some specific software:

• Language-specific SDKs: A set of (mostly language-specific) SDKs that implement the SDK and API specifications. This is what you'd install into your apps (as an alternative to a vendor-specific agent). You'd use the APIs to sprinkle manual instrumentation throughout your first-party code, and framework/library authors would use the API to add it to theirs.
• Auto-instrumentation: A vast ecosystem of auto-instrumentation libraries and tools that automatically instrument all the most popular frameworks (e.g. Django, Express, Spring etc) and libraries (http, Postgres, Redis, Mongo, etc), using the API under the hood.
• The OpenTelemetry Collector: A server application that's a sort of "universal translator" for telemetry data. It can accept telemetry in virtually any format you can imagine (e.g. statsd, Prometheus, Zipkin, InfluxDB, Loki, etc), apply arbitrary transformations, filters, and enrichments, and then export the data to an equally astonishing number of destinations formats (e.g. Jaeger, Prometheus, Splunk, Honeycomb, DataDog, Kafka, ClickHouse, etc).
• The OTel ecosystem: A huge ecosystem of tooling (e.g. Kubernetes operators, helm charts, eBPF agents) that compose the SDKs, collector, protocols, etc, to create new capabilities.

And the part of OTel that astonishes me the most:

• Vendor support: Basically every observability vendor you can think of now supports OTel.

Wait, why are all the vendors supporting OpenTelemetry?​

The most incredible thing about OpenTelemetry is that it even exists at all, much less that it's supported by virtually all observability vendors. Why would they voluntarily back a project that promotes vendor-neutrality, and undermines their ability to lock-in customers?

Early on, there was certainly an advantage to smaller vendors adopting OTel as a competitive differentiator. What's surprising is how much pressure this put on the industry, which slowly moved up-market, like a growing tsunami, until even the most dominant vendors felt pressure to offer at least some support (or lip-service for support).

The actual threat of OpenTelemetry​

Before OTel, the dominant vendors had for years enjoyed an advantage that wasn't particularly obvious: their end-to-end ownership of the telemetry pipeline gave them full understanding of the meaning of the data they collected.

If you've ever used one of the big tools, like DataDog or New Relic, you've probably experienced the customer experience benefits of this capability first hand:

1. Install the vendor's proprietary agent into your apps (directly, or through some platform-wide integration)
2. Open up the vendor's dashboard, and instantly see gorgeous dashboards with all kinds of pre-built visualizations and alerts, all out of the box. Revel in the insights and ease of use.

The obvious part is that OTel threatens this advantage by giving us the ability to create our own telemetry pipelines. But the real kicker, hidden in the list of stuff above that OTel does, is what I think the most powerful and underappreciated part of the project:

• Semantic conventions that define the meaning of fields in telemetry data

Why is this item so important?

Semantic conventions are what made the dominant observability tools so damn useful​

When you open that first dashboard in DataDog or NewRelic, you feel like you're standing in the Pentagon war room or something. There's so many graphs with data you've been dying to visualize- especially about your infrastructure- that were clearly very carefully curated. Your Kubernetes clusters are suddenly transparent and stripped of all their mystery. Within just a few seconds you can get a broad sense of the state of your system, and then drill down with a few clicks to find individual logs representing anomalies. The tooling gives you the ability to correlate data from different signals, allowing you to jump back and forth between aggregate performance graphs and individual exemplar pod data.

The dominant, legacy vendors have been able to create this user experience because they've utilized their end-to-end ownership of the telemetry pipeline to establish (proprietary) conventions about the meaning of the data they're collecting.

• They scoop up all that rich Kubernetes and cloud provider metadata (pods, nodes, namespaces, EC2 instance type, region, AMI ID, etc) and use it to enrich metrics and log lines with labels
• They collect performance data using specific conventions and units
• They auto-instrument your applications using language-specific instrumentation agents that know how to instrument all the most popular libraries and frameworks, using consistent metadata across all of them
• They track the lifecycle of requests through your system, and correlate signals using identifiers sent through as (e.g.) custom HTTP headers

By the time the telemetry data gets to their system, they already know an enormous amount about what it all actually means. They can then use this to create higher-level visualizations which understand all that Kubernetes and cloud provider metadata, supply canned alerts that handle a bunch of common Kubernetes failure modes, or build maps of services that show all the runtime relationships by using the correlation IDs they injected into your HTTP requests.

This is why I think of OpenTelemetry's semantic conventions as a secret weapon. Now all observability vendors have the ability to go beyond just being able to ingest data from an OTel-based pipeline; they can now add this kind of rich, powerful, out-of-the-box user experience powered by a deep understanding of the meaning of the data they're collecting.

And increasingly, they are!

It's not perfect, but it's already better than the alternative​

So when I read posts criticizing OpenTelemetry for its complexity, or its lack of maturity, or claiming that it's designed by committee, or it's hindered by having to satisfy the lowest common denominator, I can simultaneously agree with all that, while still believing that it's still the best way forward.

Like most technical choices, this is fundamentally about tradeoffs, and I think OpenTelemetry has hit the inflection point where its benefits have begun to outweigh its pain points for many, many situations. I'm acutely aware that there's still a lot of friction in getting OTel set up, but you've got to compare it to the alternatives.

OpenTelemetry has finally become a workable enough solution that allows an organization, with enough effort, to develop a top shelf observability capability - without having to trade-in their firstborn child in exchange for, say, a few months of custom metrics.

And OTel is only going to get better. So many brilliant people are contributing, vendors are investing heavily, the ecosystem is growing, and adoption is surging.

An OTel anecdote that brings me joy​

I'd like to leave you with one of my favorite moments in OpenTelemetry from the last year:

The OpenTelemetry community implemented a DataDog receiver for the OTel collector. When I say the community, I'm saying it specifically did NOT come from DataDog; it was driven by community members and competing vendors (e.g. Grafana Labs, Splunk). This receiver, along with some supporting processors, allows organizations that are stuck using DataDog's proprietary agents to redirect their telemetry data to any vendor that supports OTel.

Imagine how DataDog sales felt the first time this came up in a contract negotiation!

At SimpliSafe, we manage a pretty large system of microservices. Because we're entrusted by our customers to protect their homes and families, we take reliability of our systems pretty seriously, so observability is pretty important to us.

Like most companies today, our observability strategy is built around the "three pillars" approach: metrics, logs, and traces. Of the three, we're currently the most dissatisfied with our logging tooling, and have been working on finding a better product.

We're already a Honeycomb customer, and in our conversation with them, they ended up making a pretty interesting case that we should consider a new approach: ditch the three pillars and make traces the center of our strategy. We had up to this point been thinking very incrementally, and here was Honeycomb, coming in hot with a bold and revolutionary vision: Observability 2.0.

WTF is Observability 2.0?​

The term "Observability 2.0" was coined by Charity Majors (the CTO and co-founder of Honeycomb) as a shorthand for a new vision of observability, defined in opposition to the "three pillars" model (metrics, logs, and traces).

Here's the gist of Observability 2.0:

• Collect telemetry in the form "wide events" (usually traces, but also maybe structured logs) which can contain an arbitrary number of key/value pairs with high cardinality values
• Your observability tooling should allow you to query these events to answer arbitrarily complex questions about your system

In this vision, traditional metrics (i.e. timeseries emitted directly from your apps) are considerably less valuable, because they, by definition, are stripped of context when they're aggregated into timeseries. Since cardinality is the main driver of cost in a timeseries database, you have to choose which attributes you care about (ensuring they're low/moderate cardinality), and drop the rest. This means you also have to know what questions you'll want to ask of your system in advance of deploying the code.

With wide events, on the other hand, you keep all that rich, high-cardinality data, and then you get to slice and dice it in ways you couldn't have predicted you'd need in advance. You can use wide events to derive new metrics on the fly, and use the rich attributes to dig into specific anomalies in ways that are impractical with pre-aggregated timeseries data.

Furthermore, in Observability 2.0, logs become somewhat redundant with traces, too. Traces are an ideal superset of logs: structured events, but with additional conventions around how they represent units of work and their relationships (i.e. spans with durations, parents, siblings, and children).

This is a pretty compelling vision: one universal source of truth for all your observability data. You don't have to suffer the cost of 3 pillars when you can derive all the same value from wide events, and you get to simplify your whole practice around a smaller, more powerful set of tools.

Rock and roll! This sounds amazing! Sign me up!

The gap between vision and reality​

To be fair, the idea of Observability 2.0 is something my colleagues at SimpliSafe and I been thinking about for a while now. Charity has written and spoken a whole lot about it, and because of the elegance of the idea (and her fantastic writing), we already understood the core ideas, and had experience with Honeycomb, so we were primed to consider something a bit more forward thinking.

When Honeycomb came back to us with a concrete plan, we realized it was finally time to take off our incrementalist hats and start considering what it would really mean to go all in on traces as the source of truth.

It didn't take long for us to start enumerating concerns.

In defense of traditional metrics​

I may have oversimplified the argument against metrics; Charity's actual words are a bit nuanced. For instance, she still sees value in using metrics to monitor infrastructure (especially third party infra). It's just that for the code that you're writing yourself- at the core of your business- and is changing constantly, metrics alone aren't sufficient to understand what's going on.

This is absolutely true. Metrics alone are insufficient to really understand the behavior of your system. You absolutely need high-cardinality data such as logs and traces to answer many types of questions.

But I would argue that traditional metrics are still necessary as a complement to wide events due to the huge and fundamental cost discrepancy between the two.

Let's talk about cost​

The reason we still need metrics comes down to the enormous discrepancy in cost between metrics vs wide events. Metrics are extraordinarily cheap. Like a fraction of the cost of logs or traces, and the comparison gets more and more one-sided as you scale up.

• Timeseries metrics are incredibly compact and lightweight. Cost is driven by the number of timeseries you're producing: unique combinations of label/value pairs (e.g. pod, request_path, response_code). These pairs get stored once, and then the rest of the cost is basically just a number stored every interval (30 seconds or so). The tradeoff is that you lose context of the individual events from which the aggregate measurement was derived.
• Wide events are fundamentally a more heavyweight signal than metrics, since each event is basically its own bag of string attributes. Vendors generally charge for each span/event (or by number of bytes) ingested. You also pay a cost at runtime: there's more data to generate, encode, buffer, and transmit.

Let's compare costs as you scale​

As the frequency/volume of events increases, the fundamental overhead of wide events grows increasingly impractical in terms of computation, network, and storage... and ultimately dollars. This is for three fundamental reasons:

• metrics are aggregated into a single value over an interval, whereas wide events are not
• metric labels are stored only once per long-running series, unlike wide events, in which all attribute values are stored in full for every event
• wide events, by design, carry a lot more data (all the high-cardinality goodness you don't want on your metrics)

For example: imagine you have 10 pods which each handle 100 requests per second. You can instrument this with a single histogram metric, at the cost of a few active series (lets say 10 for the sake of argument) per pod. With traces, you're producing 100 spans per second, per pod (one per request).

• With metrics, you have 100 active timeseries. Using list pricing for Grafana Cloud of $8 for 1K series, this will cost 80 cents per month.
• With traces, you're producing 1000 spans per second. Using list pricing for Honeycomb of 100M events for $130, this will cost ~$3300 per month.

Now let's imagine you find a bottleneck, and optimize this code to be 10x faster. Now each of your 10 pods can handle 1000 requests per second. Awesome! Let's scale up our marketing spend and 10x traffic:

• With metrics, you'll still have 100 active timeseries, at 80 cents per month.
• With traces, you're now producing 10,000 spans per second, which will cost ~$33K per month.

I wasn't kidding, right? And this is only the cost from Honeycomb; you're also paying the cost in terms of runtime overhead for creating all those spans.

Having the ability to choose metrics vs wide events for a given use-case can give you a lot more flexibility to find a good balance between the richness of telemetry data and its cost.

Wait, so how is anybody using wide events?​

You may think the cost scaling example above is contrived (which is true)- you'd probably use a lot more labels on your metrics in practice, which increases the number of series. But calculating the total cardinality you'd generate is complicated, because the interaction between metrics labels produces a sparse matrix; you don't pay for all possible combinations of label values, only the combinations that actually occur.

So let's talk about the real world, specifically telemetry at SimpliSafe. If we were to send all of our traces to Honeycomb, it would be an order of magnitude more expensive than what we're currently paying for all our timeseries metrics.

But wait, you ask: there are plenty of other companies happily using tracing- how are they affording it?

Yep- they're probably doing what we're doing with our traces: sampling.

Parents, have you talked to your kids about sampling traces?​

Sampling is a whole topic in itself; the gist is that you randomly select one out of every N traces, drop the rest, and rely on statistical extrapolation when you run queries. The attributes necessary to do sampling are actually built into the OpenTelemetry spec.

There's some tooling available, which can use these attributes, to do tail sampling (we use Honeycomb's Refinery for this). Honeycomb also does a nice job of extrapolating sampled values at query time based on the sample ratio, so it feels pretty opaque to a user- sometimes you forget you're working with sampled spans.

We're currently sampling at a 30/1 ratio, which, at our volume, is usually sufficient to get accurate enough aggregate measurements.

Sampling is complicated and error prone​

Theres a lot of moving parts required to make sampling work at scale. All it takes is one programmer in one service to make a mistake with an attribute that controls sampling behavior, and you can end up with whole classes of missing traces, or conversely, accidentally disabling sampling and blowing up your bill. We've had both of these happen, and in a couple cases, we've had queries return very misleading results.

Luckily, in these cases, we had traditional metrics available that directly contradicted the wrong conclusions we were drawing from the traces, and saved us from making some pretty bad decisions.

So if you're doing sampling on your wide events, which you almost certainly are to manage costs, traditional timeseries metrics are extremely valuable for corroborating results from your queries against wide events.

Sampling breaks forensic use cases​

The other big problem with sampling is that it makes it virtually impossible to rely on traces for forensic/diagnostic use cases. If you're trying to diagnose an issue for a specific customer, client device, or other specific request, the chances of having a trace available is slim.

Gut check: imagine trying to investigate a potential security breach with anything other than 100% of logs/traces. Who would put themselves in that position?

So for these forensic use cases, we end up needing our unsampled logs, and not traces.

The historical inertia of metrics​

I'll finish up my defense of metrics with a couple other eminently practical reasons which make it hard to imagine giving up traditional metrics entirely:

• Metrics are boring, simple, tried and true. Virtually every mature industry uses metrics to drive their business and operations.
• Metrics are ubiquitous. Everything supports metrics. Most every tool in the cloud-native ecosystem exposes Prometheus metrics, and very few emit traces (though this is slowly changing). Cloud providers also expose telemetry primarily as metrics (e.g. CloudWatch, Azure Monitor, etc).

Logs vs traces​

While Charity's Observability 2.0 vision of wide events technically includes structured logs, they generally play second fiddle to traces. Either way, it's worth a quick tangent to enumerate reasons why logs are still valuable:

• Logs are simple and ubiquitous. Everything writes logs. Also all OSS/third-party infra produce logs, and very few produce traces.
• The developer experience for logs is dead simple, and great by default. Use printf(), start your app from a terminal, and watch the logs flow through stdout. It's a beautiful thing.
• Compared to traces, logs are a much more natural way to model events which aren't tied to requests made across services (e.g. lifecycle events, background jobs, etc).
• Logs are mature and battle-tested. They've been used since the dawn of computing.

Here's a fun example: If you're using traces only, and are not saving logs, where would you look to diagnose an app crashing? OTEL instrumentation isn't going to politely wrap up the current span and flush its buffer when your app panics. You're just going to lose any record of the cause of the panic.

Here's a few more examples we've experienced where the OTEL tracing ecosystem's maturity has bit us:

• With certain node.js Promise libraries, trace context can get lost across async function chains, leading to orphaned spans
• Long-running spans (more than a few seconds) are generally an unsolved problem- the behavior isn't well defined, and spans can get dropped or orphaned.
• It's not currently possible to configure sampling to capture related traces: single logical operations that involve multiple requests, async events, etc.
• Context propagation can get broken across a whole system by one service that's using an older OTEL SDK, listens via an esoteric protocol, or is just a little buggy
• Graceful shutdowns are a lot trickier with OTEL tracing, because you have to first drain your actual connections/queues, then ensure you're flushing all in-progress spans before exiting. Any mistakes: you guessed it, dropped spans.

The path forward for Observability 2.0​

I'm not writing this to dump on either OpenTelemetry, tracing, or the overall vision of Observability 2.0. I'm still a huge fan of the idea of simplifying signals down to minimize telemetry sprawl, and I've experienced the benefits of OTEL, tracing, and wide events firsthand.

Honeycomb is indeed a fucking pleasure to use- it's super fast, very powerful, and intuitive. Honeycomb's mere existence has prompted a virtual tidal wave of innovation across the observability space as competitors struggle to react to its power and elegance.

But despite how seductive the vision of Observability 2.0 is, when it came time to make a bold decision to go all in... we equivocated, put our incrementalist hats back on, and decided to continue with the three pillars for the time being. Observability 2.0 is just a bit too cutting edge at the moment.

The obstacles that prevented us from saying "yes" today may be solvable in time. I hear there are companies that have already gone all in on Observability 2.0, and I've got to believe it's possible.

To close the gap for a company like SimpliSafe, here's the problems we'd need solved:

Affordability​

Unless we can get the cost of wide events down, sampling will continue to be necessary, and sampling undermines the idea that we can discard the three-pillars model:

• Sampling traces makes them useless for diagnostic and forensic use cases, requiring you to retain logs
• The complexity of sampling traces makes it more important to retain traditional metrics to corroborate results

This is a fundamental catch-22: sampling and Observability 2.0 are fundamentally incompatible.

I'm seeing some pretty exciting things happening to make logs/traces more cost-efficient, such as products based on OSS columnar databases like Clickhouse, as well as other tracing projects like Grafana Tempo and QuickWit.

And I'm certainly not going to count out Honeycomb- knowing them, they'll continue to chip away at inefficiencies and find more ways to make their product more and more affordable, especially as they gain greater economies of scale.

Developer experience​

The developer tools available for tracing are legitimately a lot more complex than those available for logs. We'll need ubiquitous tracing developer tools that approach the ease of use of printf() debugging, and that make it just as easy to validate that your tracing instrumentation is working the way you intended.

The developer experience for OTEL SDK configuration is still... a work in progress. I hope to see more projects that package up the OTEL SDKs as a "distribution" to provide an installation experience that's as simple as vendor agents like NewRelic/DataDog's. Let's just admit that 200 lines of boilerplate before you can send a single span is a bit much.

Documentation is also a big opportunity for improvement. I found it impossible to get an SDK properly configured without digging through the source code. We need a lot more examples, and more focus on use-cases.

And please, for the love of god, let's not add another layer of YAML to "simplify" configuration.

Reliability and maturity​

Beyond installation and configuration, the OTEL tracing ecosystem needs some time to ripen, making it easier to get correct behavior (e.g. graceful shutdowns, context propagation, etc), handling edge cases (crashes/panics), and defining some standards for handling things like long-running spans, related traces, and unfinished spans.

Ecosystem and adoption​

As long as most infrastructure and third-party tools are emitting only logs and metrics, it's going to be hard to go all in on tracing for first-party telemetry. It's possible that OpenTelemetry's trajectory will continue, and more and more OSS tools will start emitting traces, but there's a lot of ground to cover to hit critical mass.

As a litmus test: an Observability 2.0 product would need some sort of drop-in Kubernetes infrastructure monitoring solution similar to what you can get out-of-the-box with DataDog/NewRelic, or at least something comparable to the Prometheus Operator, which are currently almost entirely driven by metrics.

Being comfortable with incrementalism​

Throughout my career, I've generally been biased towards incrementalism over revolutionary changes. I'm pretty stingy about my innovation tokens, and I tend to want to save them for things that will drive our core business strategy. Observability is something I want to keep boring and predictable.

Perhaps Observability 2.0 isn't a purist standard we should aim to achieve in black and white terms. Rather, it's a philosophy we can apply incrementally, in which we gradually get more and more value from our observability spend.

Right now, we're thinking about how to better use the three pillars we have:

• Seek out tools that integrate and visualize data from across multiple pillars (e.g. using exemplars to link metrics to traces)
• Look for ways to reduce the waste of overlapping, duplicate pillars
• Try to utilize each pillar for its strengths, with tooling that uses them to complement each other
• Look for more sustainable ways to manage costs

Hopefully in a few years, the ecosystem will have evolved around the Observability 2.0 vision, and we'll be in a position to be a bit braver about our next steps.

AWS's Graviton is a 64 bit ARM-based CPU available on EC2. Why, you ask, would one want to use Graviton-based instances when trusty old x86 instances have served us so well in the past?

Well, for one, you'ds see a roughly 30% improvement in price/performance versus instances with x86 chips. This performance difference is even more pronounced at higher utilization, because unlike x86 chips, a Graviton vCPU is an actual CPU core, NOT a hyperthread you're sharing on a core with some rando. This should allow you to scale down, increase CPU utilization more than would be safe with hyperthreads, and still handle the same load.

While you won't, as an AWS customer, see the corresponding reduction in power consumption and carbon emissions directly, you can be assured it correlates directly with cost savings. You can save money whilst congratulating yourself for being a dedicated protector of the environment. Bonus!

I had a conversation with my team, and we decided that yes, we do enjoy saving money, and that we should probably look at how hard it would be to migrate our workloads to Graviton instances.

How hard could recompiling everything be?​

At SimpliSafe, we've got a heterogenous fleet of microservices built with a bunch of different languages. While we were eager to take advantage of the cost savings of Graviton instances, we weren't about to plunge headlong before we'd incrementally built some confidence with it. We needed an approach that would support both ARM64 and AMD64 for some period, so that we could run a reasonable bake-off.

Another wrinkle: we'd been gradually replacing our developer laptops (Macbooks) with newer models with Apple Silicon (ARM) processors for about a year at this point, so we had a mix of developers using ARM and Intel chips. We knew that within about 2 years, all our developers would be on ARM, as their laptops were refreshed.

We believe strongly that developers need be able to build, test, and debug locally on their laptops, in an environment that's as close as possible to production. Whatever solution we came up with had to work for both x86 and ARM, both in AWS and on developer laptops.

Options for multi-platform builds​

The first problem we had to solve was building cross-platform (or better yet, multi-platform) container images. Here's some of the options we considered:

Docker with QEMU emulation​

Docker has features to enable CPU emulation of ARM chips on x86 hosts, or vice versa. On Linux, it uses QEMU, while on a Mac (using a Linux VM managed by Docker Desktop) it can additionally use Rosetta.

It usually works pretty well, but it's far from perfect. There's a number of cases we ran into, especially with Rust and .NET builds, where the compiler would choke hard (segfault) when run under emulation.

Even if emulation had been reliable, the performance hit when running under emulation varied from slightly annoying (e.g. node.js, Python) to mildly aggravating (Go, .NET), to unbearably slow (Rust, C++).

Cross compilation​

Many modern language compilers support cross-compilation (i.e. you can compile a binary for a different architecture than the one the compiler is running on). We toyed around with this option, but quickly realized what a mess it would be, given the breadth of languages/toolchains we use.

To make this work, you have to build in a container running on the host's architecture, and then copy the artifacts into another container with the target architecture. This would be a huge step down from the elegance of multi-stage Dockerfiles, in which you can encapsulate an arbitrarily complex build toolchain and produce a final image from a single Dockerfile and single build command. From the perspective of CI/CD and our developer tools, cross-compilation would require different strategies for different languages, where it had previously been opaque to the tooling.

On top of all this, cross-compilation still requires emulation to build the final output image, even if compilation isn't actually occurring via emulation. We'd have to make sure our CI/CD runners were available on both ARM and AMD, and that each app's image was built on the right type of runner.

We also had to support developers building locally from either ARM or Intel laptops, which means we'd need to invert the build steps' target platforms depending on the architecture of the laptop.

Yuck.

Multi-platform container images​

With either of these options, we still haven't gained the capability to build multi-platform images. With traditional Docker builds, you can only build single-architecture images. We really wanted to run per-service canary deployments to manage the risk of migration, and having to manage two separate per-architecture tagging schemes would have leaked a lot of complexity from the build into the rest of our deployment system.

Luckily, there's a number of container build tools available (e.g Buildah/Podman, Kaniko) that do support multi-platform builds.

Given that we were already pretty invested in Docker as our build tool (both locally and in CI/CD), we took a look at Docker's BuildKit- which as it turned out, solved all these problems in one fell swoop.

BuildKit builders? What is this sorcery?​

Docker has been rearchitecting its build engine for a few years now, replacing its legacy build engine with BuildKit, and extending the client interface with the buildx subcommand. Docker now has the ability to orchestrate builds using different drivers, allowing you to use one or more builders running on a potentially separate host. You define a builder with some docker commands, giving the docker client the info it needs to utilize the (possibly remote) builder running BuildKit.

The docker buildx build command establishes a network connection with the BuildKit instance, passes container registry auth info, copies the build context (source files, etc), executes builds for any number of platforms in parallel, pushes the output image to a registry, and interleaves the output streams from the builders back to the docker client.

Remote builders support the full feature set of Docker; there's no loss of functionality. We were a little worried we wouldn't be able to use some of the more advanced features, like secret mounts or ssh-agent sockets... but it turns out it worked 100% seamlessly.

After some experimentation, we found we could define a pair of builders (for ARM64 and AMD64), and construct a docker buildx build command that would reliably produce multi-platform images, without relying on either emulation or cross-compilation; all with only slightly longer build times than building a single, native-architecture image locally. All this could be done with a single Dockerfile!

docker buildx imagetools inspect <image-name>:<tag>

Trying out the kubernetes driver​

Our first iteration used Docker's kubernetes driver, which given a kubeconfig context and a few other settings (e.g. resource requests/limits), will spin up BuildKit pods, and use the Kubernetes API to exec into them and kick off processes with buildkit CLI commands.

We discovered some limitations to this approach; most notably:

• developers (and CI/CD) needed to authenticate to a Kubernetes cluster to execute builds, which added some complexity and new surface area from a security perspective.
• the driver doesn't do any intelligent load balancing between the builders; the only available load balancing algorithms are random and sticky, neither of which do much to prevent a single builder from being overwhelmed.
• there's no autoscaling capability. We would have had to choose between wasting money on over-provisioning builders, or risk builds failing due to lack of capacity.

Not ideal.

Creating a central builder service​

We quickly pivoted and tried a different approach: using Docker's remote driver with a centralized deployment of BuildKit pods behind a network load balancer.

This worked amazingly well; it solved all the scaling and auth problems we'd had with the kubernetes driver, drastically simplifying the client build tooling in the process.

buildctl debug histories

When this was all done, we could use a single command in our developer tooling to build multi-platform images, regardless of the platform of the host laptop or CI/CD runner.

Great success!

So you decided you didn't need layer caching?​

The astute reader will have guessed that Docker's extremely valuable layer caching feature probably wouldn't work when running a build on a different host, especially one that's randomly selected by a load balancer. That was very clever of you to notice!

Luckily, BuildKit has a feature called registry-based layer caching that allows you to cache layers in a external container registry. This works just like traditional layer caching, but the cache layers are stored in a specially formatted OCI image in the registry (we're using Amazon ECR).

This required some additional complexity to our build tooling to make sure we were tagging and pushing cache images with the right settings (e.g. using the max mode to support multi-stage builds), but once that was done it was completely invisible to developers.

Putting this all together​

We now had a reliable way to build multi-platform images built into our existing developer tooling, but we still needed a way to configure our pods to run on Graviton instances.

It turns out this was pretty trivial with Karpenter. We created a new NodePool and EC2NodeClass for Graviton instance types, and added a well-known taint and label (kubernetes.io/arch) our pod specs could target with a corresponding toleration and node selector.

Results​

In the end, we are generally seeing the savings/performance improvements we expected, though it varies slightly by workload. We definitely haven't encountered a workload that performs worse on Graviton than on Intel/AMD.

Don't get me wrong, the savings have been really nice, but the improvement to developer productivity has actually been a lot bigger than I expected.

As more of our developers were getting refreshed to new Apple Silicon laptops, building with emulation was becoming an increasingly huge pain (since we were previously always building AMD64 images). Ironically, having the new remote builders allowed us to safely and incrementally convert our target architecture in production, which in turn allowed our developers to switch to running native ARM64 builds locally... and not use the remote builders.

And damn, these new Macbooks run native builds really fast.

We still use the build service for CI/CD builds, since our CI/CD runners are still all AMD64, but our developer build tooling can decide on remote vs local builds automatically by detecting the host and target architecture. We haven't seen any differences in the resulting artifacts between local and remote builds, thanks to the the fact that BuildKit works exactly the same locally vs remotely.

So go out there, save some money, some carbon, and hopefully some penguins.

Special thanks​

Special thanks to Liz Fong-Jones, whose awesome talk at AWS re:invent (2024) about migrating Honeycomb's Lambda-based architecture to Graviton inspired me to finally write this up.

Appendix​

An example docker buildx command for building a multi-platform image:

docker buildx build \
    --builder remote \
    --push \
    --pull \
    --platform 'linux/amd64,linux/arm64' \
    --tag '111111111.dkr.ecr.us-east-1.amazonaws.com/my-app:some-tag' \
    --cache-from 'type=registry,ref=111111111.dkr.ecr.us-east-1.amazonaws.com/my-app:some-tag-cache' \
    --cache-from 'type=registry,ref=111111111.dkr.ecr.us-east-1.amazonaws.com/my-app:some-tag-cache' \
    --cache-to 'type=registry,mode=max,ref=111111111.dkr.ecr.us-east-1.amazonaws.com/my-app:some-tag-cache,image-manifest=true' \
    .

A few notes:

• The --push flag tells Docker to push the resulting image to the registry, since it can't necessarily store the multi-platform image in your local Docker daemon's cache (unless you have containerd enabled... long story).
• The multiple --cache-from flags allow us to use the cache from a previous build from either the main branch or a feature branch. With --pull, Docker will pull both and automatically select the right one, and will also fail gracefully if one doesn't exist.
• --cache-to tells Docker to store the resulting cache of this build. The mode=max attribute is very important here, since otherwise the cache would only contain the final stage's layers.
• The --builder flag tells Docker to use the remote builder named remote, which is defined in a file created by our tooling at ~/.docker/buildx/instances/remote, and looks like this:

{
    "Name": "remote",
    "Driver": "remote",
    "Nodes": [
        {
            "Name": "remote-arm64",
            "Platforms": [
                {
                    "architecture": "arm64",
                    "os": "linux"
                }
            ],
            "Flags": null,
            "DriverOpts": null,
            "Endpoint": "tcp://arm.docker-builders.mycompany.com:3569",
            "Files": null
        },
        {
            "Name": "remote-amd64",
            "Platforms": [
                {
                    "architecture": "amd64",
                    "os": "linux"
                }
            ],
            "Flags": null,
            "DriverOpts": null,
            "Endpoint": "tcp://amd.docker-builders.mycompany.com:3569",
            "Files": null
        }
    ],
    "Dynamic": false
}

At my work, we've been struggling a bit over the past few years with decisions made (almost 10 years ago now) about our AWS network design. While we have a full class A private network (16,777,216 IPv4 addresses), we've managed to paint ourselves into the very sad corner of looming IP address exhaustion.

There's a few reasons:

• Our integration with cell network carriers (to support our home security systems) requires a huge chunk of our IP space
• Our decision to use a multi-account architecture in AWS, and that we chose to use a flat IP space across our accounts. This means our IP space is fragmented across accounts, regions, and availability zones, making a lot of that address space effectively unusable.

Even with all of this, we might have been fine... until we went big on Kubernetes.

Kubernetes eats IPs for breakfast​

Kubernetes has been a huge win for us. But it gobbles up IP addresses like Pac-Man with a tapeworm.

It's fairly straightforward math: in AWS EKS, with the VPC CNI integration (i.e. a network plugin for Kubernetes that allows it to integrate with AWS's networking APIs), here's what happens to all your IPs:

• The EKS control plane requires at least 16 addresses (at least 6 per subnet)
• Every node (EC2 instance) requires at least one address, but depending on your CNI settings, the CNI plugin can eagerly allocate additional addresses to keep "warm" (to speed up pod creation)
• Every pod on a node gets its own IP address. This includes not only user workloads, but also every daemonset pod. In our cluster, we have at about 8-10 daemonset pods per node.

This means, as we've migrated workloads to Kubernetes, we've increased the number of IPs we're using by roughly 10x.

This adds up quickly. We've had a few close calls with IPv4 exhaustion during high traffic events where we had to scramble to temporarily kill non-critical workloads to free up IPs, rebalance across availability zones, or provision new subnets to make sure customers weren't affected.

Actually, IPv6 is a thing​

Unlike IPv4, IPv6 address space is so incomprehensibly large that it's effectively unlimited. For example, a typical IPv6 private subnet would have a /64 IPv6 CIDR, which is 18,446,744,073,709,551,616 addresses.

IPv6 has been a standard for like 25 years, but is still not widely adopted (for a lot of reasons, including backward-incompatibility, lack of ecosystem support, and ISPs squabbling and dragging their feet).

It's legitimately really difficult to migrate a large distributed architecture like ours to IPv6, because, historically, it would require simultaneous changes across many different systems, along with some scary big-bang moments. It also requires reconsidering a lot of assumptions built into your network design and security strategy.

It's been hard to figure out how to untangle that knot.

Enter EKS IPv6 mode​

Given the scarcity (and price) of public IPv4 addresses, and to support the increasing scale of its customers, AWS has been under a lot of pressure to provide more viable paths to adopting IPv6. In one of the smartest moves I've seen from them in a while, they've used Kubernetes's built-in IPv6 support to build a new IPv6 mode for EKS.

Here's the core of the hack: While each node continues to get an IPv4 address, pods get only IPv6 addresses.

Inside the cluster, all traffic is via IPv6, but traffic to and from the cluster gets NATed through the nodes' IPv4 addresses. From the perspective of anything outside the cluster, connections appear to be coming from the nodes' IPv4 addresses. This means only the software inside the cluster has to be modified to use IPv6.

This translation of IP version between inside and outside the cluster has allowed us to migrate our workloads incrementally, which has made the whole process much more tractable.

Migrating only EKS workloads, alone, looks like it's going to allow us to reduce IPv4 address usage significantly, perhaps enough to solve our IPv4 exhaustion without any further network changes. Even if not, it should buy us years of additional runway before we hit that point.

What does migrating an EKS cluster to IPv6 require?​

Unfortunately, you can't enable IPv6 mode on an existing EKS cluster; you have to create a new cluster and migrate your workloads over. There have been a bunch of specific challenges around this (mostly just minutiae around Terraform wrangling and executing DNS cutovers), but now that we've found most of the corner cases, the process is pretty mechanical.

The bulk of the remaining work is around making any code or configuration changes necessary in the individual workloads to get them to bind to IPv6 addresses.

A few basics​

I've been a programmer for like 30 years, and I had never done anything with IPv6 before this migration. There were a few embarrassingly basic things I had to learn about IPv6:

• The IPv6 "all interfaces" address is ::, which is equivalent to 0.0.0.0 in IPv4.
• The IPv6 loopback address is ::1, equivalent to 127.0.0.1 in IPv4.
• URLs that use an IPv6 address as the hostname need the address enclosed in square brackets, e.g. http://[2001:db8::1]:8080 so the colons in the address don't get confused with the port delimiter.
• Happy Eyeballs is an algorithm (implemented by most network clients) that allows apps (including browsers) to efficiently decide whether to use an IPv6 or IPv4 address when both are advertised via DNS.

Your OS and language probably supports IPv6​

One cool thing is that almost all modern OSes (Linux, Mac, Windows) support "dual-stack": they can listen on a port on both IPv6 and IPv4 from a single socket.

On top of this, most high-level programming languages (and their standard libraries) utilize this feature, so if you bind to the :: (all interfaces) address, you'll be able to listen on both IPv4 and IPv6 at the same time.

For example, in node.js:

const http = require('http');

const server = http.createServer((req, res) => {
  res.writeHead(200, { 'Content-Type': 'text/plain' });
  res.end('Hello World!\n');
});

// binds to port 8080 on all IPv6 and IPv4 interfaces by default!
server.listen(8080);

Or you can do it explicitly:

// Or you can do the same thing explicitly:
server.listen(8080, '::');

Here's the same basic thing in Go:

package main

import (
    "net"
)

func main() {
    // Binds to all IPv6 and IPv4 interfaces.
    // Note the square brackets around the address, since the 
    // interface is a subset of a URL.
    listener, err := net.Listen("tcp", "[::]:8080")

    // ...

The same thing is true in .NET, Python, Rust, Java and probably most other languages that aren't doing something weird in their networking implementation.

Of course, most languages also have lower level networking APIs that are IP version specific. If you're doing more complicated things with sockets, you may have a little more work to do.

Unfortunately, not all apps use dual-stack by default​

Even though IPv6 support is readily available in most OSes and languages, it's not always enabled by default in every application. This was particularly annoying for us, because we use a lot of OSS and 3rd party container images as mock dependencies for integration tests, and supporting IPv6 meant we had to add explicit configuration for in a lot of places where we previously just used the defaults.

In most cases, the trick is finding the magic CLI arg, environment variable, or config file setting that controls the host to bind to, and setting it to ::.

IPv6 cheat sheet​

Here's a bunch of examples of various apps I've had to learn how to get working with IPv6:

aws-load-balancer-controller​

You don't need to configure the aws-load-balancer-controller itself any differently for IPv6, but when creating Ingresses that use it, they need to have the following annotations to support IPv6:

# Tells the controller to create target groups of pod IP(v6) addresses. 
# The "instance" target type won't work on IPv6.
alb.ingress.kubernetes.io/target-type: ip
# Tells the controller to create a load balancer with IPv6 enabled
alb.ingress.kubernetes.io/ip-address-type: dualstack

The nice thing about this is that the load balancer itself will listen on IPv4 addresses (in addition to IPv6 addresses), which means IPv4 clients won't even know the app has been migrated.

ingress-nginx​

In the helm values for ingress-nginx, you need to set the ipFamilies value to include IPv6:

controller:
  service:
    ipFamilies:
      - IPv6

MongoDB​

Mongo binds to IPv4 only by default. You can get it listening to IPv6/IPv4 (dual-stack) interfaces with the following command override:

mongod --ipv6 --bind_ip ::,0.0.0.0

Here's an example of a Kubernetes pod:

apiVersion: v1
kind: Pod
metadata:
  name: mongo
spec:
  containers:
  - name: mongo
    image: mongo:8
    command:
    - mongod 
    - --ipv6
    - --bind_ip 
    - "::,0.0.0.0"
    ports:
    - containerPort: 27017
      protocol: TCP

More info: https://www.mongodb.com/docs/manual/core/security-mongodb-configuration/

Redis​

Redis binds to IPv4 only by default. You can change it to bind to all interfaces with the following command override:

redis-server --bind "0.0.0.0 ::"

Here's an example of a Kubernetes pod:

apiVersion: v1
kind: Pod
metadata:
  name: redis
spec:
  containers:
  - name: redis
    image: redis:5
    command:
    - redis-server 
    - --bind
    - "0.0.0.0 ::"
    ports:
    - containerPort: 6379
      protocol: TCP

MariaDB​

MariaDB 5.5+ already listens on :: by default, so no additional configuration is needed.

LocalStack​

LocalStack currently doesn't support IPv6. However, I've opened a PR to add IPv6 support. If that PR gets merged, then you'll be able to use an IPv6 address in the GATEWAY_LISTEN env variable:

GATEWAY_LISTEN=[::]:4566

RabbitMQ​

RabbitMQ listens on :: by default, so no additional configuration is needed.

nginx​

The nginx image's default config listens on both IPv4 and IPv6 by default.

If you're authoring your own nginx.conf, you need to add listeners for IPv6 and IPv4 separately. Here's an example of binding port 3001 on both IPv6 and IPv4:

listen       3001; # IPv4
listen  [::]:3001; # IPv6

OTEL collector​

The OpenTelemetry Collector config accepts the :: (all interfaces) address any place you could specify an IP address. For example:

config:
  receivers:
    otlp:
      protocols:
        grpc:
          endpoint: "[::]:4317"

Other OTEL collector components will automatically use IPv6. For example, the Prometheus receiver correctly uses IPv6 pod addresses when scraping metrics.

Jaeger​

Jaeger already listens on :: by default, so no additional configuration is needed.

WireMock​

WireMock already listens on :: by default, so no additional configuration is needed.

Gradio​

Gradio binds to 127.0.0.1 by default. You can use the server_name property to set up an IPv6 binding in the launch() method in the Blocks object:

blocks.launch(inline=False, server_port=5112, share=False, server_name="[::]")

Uvicorn​

Uvicorn will bind to all IPv4/6 interfaces if you set host='::' in the Config object:

ip_config = Config(app=_fastapi_server, host="::", port=8080)
return Server(ip_config)

More info

More IPv6 cheat sheet examples, please!​

I'll be adding more IPv6/dual-stack configuration examples as I encounter them.

Do you have more? Leave them in the comments and I'll add them to the list!

As a platform engineer, I enjoy the benefits of working in a field with a vibrant ecosystem of open source infrastructure and developer tools. I've spent much of the last decade building developer platforms by curating and assembling these tools, and after a number of iterations, I seem to have hit on something that's working really well for my current company (SimpliSafe).

As our platform's adoption has grown, we've gotten more and more frequent, really positive, heartwarming feedback from our developers who really like it. This is absolutely freaking delightful, and honestly never stops surprising me.

I often get asked by our developers if we should consider open-sourcing the platform. I've spent some cycles entertaining the idea, but I usually don't get very far before it seems unworkable.

This post is an experiment in thinking in public; I'd like to brain dump my thoughts on the challenges of building an open-source developer PaaS, in the hopes that the platform engineering community might provide some insight to get me past this block.

So, tell me more about this platform​

Our platform is named "dex/EKS", which is (an admittedly awkward) combination of the name of the client tool, "dex", with the AWS service the server-side is built on (EKS: AWS's managed Kubernetes service). Unsurprisingly, developers tend to just call the whole thing "dex".

In the spirit of the "Platform Engineering" buzzword, dex/EKS encapsulates our company's collective opinions, policies, and best-practices for building, deploying, and operating apps. I like to think of it as a PaaS that we've curated and glued together out of a bunch of open-source and vendor tools.

Our Kubernetes distribution​

In addition to the client tooling, we also have a fairly sophisticated Kubernetes "distribution", which consists of a bunch of curated cluster-side components, combined and configured to work well together. Our cluster configuration is defined with Terraform, and we use it with Github Actions to manage many dozens of EKS clusters. Beyond that, there's integrations with a bunch of third-party SaaS providers, including AWS services and other vendors.

Just to give you a sense of the ingredients that comprise the platform, here's a partial list:

• Kubernetes (AWS EKS)
• Docker/BuildKit
• Github Enterprise (with Github Actions)
• Okta
• Artifactory
• Grafana Cloud
• Honeycomb
• OpenSearch (for logs)
• A bunch of AWS services (ECR, S3, SSM, SecretManager, Route53, ACM, WAF, etc)

Here's a few of the tools (from the Kubernetes ecosystem) we use in our EKS configuration:

• aws-load-balancer-controller
• ingress-nginx
• Karpenter
• external-dns
• external-secrets
• Fluent Bit
• OTEL (OpenTelemetry) collectors
• KEDA
• Kyverno
• Velero

dex: the CLI tool​

The CLI tool abstracts and integrates the APIs of these infrastructure components and exposes them through a simplified, declarative set of configuration and CLI commands. Some of the things it handles:

• Configuration management (what settings your app gets in different environments)
• Secrets management
• Cross-platform and multi-platform container image builds
• User authentication (i.e. SAML auth via Okta, to Kubernetes, AWS, and Artifactory)
• AWS IAM integration (allows you to assign AWS permissions to your app)
• Kubernetes manifest management (imagine a simplified version of helm)
• Ingress management (load balancers, certs, and DNS)
• Vulnerability scanning
• CI/CD integration (Github Actions)
• Telemetry pipeline integration
• Docker/Kubernetes-based integration testing framework

One of the key metrics we hold ourselves to is that developers have to be able to get a new "hello world" service up and running in less than 10 minutes, at which point they can turn their focus to business problems. As they get closer to production, they have a few more decisions to make about autoscaling, observability, etc, but for the most part, the platform narrows down the choices to just a few fully-baked, meticulously documented, well-trodden paths.

The impact of dex at SimpliSafe​

Teams at SimpliSafe have migrated the majority of our services to dex/EKS, and most teams are planning on moving their remaining services over in the next year or so. This has happened with close to zero pressure from management; teams are moving their services to the platform because they're much happier with it than without it.

Suffice to say, I'm very proud of this outcome, and dex seems to be providing a lot of value.

A platform design reflects a company's culture​

While it may appear that dex is just a set of choices about tools and services that have been manifested in glue code, it also reflects SimpliSafe's values and organizational culture. While most of these values and cultural properties fairly well subscribed, they're by no means universal:

• Teams should have a great deal of autonomy to choose tools, languages, frameworks, and processes, and they should be accountable for operating the systems they build
• Continuous delivery is better than big-bang releases
• Microservices are a good way for a large team to build a big system
• A central team should own cross-cutting concerns like telemetry pipelines and observability backend tools, auth, infrastructure provisioning, etc
• Infrastructure should be represented as code and managed through automation

There's a bunch of others, but you get the picture.

These values deeply inform the design of dex, and it's interesting to look back on the iterations of platforms that I've built at different companies with different values, and how they were reflected in the design of the platforms.

Example: ephemeral environments​

Ephemeral developer environments are a feature I 100% know is a huge win for developers, regardless of your company culture. But there have been big differences in design, features, and implementation details when I've built this feature at different companies.

What are ephemeral environments?​

Here's the gist: A developer should be able to deploy their app into an isolated, temporary environment from their local machine into Kubernetes, with a single command, so they can iterate, tweak, and test ideas in a production-like environment. They'll need a URL to access the app, so they can play with usability, do manual testing, attach a debugger, troubleshoot config, etc. When they're done, they use another command to tear the whole thing down.

Additionally, for every feature branch they push to Github (or Gitlab, Bitbucket, etc), a CI job will deploy an ephemeral environment with the same properties, run automated tests, and tear it all down when complete (or after a specified period of time).

Ephemeral environments in a financial institution​

At my last job, in a commercial bank, we had a very small number of tightly controlled, multi-tenant Kubernetes (OpenShift) clusters. As you might imagine in a highly-regulated industry, the creation of Kubernetes namespaces (and the associated access controls) was governed by security controls, required approvals, and needed to leave an audit trail. The process was managed by a central team.

To allow creation of dynamic, isolated environments, we worked within the static structure of centrally managed namespaces by designing our tooling to generate Kubernetes objects using strict naming conventions (e.g. prefixing all resources with the name of the developer or feature branch). This allowed the tooling to manage the objects as a unit, avoid collisions, and ensure that the objects were cleaned up when the developer was done.

This design decision trickled into many other aspects of the system. For instance, we designed our client tooling to maintain pretty tight control over rendering the objects, and the relationship of objects via names, labels, and selectors.

Ephemeral environments at an IoT security company​

At SimpliSafe, the company's culture and preexisting architecture enabled a very different approach: ephemeral environments are implemented via Kubernetes namespaces, and the client tooling can create (and destroy) namespaces dynamically.

Because we have per-team AWS accounts, and our Kubernetes clusters already provide strong isolation, we're comfortable giving developers the power to manage namespaces in our non-production environments. This removes a lot of the need for strict control over object relationships in Kubernetes, and gives developers more flexibility to mess with the underlying objects more directly.

This additional power is a reflection of SimpliSafe's culture of autonomy and trust in developers.

Different tradeoffs, different design​

So even with the same feature, providing pretty similar benefits to developers, we had to make very different tradeoff decisions, and ended up with design differences which significantly impact the architecture, features, and feel of the rest of the platform.

What would this look like open-sourced?​

Given the two examples of companies with different infrastructure opinions, let's think through the possible flavors of open-sourcing a developer platform like dex:

Option 1: A hyper-opinionated "PaaS in a box"​

This option assumes that the infrastructure decisions we've made at SimpliSafe would be a good fit for at least a bunch of other companies, with minimal modification. We'd provide the whole thing, end-to-end, including the EKS cluster configuration and terraform, all the cluster-side system components, and the dex client-side tool.

I find this option hard to imagine for a few reasons:

• While I'm very confident we've got a great solution for SimpliSafe, I think it's virtually impossible that any other company would be happy with all our opinions (the bank certainly wouldn't have been). Our platform glues together scores of specific OSS products (and a number of SaaS vendor tools), and the odds that every one of them lines up with another company's preferences is close to zero.
• A platform engineering team using this version of the platform would be signing up to build expertise and support every OSS production we've chosen.
• While out-of-the-box, opinionated platform might be good for a startup, our platform is certainly NOT the the right choice for a startup. It's designed around supporting many teams, and to allow a central platform engineering team to manage infrastructure underneath teams' apps... which is not the problem engineers at a startup should be worrying about.
• Among the opinions encapsulated in our platform are some we're not happy about. We have a few compromises based on legacy infrastructure choices that are hard to change, and some choices which are an intermediate phase between where we are and where we want to go. For example, we're currently using the telegraf-operator to collect metrics for lots of our services, but we'd prefer to be using OTEL SDKs and/or Prometheus libraries.

I actually can't imagine myself choosing to use someone else's OSS platform if it were built on this philosophy.

Option 2: A whole platform, but pluggable​

In this variant, we'd provide also provide the whole platform, but allow users to bring their own infrastructure opinions via a plugin API.

I also see some big disadvantages here:

• Abstraction layers add complexity. Part of the value of dex is that the code is relatively simple, straightforward, and hackable. We often get a PR or feature request, and end up cutting a new release within hours. This would not remain the case if we started adding abstraction layers everywhere.
• Testing and maintaining compatibility with all possible plugins would be a huge burden. Right now, dex's integration tests are both comprehensive and fast, and it would be virtually impossible to maintain this level of coverage if we had to test against an ecosystem of plugins.
• It's really hard to build good abstraction layers, even for simple things. And these infrastructure components are definitely not simple. We'd be constantly expanding and modifying the APIs to support additional opinions, and the abstractions would inevitably leak.
• Many of the infrastructure choices we've made allow us to simplify the design of the platform, and these simplifying assumptions wouldn't be valid if we allowed arbitrary plugins. Tight coupling, in this case, is part of the special sauce for creating a really streamlined and cohesive developer experience.
• Comprehensive documentation would be much more complicated and far less useful, since docs would have to simultaneously support the perspective both of the platform developer as well as the end-user developer. dex has lots of docs based on developer use cases, and it wouldn't be possible to provide these if the whole experience were built on plugins.
• Kubernetes APIs already provide so much power and extensibility, and especially when you throw in Operators, CRDs, and custom controllers, it's hard to imagine how I could provide APIs that would support all the flexibility Kubernetes offers.

I think realistically, this solution would start as option 1 and then abstractions would be gradually added by contributors to support their particular infrastructure choices, so it's probably best to think of this option as a spectrum with more or less pluggability.

Option 3. A toolkit for building your own platform​

Another option is to factor out individual components of the platform as standalone libraries, and let people build their own platform. I could imagine some of dex's components being useful for someone who wants to build a different opinionated platform.

One example of a generally useful component is our config system:

• Our config schema is defined as a tree of TypeScript classes, which can be used to generate a JSON schema, which can be used by other tooling to provide instant validation (e.g. via VSCode's JSON schema integration), to validate at runtime, an also to generate documentation.
• The config system supports defining arbitrary target environments, which can use inheritance (and other mechanisms) to share common settings, and override them as needed.
• It has a mechanism for declaring dynamic config values (e.g. a value from a Parameter Store secret, or based on the current git branch name, etc).
• The config loader returns a config tree object which is built out of JavaScript Proxy objects, which allows us to do very smart validation, with user friendly error messages, and play to TypeScript's strengths.

That said, turning this config module into a separate npm package would have some tradeoffs:

• The inherent packaging tax: working with multiple npm packages is more complex to develop, debug, and test locally.
• It's abstractions would feel leaky to a user. For example, JSON schema generation requires some special build configuration, and this would appear a bit finicky if it was intended to be used off the shelf.
• There's some other aspects of our config system that are currently tightly coupled with other parts of the platform. This is all just code, so of course we could figure out how to decouple it, but there would be a decent amount of net-new complexity as a result.

More generally, I think the challenge with this approach is that most of the value of the platform stems not from the individual components, but from their integration. For example:

• dex's packaging/distribution mechanism (a stable CLI + a fast-moving, versioned library) has many moving parts
• dex's own build system is very sophisticated, and has a lot of features around building canary releases, and enabling debugging in a sample host project
• dex also has a suite of integration tests are fairly involved and comprehensive
• The documentation of dex's UI (both its command line interface and config interface) is a huge factor in dex's success at SimpliSafe, and would have to be built from scratch for a new platform.

A plea for help​

So I'm sitting on this great set of tooling, which is providing a ton of value for my company. It's built on OSS, public cloud, and SaaS services, and there's no proprietary magic or novel intellectual property we're trying to protect. It solves a problem that a huge number of medium-large technology companies would have to tackle.

Why can't I see a way to share this with the world? Maybe I'm just not being imaginative enough. I'm 100% certain this isn't a novel situation.

What do you think?

My current company is struggling with some challenges that are pretty typical for a wildly successful startup that's rapidly grown into a medium-sized company. We've got the expected technical debt, organizational design challenges, and a seemingly infinite number of small systems that work great... until they catch fire as we hit new scaling thresholds.

I've been through this a few times before, and none of this is worrisome to me. It's exactly where I'd expect a company be after a period of frenetic growth. In a lot of ways, this is a really fun stage; if you're someone who can tolerate a bit of chaos and ambiguity, there's tons of opportunities to shape the direction of a company.

On one hand, at a startup, your priorities are dominated by survival and existential risk. But once you grow beyond a certain size, you begin to lose leverage as cultural inertia takes over. A scrappy, determined visionary can exert a lot of leverage during the awkward teenage years of a mid-sized company.

I want to tell a story about how I watched such a visionary person, at this moment in a company's lifespan, solve a particular problem around knowledge management, and how he drove a really compelling, sustained, positive cultural change.

Yeah, and also how it eventually collapsed, and why.

Someone should go ask Rob why this one button is purple​

When I joined Vistaprint, it was already very successful and growing rapidly. It'd been a few years since I'd hopped jobs, so I was a bit nervous and eager to dive in and start getting shit done. This turned out to be a bit of a longer process than I had hoped.

While Vistaprint had a number of cultural virtues, one very annoying feature was how much it relied on oral tradition, and a few linchpin people who functioned as repositories of all knowledge. Seriously, there were like 5 or 6 people in the company that collectively held about 80% of the total knowledge, and it was getting increasingly hard for them to get any work done as the number of people whose questions they needed to answer was growing.

Here's some of the kinds of things I'm talking about:

• how to get your developer environment set up
• the reasoning behind a particular software system's design
• what UI ideas had been tested, and if they had succeeded or failed
• why a particular compiler flag was used
• how you'd get a new feature toggle enabled for analytics
• who the hell owns any particular piece of code

It was also more than just technical knowledge. HR policies, business strategy docs, organizational charts, holiday schedules; if these things existed, they were stored as a Word doc on someone's share drive, and were impossible to discover unless you already knew what you were looking for.

Dan builds a wiki​

My colleague Daniel Barrett (who you may know as a prolific author of some fantastic books on Linux and other topics) was already a veteran at Vistaprint when I joined, and was notably one of the "grownups" amidst a bunch of young upstarts. As part of his more general management duties, he had volunteered to figure out what how we were going to train the raging torrent of new hires that seemed to be showing up every day.

Dan recognized that training was actually a subset of the larger problem of knowledge management, and convinced the technology leadership team that we needed a more fundamental solution. He got to work on some ideas and started building a small prototype.

Not long afterwards, Dan presented at a technology team all-hands meeting, and introduced us all to a new wiki system (which at the time was called "TechWiki") that he'd built on top of MediaWiki (the software that powers Wikipedia). He had already seeded it with a straw-man categorization (based on his personal knowledge of the systems), and a number of stub pages. He explained his intention that we should all just start using it to write stuff down, and not worry so much about organization. He was going to work with teams to watch and learn how it was being used, and help coordinate efforts as it evolved.

There was a lot of skepticism across a number of fronts. Here are a few of the major objections that I remember:

• Everyone had full access to edit any page, at any time. How would we prevent people from screwing up each other's content?
• Without a strict, hierarchical taxonomy, wouldn't everything just spiral into an unmaintainable mess?
• Developers aren't particularly known to enjoy writing documentation. How are we going to get people to take time away from engineering to write stuff down?

Dan had the wisdom to respond to these questions with the only correct answer: he didn't know. We were just going to try some stuff and see what happened.

The wiki takes off​

It took a few months, but we started to notice that adoption of the wiki seemed to be growing pretty quickly. Daniel would pop by to check on us, let us know he had been reading what we'd been writing, and had some tips on how to use the wiki more effectively. He'd give us suggestions on organization, tone, and some general style tips.

He also encouraged us to be less "precious" with the wiki. He said it was a great place to take meeting notes, add team-specific content, and that we should feel free to create stub pages for topics we wished existed (but didn't know much about ourselves). What's more, he said we shouldn't hesitate at all to edit articles when we found missing or out-of-date information, regardless of whether we were the original author, or even if we didn't have any specific expertise or claim on the topic.

In the beginning, I'd often get email notifications that Dan had made minor edits to pages I'd created, usually normalizing titles, fixing typos, or adding category tags. It wasn't very long until I started noticing other people editing my pages. At first, I'd look at every change with some suspicion, but as it turns out... everyone editing my pages was actually doing a great job. They were genuinely improving content I'd written and adding details I'd missed. Even if the content they added wasn't 100% right, it was at least useful for me to know what answers they actually wanted, so I could correct any errors.

Meanwhile, while Dan continued his evangelization, he was also building a team that was driving improvements to the wiki software itself. We got more integrations with our other systems (e.g. JIRA integration, queries into our analytics database, links to shared drives, etc), and better search indexing. They were also working furiously in the background to keep the content categorized in a sane way, fixing typos and structural inconsistencies, adding searchable summaries, and making it possible to relax the stress on authors, while also maintaining some semblance of consistent organization.

Imagine a world where all the shit is written down​

This momentum built on itself, and accelerated exponentially. It wasn't that much longer before the wiki became the go-to place for everything. The most notable effect of this change was that when you found yourself with a question, instead of looking for the expert, the first thing you'd do is just look it up on the wiki. It was shocking how often you'd find the answer, and if you didn't, you'd go find the answer offline, and then go and create a damn wiki page about it.

In stand-ups, team members would remind each other to update the wiki page for that system they just changed, process they added, or question they couldn't find the answer to yesterday. Managers would look at the volume (and quality) of wiki contributions when doing quarterly reviews (I remember one time I was the 2nd or 3rd biggest contributor, and was very proud).

This success was so significant that the rest of the company (outside of the technology team) took notice. There were originally concerns that MediaWiki's content editing interface, which required learning Wikitext (a content markup language similar to markdown) was going to be too much of a barrier for our non-technical colleagues. This turned out not to be a big deal, as it was pretty easy to learn just the small subset of features you needed to create to create most content. It wasn't much longer before "TechWiki" was rebranded to "VistaWiki", and was adopted across the whole company.

For someone who hasn't spent time in a company with this level of knowledge management practice, it's hard to describe how positive it was. I have little doubt that the wiki provided us a real, tangible competitive advantage. Experienced new hires inevitably noted how useful the wiki was, and how much it contributed to their effectiveness.

The era of VistaWiki lasted for about a decade, across several transitions of leadership, and through a period of really huge growth. All the while, Dan was working in the background to keep this culture alive and healthy.

Entropy affects culture too​

A few years before I left, I noticed that something had changed. The wiki was still there, and Dan and his team were still plugging away, but it became increasingly obvious that employees, more and more, had started taking the wiki for granted. As employees turned over, the wiki culture was emphasized less and less to new hires, who became gradually more likely to try to find an experienced colleague instead of searching the wiki first.

Unfortunately, our tech leadership at this time, who had inherited this culture but didn't fully grasp its significance, wasn't spending much energy on preserving and promoting it. To be fair, they had a lot of other fires to fight, and its understandable that, like an increasing share of the population, they took it for granted too.

I was part of (middle) upper management at this time, and though we were spending a lot of energy on nurturing culture, knowledge management wasn't included in the features we were supposed to be promoting.

Some teams across the organization started experimenting with different documentation systems; ones that had features that MediaWiki lacked, or supported documentation that was generated from code, or were just more familiar to them from previous jobs. Most of these were implemented in a half-hearted way, contained siloed information, weren't well maintained, and didn't uphold the core value the wiki was built on: that knowledge should be free, transparent, and discoverable across the whole company.

About a year before I left, Dan popped over to let me know he was leaving, and was off to do something new. I was sad to see him go, but not particularly surprised. He'd been working tirelessly on this stuff for years, and was clearly exhausted trying to keep this thing alive in spite of a leadership that wasn't fighting very hard to keep it from disintegrating.

After Dan left, the degradation accelerated. All the same factors that contributed to the exponential adoption seemed to be working in reverse. At one point I remember someone sending me a Word doc on Slack for something that would have 100% been a wiki article a few years before. It turns out, while this person had used the wiki, they weren't sure how to create an article, and didn't know if anyone would ever look for it there.

This moment reminded me of stories of medieval British peasants, walking past the ruins of Roman aqueducts, wondering where these strange, huge structures had come from, and what kind of people could have built such grand, otherworldly things. Maybe they had been built by giants?

Tips for building your own culture of knowledge management​

I think about this experience as just one data point in my broader understanding about what it takes to build (or change) a company's culture. To avoid overstating my point, I'll try to stick specifically to knowledge management.

Here's the lowdown:

Minimize friction for contributors​

The experience for contributors needs to be as frictionless as possible. Every barrier you put in front of a potential contributor (having to create a PR, having to follow specific procedures, feeling like they have to ask permission) gets multiplied across every person in the organization.

Culture change is always an uphill battle; don't add unnecessary weight to your pack.

Access control is counter-productive​

Don't fool yourself into thinking that locking your shit down is going to improve quality. Access controls are some of the worst kind of friction; they incentivize information silos, and discourage potential contributors that approach content with an outsider's perspective.

Empower everyone to edit everything. In any reasonably healthy organization, the number of people creating good, quality content will vastly outnumber the people messing stuff up.

Either way, if you have employees deliberately producing bad edits, you've got bigger problems than knowledge management.

Don't be precious with content​

Avoid imposing implicit, social friction. Encourage contributors not to worry about taxonomy and organization; it creates unnecessary stress which discourages contribution. A pristine taxonomy is worthless without good content.

Also, don't be too pedantic about style, structure, or tone. If you're successful with adoption, you'll have too much content to ever have manually reviewed, so quality is something that inevitably will have to be addressed via culture (and not by creating speed bumps). If you get it right, people will want to write good content by virtue of social incentives, and because they get positive feedback from their peers and managers... or because they want the content for their own team or future selves.

In this situation, coaching and continuous feedback is much more effective than gatekeeping.

Prefer a single source of truth​

From the perspective of information consumers, you really want the knowledge repository to feel like a single system. You don't want users to have to ponder which of 7 different intranet portals to visit, depending on the type of document. If you don't have an actual unified knowledge management system, a good solution might be a unified search portal, or even just a norm of linking any external content from your primary system.

While a single source of truth isn't necessarily as important for contributors, I haven't yet seen a system with multiple sources of documentation where the fragmented experience ends up discouraging contributors. That said, I'd be curious if there's something that could be done with a federated system (e.g. a central system that scrapes individual sources, but generates content that contains links to edit the source).

You need a jump-start to get critical mass​

Knowledge management culture has a chicken/egg paradox component: consumers won't use it if you don't have sufficient content, and it's hard to incentivize contributors to create content unless they believe it will be used by consumers. You need to find a way to prime the system and get it to become self-sustaining.

This probably requires a few different strategies deployed in parallel:

• Recruit key, influential people (probably the people who are already information bottlenecks) to start creating content
• Practice sustained engagement with teams. Use this to help encourage content creation, remove implicit barriers, and identify sources of friction or stress.
• Consider ways to seed the new system with content pulled (probably via automation) from existing sources. Even if this isn't perfect, there's something about seeing incomplete content that gets people to want to fill in the details.

And probably a bunch of other you'll need to discover incrementally along the way. Much like developer experience is a product, so is content contributor experience.

You need a team to sustain and nurture it​

Probably most importantly, you'll need someone (and probably a team) championing and driving the culture change. Some companies refer to this role as a "Librarian", but I think it's a lot more than that title evokes. While it is important that this person/team has good instincts around information architecture, it's much more about being an effective evangelist and relationship builder. It's fundamentally about changing the behavior of a large group of people, and that's legitimately hard.

This ongoing battle to sustain culture has two fronts:

• Grassroots: contributors have to fully buy in, and understand that the effort they put into creating content benefits them in short order.
• Managing up: If upper management doesn't understand the value of a knowledge culture, it's going to be very hard to sustain it. The management hierarchy, at every level, has to be reminded how much they're getting out of the culture, and that they're responsible for keeping it alive.

Let's extrapolate​

It might be worth making extrapolating a more general principle here about culture change: a great company culture can be a major driver for business success, but culture is a delicate, fickle thing. It can wither and die just as quickly as it was grown.

As leaders, we make decisions all the time about where to direct our limited resources. If there are aspects of your company culture that you believe really matter, take stock of what investment you're actually putting into it. I'm talking real, tangible resources; if you're not making real tradeoffs to sustain it, then you're choosing to let entropy take over. You may one day wake up and find that your teams are behaving in ways that are very counter to the culture you thought you valued.

A commonly accepted notion in software engineering leadership is that managers have a much bigger potential for impact on a business than an individual contributor. This is certainly a credible argument, given that a great manager can have a huge impact through building a great team. They're responsible for recruiting the right people, steering the culture, and making the biggest decisions about what risks to take, what opportunities to pursue, etc. Ultimately, they're accountable for what the team delivers.

And of course, a big team can deliver bigger outcomes, with bigger impact, than any individual contributor could on their own. Given this leverage, its no wonder that it's very rare for companies to have pay scales for individual contributors that match that of managers, especially in senior management.

For those of us who find happiness and fulfillment in working directly with technology, our decision to avoid management can come with a significant economic penalty.

The penalty for optimizing your career for joy​

I made an explicit decision a few years ago that I would leave management in order to get back to the things that originally drew me to technology. While I still think of what I do as leadership, I've come to terms with the fact that for the remainder of my career, I'm going to watch my former peers surpass me in titles, power, and especially in compensation.

At the beginning, this was a little hard on my ego, but over the past few years, I've come to a place of contentment. The amount I look forward to any given day of work is directly proportionate to the amount of uninterrupted time I have to work on engineering problems. I've decided my goal should be optimizing my career for happiness, so this tradeoff works for me.

But I want to push back a little on the idea that management is categorically more impactful than individual contribution. The concept is a bit of a tautology; managers take credit for the innovation and impact of the individual contributors they hire. But because of the nature of software, a single person with the right idea, at the right time, can manifest that idea into the world to great effect- sometimes without any organization supporting them at all.

I'd like to tell you the story of the most impactful thing I've ever done, which was as an individual contributor. Even though this was 18 years ago, I honestly don't know if I'll have another chance to do something quite like it.

Do you remember Web 2.0?​

Back in 2006, I joined Vistaprint to work on the team that owned its "Studio" application. Studio is Vistaprint's "PhotoShop in the browser", where customers can customize and edit designs that will then be used on custom-printed products (most famously business cards, but they also have hundreds of other products). This was a client-side web app, written in JavaScript/CSS, with a backend built (at the time I joined) in VB.NET.

Let me just set the stage, and remind my readers what the web was like in 2006: Microsoft absolutely dominated the browser space since winning the so-called "browser wars" back in the late nineties. Chrome didn't exist. Firefox had been around for 4 years, but it held a fraction of the market share of IE (version 7 at the time), and had virtually no appreciable advantages in user experience over IE. JavaScript was widely regarded as a toy language, and the browsers' engines were all equally, painfully slow. Honestly, none of us ever imagined JavaScript could be faster.

There was nothing like modern web frameworks like React or Vue. Even frameworks now considered legacy, such as jQuery, YUI, and MooTools, wouldn't have their first releases until later that year. The leading JavaScript frameworks at the time were Prototype and Dojo. Flash was still considered the technology for interactive web applications.

One thing that had happened the year before, however, was the initial release of Google Maps. While scrappy browser hackers had done some really cool and innovative stuff before, the effect of the launch of Maps was like a bomb going off in the web development world. It felt legitimately groundbreaking, and it was obvious at the time that this app was going to change the world.

I'm just putting that picture in your mind so you'll have a sense of what we even thought was possible at the time.

The origin of Vistaprint's studio​

Vistaprint's first Studio had been built back around 2001, by a brilliant hacker who embodied the type of contrarian scrappiness that was required to do anything on the web at the time. Despite the primitive browsers of the era, he was able to use a number of techniques, moderately well-known at the time (but not widely used), to do the kinds of things that web developers would later do with AJAX.

The most important of these hacks was to use multiply-nested iframes (and a makeshift protocol that looked a little like JSONP) to communicate with the server without requiring a navigation on the main page. This allowed him to effectively simulate AJAX requests before browsers even had the capability.

What's even crazier about this is that the client-side code for Studio was used in an insane (but also brilliant) hack, where they loaded instances of IE server-side in order to measure text bounding boxes, so that they could render documents for manufacturing. It turns out text rendering engines are really freaking complex, and given the fact that we relied on IE for text layout on the client, the most reliable way to ensure that customers' printed products were faithful to their browser renderings was to use the browser to render them on the server.

Studio in 2006​

By the time I arrived in 2006, Vistaprint had built a very successful business on Studio, and had recently had their IPO, after delivering revenues of about $90MM in 2005. Studio was considered one of their major strategic "pillars", along with their novel manufacturing capabilities.

I spent the first couple of months working on Studio, trying to get my bearings, and wrap my head around what had become a pretty nasty mess of spaghetti over the past few years. Around that time, the team lead had decided to move away to follow his girlfriend out of state, and I was left in charge of an app I barely understood. It wasn't just me, though- my colleagues admitted that none of them had any confidence they could make any significant changes to the application without breaking things.

This is, as a matter of fact, exactly what happened... repeatedly. Vistaprint was growing its product portfolio, as well as trying to iterate to improve usability, and there had been a series of disastrous attempts to add new features to Studio to support this, each followed by an emergency rollback.

Beyond the maintainability challenges, you have to understand what the user experience for Studio was actually like. Because of all the complexity required to make the multiple-iframe communication mechanism work, along with years of features being layered on top, the user interface took a minimum of 60 seconds to load, even on a fast internet connection. About 20% of the time, something would fail (usually due to race conditions) and require a reload.

The app was deliberately styled to look like a Windows 95 desktop app, with CSS that had been carefully crafted to match the beveled edges, corporate grays, button styles, and fonts.

Studio only worked in IE. If you were unlucky enough to be using Firefox, Opera, or Konqueror, or you were on a Mac, you'd get redirected to a very limited, server-side, form-based page where you couldn't do any customization to your document other than edit the text.

The user interface was rife with bugs. We didn't have any real observability, but anecdotally, users would experience a blocking bug in at least 15% of sessions.

What's more, it was becoming less and less reliable to depend on IE to do server-side rendering, since over time, IE's text engine became more and more influenced by settings on Windows, graphics drivers, etc. We had a certain amount of documents that just got printed completely wrong, and had to be manually modified, by humans in our manufacturing plants.

Hitting rock bottom​

It wasn't long before I caused my first major production incident by attempting a bug fix in Studio, despite having been through what felt like a very meticulous QA cycle. After the rollback, we calculated the losses from the incident at about $20K, and I felt pretty deflated. My boss helped to put things in perspective, noting that these kind of losses were common in Studio, and my predecessor had caused many such incidents.

I spent a couple days feeling sorry for myself, and then resolve set in. I was having none of this. This was not OK.

Kindling and a spark​

Around this time, a couple engineers had been working on a new set of server-side text rendering services that we could use for simpler products that didn't require Studio (this was especially appealing at the time, because the conversion rate in Studio was so terrible). I saw a demo that they'd built, and found myself unable to stop thinking about it for several days. One evening, while trying to get to sleep, I had a crazy idea.

What if we could build a brand-new Studio from scratch, where the document's elements would be composed of a set of server-rendered images? The client-side code would just be an interface for moving, resizing, and opening an editor for these elements, which from the perspective of the client, would just be rectangles with a set of properties. These elements would be the same types of elements users could edit in the legacy studio (e.g. text boxes, images, vector shapes, etc), but we'd build server-side rendering services for each one, which would output transparent PNG images so they could be composited together on the client.

The user could then just double click on any of the rectangles to open an element-specific editor. So for text boxes, this would open a simple text-box editor, which would allow the user to type, and then we'd debounce the keypress events to trigger a refresh of the server-rendered text.

This way, the documents we produced client-side could be faithfully rendered server-side using the same text-layout engine, and we could remove a huge amount of complexity on the client.

The pitch​

The next day, I spent some time with the engineers who had done the text rendering work, and we started working through the details of the idea. Once we felt like we had something viable, we brought my boss, Satish, into the conversation. Having a shared experience of pain with Studio, he immediately arranged a meeting with Wendy, Vistaprint's head of "Capabilities Development" (I think she was technically the CIO at the time, but was still directly leading the Engineering team).

I explained the idea to her over the next half hour, and left with permission to suspend feature development on Studio, and to work with one of the rendering guys for a few weeks to build a prototype.

Building the new Studio​

Within a week, the two of us had a working version of Studio that could create a new document, and had some basic editing features, including text editing and drag/drop positioning for all document elements. The results were fairly stunning in contrast to the legacy studio:

• It loaded in just a couple seconds
• It worked in IE, but also Firefox and Opera. It also worked on a Mac.
• It was smooth, snappy and responsive
• The feel of the pop-up text editor, which we were afraid might be weird, was totally fine.

As soon as she saw the prototype, Wendy gave us the green light to go all in. I spent the next few months turning the prototype into a real replacement for the legacy Studio, adding support for each of the element types needed to support our most important products, including business cards and postcards. My colleagues on the rendering side worked on building a new version of the program that would transform Studio documents into press-ready PDFs, using the new server-side text rendering engine (and NOT using IE).

We launched via an A/B test shortly thereafter. Most A/B tests run for changes to Studio for the last few years had either negative or statistically insignificant results, and despite how much better our new version felt, we thought the odds of hitting a home run on the first pitch were pretty low. We would have been happy with breaking even- at which point we would have been able to take advantage of the more maintainable codebase, and focus on optimizing.

When the first A/B test came back, we were floored. Conversion rate was up by about 5 points. Out of the gate, this hack was immediately worth tens of millions of dollars a year for Vistaprint, just for one product! And we had done zero optimization.

The results​

Over the next year, the effects of this success rippled throughout the company. A large number of engineers were redirected to execute changes throughout the system needed to replace the old IE-based document rendering with the new server-side rendering engine. A team was built around me to keep developing the new Studio, and we gradually added the features needed to support an increasing share of Vistaprint's product portfolio.

All this time, Vistaprint was growing like crazy. Each time we'd move a product over to the new Studio, we'd see a huge jump in conversion rate. New core capabilities were being built on top the new Studio architecture, and a ton of new design content, products, and features were enabled. The process of rendering documents for manufacturing was far more efficient now that we had a reliable way to render documents that were faithful to the users' intentions, and we no longer needed a small army of humans to fix broken documents.

Every year we were growing revenue by hundreds of millions of dollars. It was an incredible ride.

Reflecting on this success​

I want to be clear that Vistaprint's success was due to many critical innovations and an enormous amount of work by many, many people, in areas like manufacturing, content design tooling, marketing, ecommerce, etc.

Also: even though I had come up with the core idea for the new Studio, it was based on many of the ideas from the old Studio, which itself required a lot of independent innovations. Beyond that, there's no way I could have come up with this idea, or had any hope of making it work, without the insight and skill of my teammates who had figured out the server-side rendering.

What's more, none of the engineering work I did was groundbreaking or mindblowing. I just synthesized some disparate ideas, from both inside and outside Vistaprint, and glued it all together with some (fairly decent) JavaScript and C#. I was just the right person, at the right place, at the right moment, with the right idea.

Even so, I still occasionally catch myself dwelling pridefully on this achievement. I imagine an alternate universe where I never joined Vistaprint, where they tried to incrementally improve the old Studio architecture. I don't see how they could have had the success that they did in this universe; the difference might be measured in hundreds of millions (maybe billions?) of dollars at this point.

I've done a lot of things I'm proud of since then, but I don't know how likely it is that I'll ever play such a pivotal role in building a multi-billion dollar company again.

Innovation comes from individuals​

Thinking back on this episode in my career has been useful to remind myself that impact, especially via innovation, is ultimately driven by individual contributors. This is really important to remember for those of us who've chosen to optimize our careers around the joy of being a technologist, especially when the social and financial pressures to advance our careers through management are so potent.

My contrarian thesis aside, I have to acknowledge the complex interrelationship between ICs and managers when it comes to innovation. Technology leaders play their part to drive innovation by actively building a culture of empowerment and risk-taking, and being willing to make big bets on individuals with vision.

Perhaps it was only implied in my story, but this is exactly what Wendy had done for Vistaprint, long before I had arrived. She built an amazing team, and fostered a culture where engineers felt supported, trusted, and safe enough to invest their time where they thought opportunities existed.

I feel a great deal of gratitude to Wendy for having taken a risk in empowering me. It was a truly formative experience, and I still cant believe how lucky I was that the work I did had such a lasting effect on a great company.

Over the past 15 years, I've been working on one form or another of internal developer platform. Even long before, while working at small startups, I inevitably ended up building (or curating) some little web framework, a build system, and slapping together scripts to package and deploy our stuff reliably. No one ever told me to do this, it was just obviously necessary.

In these cases, I was building a product for myself and my immediate team members, so it was a pretty tight feedback loop with the customer. I'd put a little extra effort to make things nice for other developers on my team, and also out of a bit of pride in making something that felt elegant.

Developer experience in the monolith​

At the first larger company I worked for, I worked on improving the developer (and user) experience on top of a giant, pre-existing monolithic app, with a lot of custom tooling. One needed custom tooling when dealing with a monolith of several million lines of code being concurrently modified by 200 developers, especially since there wasn't really any off-the-shelf tooling available that could handle this scale.

Since there was already an established build and deployment system, I was mostly focused on improving the experience of web developers. At that time, the challenge was mostly around providing a sprawling army of mostly backend developers with a decent library of web UI components, and achieving some semblance of brand consistency.

This whole thing required some culture change, and a lot of outreach. I had no power to enforce usage of our web framework, nor any power to force web designers to work within the constraints we'd defined together. To get the designers on board, we needed to build some trust, listen to their concerns, and help them see we were trying to help them realize their vision with greater fidelity.

For developers, it just required that our framework was easier and better at helping teams make their pages look like what the designers wanted. Ultimately, no one ever had to force anyone to use our framework, it just made things easier for everyone, so they did it.

The next time we had to do a brand refresh of the site, it only took a couple people a week or so, whereas the last rebrand had been a major project across the whole company that took months. This was a small win against the entropy which was slowly devouring our monolith.

Microservice babies​

A few years later, it was becoming apparent that we had been gradually losing productivity in our monolith, and there were some factions interested in pursuing a service-oriented architecture. A new platform team started working on a set of tooling to enable teams to stand up independent services outside of the monolith.

In a marked contrast to the proprietary infrastructure we'd been using for the monolith, they were toying around with a bunch of different open source and vendor tools. After some prototyping and getting an initial MVP built, some teams started using their stuff.

Unfortunately, the fate of this particular platform was to fizzle. In retrospect, there was a lot going against it:

• We weren't yet using a public cloud (they were targeting on-prem infrastructure)
• Kubernetes and containers were in their infancy
• We were legitimately deluded about what it would take to make microservices actually work. Seriously, we were like little babies.

These were pretty strong headwinds, but there was another factor I can see in retrospect, which was even more critical:

The platform team didn't spend enough time learning from their customers, or trying to understand the actual problems they were facing.

I remember several specific tales of teams working on building services outside our monolith using their tooling, running into some friction, and experiencing something other than empathy from the platform team. At least once I remember a VP getting involved to put pressure on a team that was expressing reservations and looking for alternative approaches.

Building a great platform for the wrong customer​

What was wrong with the product? Let me explain:

The toolkit was built as a set of Ruby gems, referenced via a root gem that composed them to enable some higher-level operations. Each gem was responsible for interacting with some of the platform's parts, such as Artifactory, Jenkins, or whatever deployment tool we were using (I think at one point it was Octopus Deploy?). The tool would scaffold out a rakefile, with predefined tasks (e.g. build, deploy) the user could execute with the rake CLI. The user could then customize their rakefile, combining these gems to implement all the custom processes their service needed, but a lot of the low-level details would be taken care of within the gems.

Here's where the problems started: the gems were fairly course-grained, strongly opinionated, and had pretty limited extensibility. They were also composed and packaged in a way that made it hard to replace any single gem with an alternative implementation. The options for a user that had an unsupported use case were pretty limited:

• Get the tooling owners to implement the feature
• Implement the feature and try to convince the owners to take the patch
• Implement the feature from scratch outside of the pre-existing gems

All of these options were particularly unappealing, partially because there was virtually no Ruby experience to be found among our developer population. But more importantly, after a number of teams' feature requests were met with apathy (and a bit of paternalistic "you're doing it wrong"), the "brand" of the tooling began to suffer.

Despite strong top-down pressure to use the tooling, teams openly rebelled and began piecing together their own bespoke solutions. In the long run, management gave up the fight, because ultimately, they just cared that business problems were getting solved.

In retrospect, I think leadership's decision to push the tooling was its death knell. The platform team lost its incentive to win the trust of the developers, and got caught up in their own vision. They built a beautiful product, it just didn't happen to be the product we needed.

Kubernetes emerges from the chaos​

A few years and a regime change later, we had a whole lot of teams individually managing their own build/deployment tooling. This was, in no small part, a reaction to the bad experiences many of us had with the aforementioned platform team. It seemed obvious to me at the time that there was a lot of waste in having every team have to rediscover their own solution, but I also acknowledged that the alternative of a central platform team managing this for everyone hadn't worked so well last time.

Meanwhile, management had blessed adoption of AWS. At the time, we had a vague and naive idea that AWS was a ready-to-use platform. and hadn't yet come to terms with its true character: an extremely powerful, but low-level set of primitives. They had a few offerings at the time that looked a little like a PaaS if you squinted, but we seriously underestimated the amount of boilerplate glue scripts we had to write to, for example, get a service built and deployed on ECS or Elastic Beanstalk.

One team in particular had been toying around with Kubernetes and was having some success. While I'd used docker a bit, and had been following the orchestrator wars (mostly as a lurker on Hacker News), I didn't yet see what the big deal was. But smart people I respected were saying good things about it, including words I liked, like "rolling deployments", "autoscaling", and "self-healing".

I had just spent the previous 2 months trying to help another team, who had been struggling to execute a basic blue-green deployment with CloudFormation. Then I saw a demo in which a kubectl apply of a single deployment.yaml file executed a seamless rolling update of a service within a minute, and I was sold.

As I learned more about the abstractions Kubernetes was built around, my thoughts returned to the idea of creating a developer platform. It seemed possible that containers and the Kubernetes API might be the membrane we needed to give developers autonomy over all things they cared about, while enabling central management of the stuff they didn't. The ingredients of the devops stew were finally all out on the counter.

It took some convincing, but I managed to get some of the influential developers on board with the idea that we'd create a new platform team, and attempt to build a scaled-up, multi-tenant version of the Kubernetes solution they had pioneered. We started the team, and spent most of the first month learning how to build and operate a cluster with kOps (EKS either didn't exist yet, or was too new to consider seriously).

We got a couple of the teams to try it out, and found that it was, indeed a Kubernetes cluster; it allowed us to define workloads and roll them out reliably. This was a huge improvement. But it didn't take long until the teams using it had accumulated a bunch of shell scripts and additional tooling to manage a few other things:

• Authentication to Kubernetes, Artifactory, and other services
• Running docker builds (passing in build args, ssh-agent sockets, managing cache volumes, etc)
• Defining per-environment configurations
• Syncing secrets between our secret store and Kubernetes
• Managing load balancers, DNS, and certs
• Orchestrating integration tests with a bunch of docker containers

Once again, each individual team was re-solving the same problems, each with their own flavor of tradeoffs and bugs. Clearly, we had a lot more opportunity to provide value here.

Coincidentally, I was fighting a little burnout around this time, and ended up deciding that 13 years was long enough in one company. I never got to take this particular platform further, but the shape of the problem space had become a lot clearer in my head.

Three developer platforms later, lessons learned​

Over the next 7 years, I've iterated on this idea three more times at two different companies, all built on Kubernetes. The results have been increasingly compelling with each iteration, and I've added a lot of key elements to the approach. The central idea has become:

The platform encapsulates the operational, cultural, and security opinions of the organization, gluing together the company's chosen infrastructure and tooling.

There are a lot of principles and patterns underneath this high-level idea, but there are a few, universal key dimensions along which you have to strike a balance:

• Finding the right line between things that have to be standardized, and things where there's value in flexibility and autonomy for teams.
• Adding enough power so that the platform can support all the use cases in your company, while also having a small number of simple, default paths that work for the vast majority of cases.
• Creating the right extensibility points, allowing teams to solve their own problems without the platform team being a bottleneck, but still maintaining enough coherence in the core aspects of the platform so it can evolve and improve over time.

The right balance in these dimensions is highly dependent on the culture and values of your organization, but there's one thing I'm pretty sure is universal, which is how you find that balance:

Treat your developer experience like a product.

Finding product-market fit​

I don't think this is any different than a startup would approach things:

• Observe your developers, listen to them, learn about their pain
• Formulate hypotheses about how you can alleviate that pain
• Build a minimum viable product
• Get it in front of developers
• Listen, learn, and iterate
• Pivot if what you're building doesn't resonate

Don't fall in love with your own vision. When developers ask for a feature, don't dismiss them, even if you don't see where it fits on your roadmap. Regardless of the implementation details they may be stuck on, they're giving you critical information about their pain points. Don't squander that opportunity.

If you're exceptionally visionary, you may have innovative, paradigm-shifting ideas for solutions that developers don't even know they need. That's great, but you should slow your roll. Use the scientific method: test and learn. Maybe you have the wisdom of Solomon Hykes, but the odds are against you. In reality, 99% of ideas you think are novel aren't actually new, they just got quietly discarded because they didn't work in practice.

For a internal developer platform, you don't have to be particularly innovative, and you certainly don't have to be original. In fact, it's usually a lot better to shamelessly steal ideas from successful platforms outside your company. Bias towards open-source tools, vendor or cloud-provider products. Rip off the CLI interface of a popular PaaS product your developers are already using for their side hustle.

Congratulations, you're a brand manager​

And like a startup, you're also the steward of your product's brand. You have to earn trust with your customers, show them that you're listening to their feedback, and that you're committed to making their lives better.

Your brand is also relevant to stakeholders besides your direct customers, including leadership, security teams, product owners, etc. If they don't understand your value proposition, they'll a good chance they'll be asking uncomfortable questions at a moment when its least helpful.

Remember, you don't have the monopoly you think you do​

In a few cases, especially with the latest developer platform I've worked on, I've had to fend of requests from leadership who'd like to accelerate adoption by cranking up pressure on teams to use our stuff. Certainly, there are benefits to the organization in standardizing (especially for security and cost management). But each time I push back.

For one thing, we haven't needed to do anything to drive demand; teams are migrating services whenever they can spare a sprint... because they like what we've built and they know they have a say in the direction we take it. We're going to get to 100% adoption at some point soon, and we won't have ever forced anyone's hand.

I think this principle is pretty universal for teams working on internal tooling. When you're tempted to use management to force people to use your product, step back and consider the big picture. You don't have the monopoly you think you do. Companies evolve and change; new executives and managers come into power, technologies evolve, and the business climate changes.

If you want to stay on top, you have to acknowledge that you're always competing for your customers' business. If they're happy with the platform, and trust you to keep improving it, they'll defend you against shifting tides. If they're not, they'll abandon ship as soon as another option presents itself.

At work, we're in the process of rebuilding our metrics pipeline, as we've outgrown our old self-managed TIG (Telegraf, InfluxDB, Grafana) solution. We've had this solution in place for many years, and it's served us well. Especially given the increasingly predatory pricing models of observability vendors, it's been extraordinarily cost-effective.

But over the last couple years, as we've grown, we've started to hit the limits of what we can handle with a single, vertically scaled instance of InfluxDB (especially using InfluxDB v1). It was increasingly stressful to keep it running smoothly, and we had to be very vigilant about cardinality, as it's very easy to accidentally introduce a cardinality explosion that can bring down the entire database.

Just upgrading InfluxDB would have been similar in scope to moving to a new vendor, since InfluxDB v2 has a new query language, and we would have had to rewrite all our queries and dashboards anyways. So we decided to take the opportunity to do a proper RFP, and see if we could find a vendor to take this entire problem off our plate.

The landscape​

We decided to focus specifically on metrics, since we needed to limit the scope of our evaluation by some criteria. Observability products vary drastically across many dimensions, and metrics were the area where we were in the most pain.

The vendors we considered fell into a few categories:

The dominant players​

These include both Datadog and New Relic, which are both well established and very feature-rich. They're also known for being extremely expensive, and having pricing models that are difficult to predict or control. I've talked to some friends who've worked with them, and they said although they were great products, it was pretty typical that the cost would be 30% over an already bloated budget every year. But because they were so locked in, every year they'd have to renew, and the sales people would show up looking for another pound of flesh.

Another thing we noticed about the dominant players was their transparently conflicted relationship with Open Telemetry. While OTEL support features prominently in their marketing materials, their documentation tells a different story. Customers who choose to instrument their systems with OTEL SDKs will find that they're missing a whole lot of the best features of these products. The sales folks were not exactly subtle about recommending we run the evaluation using their proprietary agents instead.

Yuck on both fronts.

The up-and-comers​

There's a few interesting, smaller players in the space. We looked at SigNoz, Logit, and a few others. They all appeared to be offering basically the same thing: a hosted, Prometheus-compatible backend along with a Grafana-based front end. They all had very competitive pricing, but we felt a bit concerned at how immature they were, and decided against doing a full evaluation.

The cloud-provider option​

Since we're an AWS shop, we also considered using AWS's Managed Prometheus offering, which would have simplified some of the operational complexity of running a Prometheus backend (e.g. Thanos or Cortex) ourselves. Doing some back-of-the-napkin math, we realized that, if we didn't do anything differently, we'd end up spending about 3x what we were currently spending on InfluxDB. Plus, we'd still have to manage our own Grafana instance.

The goldilocks zone​

We also looked at a few vendors that were in the middle of price distribution, such as Chronosphere and Grafana Cloud. These were also Prometheus-compatible, with Grafana front-ends, but both companies were reasonably established, and had similar looking feature sets.

Chronosphere was the first vendor we decided to evaluate, because their sales pitch included something we hadn't heard from any other vendor; they'd provide a way for us to manage costs with powerful, centralized ingestion controls, an as a result, could offer us predictable pricing.

This piqued our interest, not just for managing costs, but because we'd long had problems with cardinality. At any moment, cardinality from any given service could unexpectedly explode- based on the decisions of a single programmer. For example, we've occasionally had instances of programmers inserting a metric label where the value is a unique ID (such as a customer ID), which could have hundreds of thousands of possible values. Previously, we had to be extremely vigilant about this, and pounce on any team that introduced cardinality explosions before they could bring down our InfluxDB backend.

So having a way to manage cardinality, centrally, was very enticing.

Evaluating Chronosphere​

We decided to proceed with a PoC of Chronosphere. We started with some changes to our metrics pipeline infrastructure, adding OpenTelemetry Collectors to help capture and redirect our current metrics data (which was coming mostly from telegraf), so that we could send metrics to both our in-house InfluxDB and Chronosphere concurrently.

We had, as part of a previous set of experiments, already set up some common Prometheus metrics infrastructure in our Kubernetes clusters, including kube-state-metrics, node-exporter, and cadvisor. We were able to easily point these at Chronosphere as well.

The sheer volume of metrics​

The first thing we realized was that we were sitting on an enormous amount of cardinality. Chronosphere reported that we were generating over 8 million active series, and our sales engineers were a flabbergasted about how we were even able to handle all of it with a single InfluxDB server.

The Chronosphere control plane​

Our sales engineers immediately got to work helping us learn how to use their "control plane" feature, which allows you to write fairly arbitrary rules which can select metrics by virtually any criteria (names, label values, and/or combinations based on boolean expressions), and perform complex transformations on them, including:

• Drop them entirely
• Aggregate away high-cardinality labels
• More complex transformations, such as changing the temporality of the counters (e.g. change a "delta" counter to a "cumulative" counter)

It was immediately clear that their control plane was extremely powerful. We did a bit of analysis on the highest cardinality metrics coming in, and by cross-referencing them with a JSON export of our existing Grafana dashboards, were able to create a relatively small number of rules that reduced our cardinality by about 60%. It took a bit more work to go farther, but we eventually got our cardinality down to about ~1.7 million active series.

At this point, since we had a handle on our active series volume, Chronosphere's sales folks gave us an initial price, which turned out to be very reasonable.

Holy cow, it looked like this just might work!

Using Chronosphere​

Once we had addressed the initial concerns around affordability, we got to work evaluating the product's overall fit. We had a bunch of teams convert some of their InfluxDB-backed dashboards and alerts over to Chronosphere, and started to get a feel for how it would be to use it day-to-day.

Since Chronosphere's UI was based on Grafana (v7), it turned out to be very similar to our self-managed Grafana/InfluxDB from a developer perspective, with the main differences being:

• The PromQL language
• Much better query performance

After a few weeks of playing with the product, we were satisfied it would do the job. We gave it the thumbs up.

Evaluating Grafana Cloud​

Initially, we had sort of written off Grafana Cloud, since the price they gave us originally, based on our active series in InfluxDB, was in the same range as New Relic and DataDog. However, this was before we realized that they had a feature that was similar to Chronosphere's control plane, called Adaptive Metrics.

We told the Grafana sales team that, using Chronosphere, we'd been able to reduce our metrics to under 2 million active series, and asked for a new quote based on the assumption we could use Adaptive Metrics to get similar results in Grafana Cloud.

They came back with a price that was almost exactly the same as Chronosphere. The race was back on!

Using Adaptive Metrics​

Once we updated our metrics pipeline to export to Grafana Cloud, and had a chance to start playing with Adaptive Metrics, we were disappointed to find that it wasn't nearly as powerful as Chronosphere's control plane. The biggest difference was that you could only target metrics based on their names, and not their labels or values. This was a big limitation, as we had a lot rules we had written in Chronosphere that did things like:

• Drop all metrics from a specific service, except for a few key ones
• Drop a particular metric generated by a telegraf plugin (e.g. procstat or diskio), but not for services in an "allowlist"

But we really, really liked Grafana Cloud​

Aside from cardinality management, where Chronosphere clearly had the lead, we found a lot of areas where we preferred Grafana Cloud:

• They had a more modern, polished user experience (both used Grafana as a front-end, but Grafana Cloud has the latest version, while Chronosphere's was pinned to v7, which is very old)
• Their documentation was significantly better
• They had support for multiple data sources, including CloudWatch, ElasticSearch, and Athena (which were important to us)
• They were strong leaders in the open source observability community
• Grafana Labs was a larger and more established company, with a more robust and mature product portfolio
• It seemed credible that we may eventually be able to migrate our traces and logs to them as well, giving us a unified observability platform

It was clear that, besides the discrepancy in cardinality management, we'd prefer to go with Grafana Cloud. However, if we wanted to make this work, we'd need to find a way to handle the use cases that adaptive metrics wouldn't cover.

Taking another look at the OTEL collector​

It turns out that the OTEL Collector (which I mentioned we were already using) is an insanely useful swiss-army knife for building observability pipelines. It can collect metrics, traces, and logs in virtually any format, run a pipeline of transformations, and output them in virtually any other format.

I knew that the OTEL collector had a number of processors available, though we hadn't used them much previously. I wondered if we could use these to replicate some of the more advanced metrics selector functionality that Chronosphere offered.

It took me a bit of time to figure it all out, mostly because the OTEL collector documentation isn't amazing, but I was eventually able to replicate pretty much all of the advanced "drop" rules we needed using the OTEL collector's processors.

In the end, with the combination of Grafana Cloud's adaptive metrics and the OTEL collector processors, we were able to get our total cardinality down to a similar level as we had with Chronosphere. While the resulting solution was a bit more complicated, it was an acceptable tradeoff given the other advantages of Grafana Cloud.

Conclusion​

The experience of running a head-to-head evaluation of two vendors, especially given the penetration of OpenTelemetry and Prometheus in the market, was a real eye-opener. I'm more bullish than ever on OTEL (and cloud-native standardization initiatives in general), and I think its going to continue to reshape the observability landscape in the coming years.

I should point out that, even though we selected Grafana Cloud, I think Chronosphere would have also been an excellent choice. I think it might even be a better choice for a company that meets a few criteria:

• Your biggest pain point is cardinality and/or cost management
• You have a large number of metrics producers that would be hard to corral into a uniform schema
• You don't have a lot of third party metrics sources (e.g. CloudWatch, ElasticSearch) that you want to query directly (Chronosphere integrates with those data sources by eagerly scraping them and converting them to Prometheus metrics... which can increase costs for sources, like CloudWatch, that charge by the API call)
• You're OK with a slightly less polished user experience (or you're willing to wait for Chronosphere to catch up)

That said, Grafana Cloud has been a great fit for us. Their support and customer success teams, in particular, have been really effective in helping get our team ramped up and successful. Given this experience, we're interested in expanding our use into their logging (Loki) and tracing (Tempo) products. I'll let you know how that goes.

Addendum​

I reached out to both Grafana Labs and Chronosphere with a draft of this post. I'm glad I did, because Chronosphere let me know that due to feedback like ours, they've been investing in some of the areas in which they were weakest relative to Grafana Cloud, namely UI quality:

• https://chronosphere.io/learn/chronosphere-bolsters-oss-contributions-for-perses-otel/
• https://chronosphere.io/resource/chronosphere-lens-solutions-brief-pdf/

They're the primary force behind Perses, which a competitor for OSS Grafana (which Chronosphere was previously using for visualizations). They weren't specific about the details, but my guess is the monolithic design of Grafana, combined with its AGPL license, limited their ability to integrate it effectively into their product without having their proprietary UI be infected with the AGPL redistribution terms. Perses is permissively licensed (Apache 2) and backed by the Linux Foundation.

It looks like it's designed to be modular and embedable, as well as be more IaC/GitOps-friendly than Grafana. The project is very young, but I'm excited to see some more open-source visualization options available.

As part of an evaluation of Prometheus compatible monitoring solutions, I found the need to push our use of the OTEL Collector to handle some use cases like creating metrics allowlists, renaming metrics, or adding and modifying labels.

Here's some examples, based on what I learned, of the crazy and powerful things you can do with OTEL collector processors to manipulate metrics.

Inserting static labels​

As part of a multi-account AWS strategy, we have many Kubernetes clusters, spread across AWS accounts for each of our teams. We wanted to make sure that all metrics coming from Kubernetes clusters contain labels with metadata about which cluster and account they came from (beyond what comes with the k8sattributes processor).

We use the OTEL collector as a daemonset (so it runs on all nodes in our clusters), and all telemetry from our pods goes through them.

attributes/cluster-metadata:
  actions:
  - action: upsert
    value: "${CLUSTER_ENV}"
    key: env
  - action: upsert
    value: "${CLUSTER_LABEL}"
    key: cluster_label
  - action: upsert
    value: "${CLUSTER_NAME}"
    key: cluster
  - action: upsert
    value: "${CLUSTER_TEAM}"
    key: team
  - action: upsert
    value: "${CLUSTER_REGION}"
    key: region

Inserting dynamic labels​

We have a bunch of legacy services, deployed outside Kubernetes, that don't have an instance label (which is idiomatic in Prometheus). These metrics (generated by telegraf) do have a host label, however, so we used that to create an instance label, also using the attributes processor:

attributes/instance-label:
  actions:
  - action: insert
    from_attribute: host
    key: instance

Replacing useless labels with useful ones​

When we scrape kube-state-metrics, the pod and namespace labels on the metrics are the pod and namespace of the kube-state-metrics pod itself. This isn't so useful; we don't care about the kube-state-metrics pod names, we only care about the pods that are the subject of the metrics.

Here's a trick where we use the attributes processor to remove the kube-state-metric pod/namespace labels, and then rename the exported pod/namespace labels to replace them.

This way, when users are querying for metrics on their pod, they can just use the pod label, and don't have to worry about the implementation details of how kube-state-metrics is scraped:

# Delete the pod and namespace labels which refer to the kube-state-metrics
# pod itself, not the pods the metrics refer to.
attributes/kube-state-metrics:
  include:
    match_type: regexp
    metric_names: "^kube_.+$"
  actions:
  - action: delete
    key: pod
  - action: delete
    key: namespace

# Rename the exported_pod and exported_namespace labels to pod and namespace
metricstransform/kube-state-metrics:
  transforms:
  - include: "^kube_.*$"
    match_type: regexp
    action: update
    operations:
    - action: update_label
      label: exported_namespace
      new_label: namespace
    - action: update_label
      label: exported_pod
      new_label: pod

You'll need to make sure that the attributes/kube-state-metrics processor runs before the metricstransform/kube-state-metrics processor in your pipeline, so that the old labels are deleted before the new ones are renamed.

Renaming metrics​

Sometimes, we'd find older services had metrics that had been named in various problematic ways, so we wanted a way to rename metrics (e.g. to adhere to a naming convention). Here's a use of the metricstransform processor that renames all metrics with a badsuffix to have a goodsuffix instead:

metricstransform/fix-suffix:
  transforms:
  - include: ^(.*)_badsuffix$
    match_type: regexp
    action: update
    new_name: "$${1}_goodsuffix"

Truncating long label values​

Grafana Cloud has a maximum label length of 1024 characters. Any metrics with labels exceeding this length will be dropped before they're ingested. Here's a nifty transform that truncates all label values this length:

transform/truncate-labels:
  metric_statements:
  - context: datapoint
    statements:
    - truncate_all(attributes, 1024)

Filtering metrics with arbitrary queries​

Here's where we get to the use case that we really needed: a way to drop large swaths of metrics entirely, based on arbitrary queries. The most common pattern we were trying to replicate was an "allowlist", where we drop most metrics, except for those that meet some criteria.

The OTEL collector has a filter processor to do this, and it supports the OpenTelemetry Transformation Language (OTTL), which allows you to write complex expressions to represent your filter criteria:

filter/drop-rules:
  error_mode: ignore
  metrics:
    datapoint:

    # An "allowlist" that drops all metrics from 'some-service' except for
    # the two specified.
    - >
      resource.attributes["service.name"] == "some-service" and
      metric.name != "some_service_important_metric" and
      metric.name != "up"

    # A similar allowlist, but using a regex to match the service name
    # for various sized fluent-bit daemonset pods (we have small, medium, 
    # and large variants of fluent-bit).
    - >
      IsMatch(resource.attributes["service.name"], "fluent-bit-.*") and
      metric.name != "fluentbit_output_dropped_records" and
      metric.name != "up"

    # A drop rule that drops a specific cadvisor metric for all services
    # except for those in a specific namespace.
    - >
      resource.attributes["service.name"] == "cadvisor" and
      metric.name == "container_file_descriptors" and
      (not IsMatch(attributes["namespace"], "^someservice.*$"))

    # A drop rule that shows you can use more complex boolean expressions
    # with parentheses to group conditions.
    - >
      attributes["telegraf"] == "1" and (
        IsMatch(metric.name, "^internal_(agent|gather|memstats|serializer|statsd|write)_.*") or
        IsMatch(metric.name, ".+[-_]request[-_]metrics[-_](median|sum)$") or
        IsMatch(metric.name, ".+_stddev$")
      )

    # Another complex drop rule, where we're dropping metrics matching a
    # regex for all services, but with a list of exceptions.
    - >
      IsMatch(metric.name, ".+[-_]request[-_]metrics[-_]upper$") and not (
        attributes["service"] == "service1" or
        attributes["service"] == "service2" or 
        metric.name == "inconsistently_named_service_request_metrics_upper"
      )

Limitations of the OTEL collector​

It can't do aggregations​

While the OTEL collector has many powerful processors, it doesn't currently have the ability to do aggregations (i.e. drop a particular label from a metric and create a new metric by aggregating the metrics that had that label). This is a much harder problem to solve than just dropping metrics, since all the OTEL collector instances that could process any metric you'd want to aggregate would need to coordinate with each other, creating some scaling challenges.

Both Grafana Cloud and Chronosphere offer powerful features around metrics aggregation.

It can't convert delta to cumulative counters​

When I evaluated Chronosphere, I was delighted to find that it had a feature to change the temporality of metrics (e.g. change a "delta" counter to a "cumulative" counter), and I was hoping to replicate it with the OTEL collector. While the OTEL collector does have a converter processor in the works, it's still early in development.

In our case, we were able to work around this with some hackery in telegraf (setting the delete_counters setting of the statsd plugin to false).

Wait, you need some background first​

Let's back up for a minute... It was 2000, and I had been hired for my first actual job as a programmer at a company called Corex (makers of CardScan, a business card scanner). At this point I had a few years of employment under my belt, but this was as a graphic designer who did a little programming on the side. I was pretty clear about this in the interview for Corex, but I guess I did well enough on some programming problems (or there was some misunderstanding?) that my new boss was reasonably confident I could grow into a programmer who did a little graphic design.

They dropped me onto a team comprised entirely of people who had gone to engineering schools, wrote C++, and used words like "orthogonal". They gave me a web-oriented project, put me under the watchful eye of a cranky Russian PhD who would write COM objects that I could script against, and left me to decide how to glue this all together with a true gem of 2000-era web tech: ASP and VBScript.

I crammed books on programming most nights in bed, trying desperately to incorporate some high-level theory to make sense of the trial-and-error hacking I was doing during the day. The feeling of barely keeping myself from drowning eventually gave way to the sense of floating; I was making real, tangible progress, and I was having a ton of fun using a computer to put pixels on a screen.

Three years into this job, I got the opportunity to join a startup that had recently spun out of Corex called ZoomInfo (they're still around). I'm not 100% certain exactly what number employee I was, but I was definitely in the first ten. Most folks on the team were roughly the same age as me, but like the previous job, they had all studied computer science, and knew things like whatever a "heapsort" is for. They were all C++ slingers too, and once again, I was the rookie who was hanging with the pros.

OK, so back to 2004 and that giant chip​

At this point, despite my questionable background, I'd earned enough trust to be put in charge of a small team working on our B2B products. The main product was also built with ASP (this is still classic ASP; .NET existed, but was still the bleeding edge) with an era-appropriate amount of CSS and JavaScript. The app got its data from a massive SQL Server that had been populated by web crawlers, using some natural language processing. This was some legitimately groundbreaking stuff, written by a French Canadian genius whose brilliance probably made the rest of the team feel like, well, how I felt around all of them.

The core UI of the product was pretty simple: you'd search for people by attributes (what industry they were in, what titles they may have had, what their field of expertise was), get a list of results, and click on one of them to get to the detail view.

I had inherited the implementation of this from some more senior members of the team a year previously, and had been slowly dolling it up, adding features and generally just making it look spiffier.

One thing had been bothering me since the first time I used the product, though: this detail view page took at least six or seven seconds to load, at a minimum. I mean, the page had a lot of great stuff on it, but it got annoying waiting for it on every click. It didn't seem to be a huge deal to anyone on the dev team, but the sales people occasionally grumbled about it.

The page was built (as lots of ASP applications were at the time) out of C++ COM components, glued together into a UI with VBScript. With some very crude debugging, I could see that all the time was being spent in the C++ code, which I still had very little ability to wrangle.

I went to my boss (who may have been one of the original authors of said C++ component.. I'm not 100% sure), and asked about the performance bottleneck. He told me that C++ was the fastest, most efficient, and powerful language we had available, and that there was really not much that could be done to make this component any faster. The real engineers had already taken their pass at this, so I should let it go and get back to delivering that feature the sales team wanted.

This felt like a dam breaking; at this moment, I felt like I had endured the subtle condescension of these "real" engineers for 4 years now, and despite how much I had grown, I realized they were always going to view me as 2nd rate.

Programming out of spite​

I knocked out the feature the sales team wanted within an hour of this conversation, and instead of going back to JIRA for more work, I made the deliberate decision to carve out some time to prove my boss wrong.

First, I had to decipher this god forsaken C++ code. I was able to hack in some printf statements to log each SQL query as it ran, compiled it, and within another hour, was able to replicate the data access pattern in an interactive SQL Server query window. As expected, these queries did indeed take about 6-7 seconds to run.

This component had a fancy, object-oriented design, which elegantly encapsulated each row as an object which was responsible for fetching its own data. The effect was it was hitting the database like a machine gun- running a separate query for every row in the grid it was rendering. In most cases it was running like 60 queries to render this page.

So I concocted 4 very, very, dumb queries to retrieve all the same information that the C++ component did, and ran it in the SQL Explorer. After a little bit of tuning indices, the whole thing ran in like 300 milliseconds.

At this point, I was pretty confident I was on to something, so I snuck a little time over the next couple days to wrap this all up into a VBScript function (with proper connection pooling) and replace the C++ component altogether. I fired up my localhost server, started clicking around from the search screen into the details pages... and it felt insanely fast!

Err, so what do I do with this now?​

At this point I was in a bind. In the sober light of day, I realized that a fit of rage, I had engaged in an unauthorized product improvement. I really wanted to show my boss, but I was honestly a bit afraid.

So I tested the water by showing a couple of my colleagues, who initially thought it was some kind of trick. Once they realized this was for real, there was no keeping the horse in the barn. Within a few minutes, my boss caught wind and came over to see what was going on.

So once I explained what I did, he was actually fairly impressed, and called the CEO over to see it too. The CEO was a legitimately intimidating Israeli guy (who had worked in the Mossad), but even he had a hard time moderating his delight over this unexpected gift.

Looking back, this was fairly trivial​

In retrospect, this was a really elementary little exercise of basic engineering. I've done many more difficult and interesting things since, but this one still sticks out in my mind, because it was my youthful impertinence that pushed me to do something no one else thought I could do... or even try to do.

20 years later...​

Fast forward to just a few days ago.

I had a meeting scheduled with a guy at work who I hadn't met before. The invite said he had some questions about Helm and Kubernetes, so I thought, perfect, I can help.

Right out of the gate, the meeting went a little sideways, when I realized he was asking me for help finding a way to deploy his app to our Kubernetes cluster without using the client tooling that my team had built for this purpose. He was generally skeptical of any internal tools, and assumed they must obviously be inferior junk, and would just get in his way.

It took me a second, since I was self-aware enough to recognize I was feeling defensive, so I took a deep breath and asked some questions about his app. After a few minutes of questions, we had a pretty good understanding what he wanted. Before I did any advocacy, I clarified that I didn't believe in forcing our tools on anyone, and that he was free to make any decision that made sense to him and is team. But I asked him to take a few moments to listen while I listed some of the problems that were solved in our platform tooling that he would have to replicate if he decided not to use it.

This included stuff like IAM integration, Service/Ingress integration with AWS load balancers, cross-platform docker builds, configuration management, ephemeral environments, and test orchestration. I gave him a chance to ask some questions, to help him understand what he was getting himself into. The tide turned a little bit when, while arguing that IAM integration shouldn't be so hard, he said he could just inject some (long lived) AWS credentials into his pods. At this point, one of his colleagues realized he was advocating doing something that was totally bonkers (and a violation of our security policy).

After this, opened up a little and we were able to figure out that a lot of what he thought about our platform tooling were misconceptions, and it actually did all the things he wanted. He agreed he'd start out with our stuff, and let us know how it went.

The nerve of that guy!​

It took me about an hour to unwind from that meeting. I was so miffed! Our team's platform tooling has been wildly successful, and its been a couple years since we needed to do any proactive advocacy for it. Demand has been spreading mostly by word-of-mouth, as our developer teams have been really happy with it.

I paced my kitchen while obsessing over the interaction. How impertinent! Doesn't he know that I've already solved these problems? He just casually dismissed all the work my team has done over the last 2 years! He thinks its trivial; he'll just whip out some shell scripts to solve everything. It can't be that hard.

Recalling the virtues of impertinence​

At this moment I remembered my own experience as an impertinent young programmer, and my mind began to settle. I realized he was offering me a gift: the perspective of someone who, however naive, might have ideas or insight that I was missing. It had been a while since I'd faced this kind of skepticism, and I realized this was a good thing- it's important to have someone keep you on your toes.

This was a reminder that impertinence can be a virtue: a fuel to do cool stuff. Hopefully the next time I meet an impertinent upstart programmer, I'll smile and keep my thoughts to myself.

Or wait... maybe I should be really condescending to get them fired up? I'm gonna have to think about that.

For most of my career, I've found that I tend to quickly develop a reputation in whatever company I'm working at. I've never been the best programmer, but I've got some breadth, creativity, and critical thinking skills, and I'm good at synthesis and communication. This has helped me see the big picture in moments where a novel idea was needed, and I was able to connect some talented people to come up with some cool stuff together.

The small pond​

Because of this, people have mostly taken me seriously within my companies. In my work as both a platform engineer as well as in engineering leadership, this has come in pretty handy. I do a lot of internal communications, mostly with developers, but also across organizations, including occasionally the C-suite, where the objective is often to influence behavior in some way. For example:

• I have to convince team managers to commit some of their team's time to test out a new observability vendor and get me feedback before a purchasing deadline
• I need to convince a product owner that if their team spends some time migrating their app to our new platform, it will pay off by making their team faster and their product more reliable
• I need to convince developers they need to adopt some new front-end standards, because mobile web is actually a thing now, and we need to make sure our site works on phones (lol, this one is a bit dated now)
• I need to convince the CTO that an assumption they made isn't correct, and we need to pivot ASAP

In all these kinds of communications, I usually start with writing. Not only does it help me clarify my own thoughts, but I feel like its a conscientious way to engage with someone when you're obviously trying to influence them. It gives them a chance to read, absorb, and process, without being put on the spot. Then it can be a lot easier to have subsequent conversations.

Writing is an even more essential tool for communicating to a larger audience, and interestingly, the same dynamic applies. Sending an email to a whole division of a company has a slightly different flavor than to an individual, but I've found that the more you treat it like an invitation for further conversation, the more effective it is at actually influencing behavior.

My inside voice​

So I ended up writing a lot of internal documentation, emails (and increasingly more multi-paragraph slacks), Confluence articles, and occasionally some slides for a presentation. In all of these, I've found a particular voice that feels appropriately authoritative, but also approachable and informal, with a bit of self-deprecating humor, and the occasional pop culture reference thrown in.

These communications have always been relatively well received, and usually effective. I've also found that developers tend to read my emails at a much higher rate than those from others (sometimes more than those from senior executives).

But at some level, I've always known that a big part of this is my reputation and pre-existing standing in the company, and I'm honestly not sure how big. It's easy to be confident about my communication when I know a big chunk of the people in my audience personally, and understand their context, concerns, and environment.

Finding my outside voice is a little scary​

I've been procrastinating on starting this blog for over 15 years. I wrote my first entry a few months before my youngest child was born... and she's now in high school. I have to admit it's partially because I'm a bit nervous about leaving my comfortable little corporate bubble where everyone knows me.

Part of me thinks all these "great insights" I've been wanting to share for years are going to get absolutely shredded in the daylight, given the tech world is filled with brilliant people, and on any single topic I think I know anything about, there's someone who knows a whole lot more. Perhaps I'm absolutely full of crap, or perhaps my insights are just boring and obvious.

Something changed in the past few months, and the urge to start sharing some ideas has finally overcome my fear of getting pilloried. I'm finally getting to the point where, however masochistic it may be, I'm really starting to crave external feedback. I want a way test out some of these crazy ideas in the open, where I can see how a more diverse and knowledgeable population reacts to them.

Mustering the energy​

The other obstacle was finding the time and energy to do the technical work to get this thing up and running. My site used to be on Vistaprint's website platform (which I helped build back in the day, so it was free for me), and had since been (forcibly) migrated to Wix. Frankly, working with Wix... did not make me happy.

At my current job, I finally found a static site generator that I like a lot for documentation, so that gave me the push I needed to export my stuff out of a languishing Wordpress, and get it published to Github Pages.

At some point soon I have to figure out SEO, and probably get over my disdain for social media and start putting myself out there.

Conclusion​

I remember as a kid, I was sometimes loud, and my teachers and parents would occasionally admonish me to use my inside voice. Perhaps it was good advice at the time. For me, though, I think it's finally time to find a voice for speaking outside.

Hopefully those mean kids across the street won't throw rocks at me again.

I've been reflecting recently on a really formative period in my career, when I had a chance to be part of a massive experiment in progressive engineering management.

About 3 and a half years before I left Vistaprint, I was asked to join the Engineering leadership team by our (relatively) new VP of Engineering, Erin DeCesare (who is now the CTO of EZCater). She was a particularly bold leader in terms of her progressive management ideas, and was rapidly reshaping the organization with a strong set of values around empowerment and servant leadership.

We were responsible for about 200 developers, who were organized into squads (a.k.a. two-pizza teams), who were then loosely grouped into "tribes" (an idea borrowed from Spotify). The real difference from the previous regime was a pretty extreme amount of autonomy given to teams; they could choose their own technologies, work processes, architectures.

On top of this, Erin was pushing farther into some even more progressive empowerment concepts. For example:

• Managers were given a lot of coaching on servant leadership, and the ones who weren't able to evolve were managed out
• Teams would be supported by embedded agile coaches, who helped them optimize team health properties, such as psychological safety and culture of feedback
• Teams were given the freedom to decide what work to pull from our enterprise backlog
• Teams would decide how to distribute bonuses between them (this one went a bit sideways)

All this stuff was a bit mindblowing to me, but I was doing my best to commit to making it all work. It was an extraordinary amount of cat-herding, managing through influence, and general chaos, but it felt crazy enough that it just might work.

We set up some initial organizational mechanisms to attempt to manage all this chaos, since the whole point was to build a better team that could deliver features and products that made our customers happy. We set up feedback loops, elevated thoughtful and visionary people, and set up structures to help make sane architectural decisions.

Then, things got very interesting.

Then we watched the experiments unfold​

We had something like 30 squads, each running themselves in almost any way they saw fit. There was a lot of diversity between teams; folks with different backgrounds, different technology preferences, stronger or weaker opinions, different seniority levels, etc.

What I witnessed was 30 different teams running 30 different experiments into what makes a team successful... or not.

Some examples:

• Some teams stuck conservatively to working in our monolith, and did a release every 3 weeks, others spun up Kubernetes clusters in AWS with KOPS and deployed their apps via helm charts several times a day.
• Some were absolutely religious about automated testing, and obsessive about code coverage, others had a more deliberate risk-management strategy.
• Some teams spent a long time getting observability working, and others relied more on signals from external SRE and QA.
• Some worked via consensus and mostly made decisions together, others had one or two very senior folks which set the direction for the team
• Some teams did all their work in pairs, or via "mobbing", while others only interacted with their teammates when they were stuck.

There was also a lot of variation in the adoption of the agile philosophies we were coaching the teams into adopting.

Things I noticed​

After running in this mode for about 9 months, there were a few themes I noticed which have really informed my perspective today:

Engineers will fill all available space with engineering work​

One of the "tribes" (4 squads, around 25 people) was given an objective, some constraints, and 6-12 months to deliver a new platform for managing our product catalog. We were replacing our "legacy" product system because it had grown too creaky and complex over the years, and we wanted a bunch of new features, especially for marketers to be able to manage content without needing engineering time.

Yeah, I know, this is a classic case of second-system syndrome, but everything just seemed so obvious to us at the time; we were smart people, and it seemed achievable to us. So we did some design, broke the work up and gave pieces to each of the squads.

As it turns out, our teams invented engineering problems to fill all available space. They created an elaborate network of microservices, built SPAs out of hip new JavaScript frameworks, integrated some truly abominable "enterprise" vendor products, and designed massively complex processes and workflows to solve every use case that we failed to account for in the legacy system.

It started to hit me that we had jumped the shark when I noticed that a microservice one of the teams built could have been implemented as a single text file. Once that clicked, I started realizing, to my horror, that everywhere I looked, the entire system was like this. And then, through the haze of the groupthink, it occurred to me:

We'd have been better off giving this entire objective to a team of 5 people for a month.

A small, constrained team would have had no choice but to build something small and simple that worked. Then they would have had to evolve it incrementally, which as we now all know, is the only way to build a working system.

Agile can be so powerful, or so horrible​

Some teams got really religious about their agile methodology (usually Scrum), and used agile "rules" as a weapon against their PO, manager, and occasionally each other. They'd spend a lot more time playing games with ticket management, ceremonies, and storypoints than they did thinking about customers, products, or using their common sense.

One of the best defenses against teams going this direction was having great agile coaches (NOT the high priests from big consultancies who spend their time promoting seminars on LinkedIn). A good agile coach can provide a sort of continous intervention for a team; they hold up a mirror, allow them see their own disfunction, and help re-center them on what matters.

From observing these coaches in action, I formed two separate beliefs:

• I've noticed that great agile coaches tend to also be very product-centered, and are genuinely interested in developing technical and domain knowledge from the team they're working with. They ask a lot of questions, and develop insights that are specific to the practical limitations the team is managing. They don't just waltz in and start unloading a bunch of dogma.

• Its interesting that when you see a healthy, high-performing team, agile methodologies are sort of invisible; they're there, but they sort of melt away into the background. The team is just focusing on the actual work: how to meet a customer's need, what hypotheses they should be testing, or thinking critically about business requirements in order to find an 80/20 solution.

The cult of test​

We had a few teams that drank the TDD kool-aid hard. One in particular had agreed as a team to take testing really seriously. They set about implementing extensive test suites, held book groups about BDD, and spent hours trying to compile Gherkin files so their business partners could define test cases (which they never actually did). They believed in the singular truth of the test pyramid, and went big on unit tests, shunning higher-level approaches as lacking in virtue. They set up CI/CD to fail builds when code coverage dipped below 95%.

Things started to go sideways fast. Because their test suites were targeting low-level implementation details, every feature they implemented broke a gazillion tests. They were terrified to refactor anything (including the tests) because it would ensure weeks of work and an avalanche of merge conflicts. They'd get nothing done, sprint after sprint. The result wasn't amazing quality, it was utter paralysis.

With some perspective and coaching, they started to re-think their testing strategy. They began to see that some risks were more important to mitigate than others, and some testing techniques gave you a lot more bang for your buck. They came to the conclusion that 95% on a code coverage report wasn't a business objective, since we were building a freaking e-commerce site, not pacemakers or cruise missiles. They gradually woke up from their fog, and came up with some much more clever testing strategies, such as testing the high-level interfaces and APIs, which were far more stable, and gave them the freedom to refactor implementation details.

Moderately strong teams outperform superstars surrounded by a meh team​

We had a few squads where 4-5 (out of 6) team members were solid "A-" or "B+" players. In contrast, there were other teams held together by one very strong "superstar" lead surrounded by "B"s and "C"s. It was strikingly obvious to everyone that the teams with the more homogenous, moderately strong members significantly outperformed the teams with the superstar.

My take on the underlying cause was that the superstar would get randomized trying to support the rest of the team, and didn't have enough time to do anything innovative. If they ever did manage to spend some time on something interesting, it was generally too advanced for the team to run with, and the momentum fizzled. The rest of the team became helpless and dependent on the superstar to make decisions.

Of course, there was occasionally the more enlightened superstar, who would spend their energy trying to elevate their lower-performing team members. The effectiveness of this was highly dependent on the latent potential of the rest of the team, and usually didn't work without management intervening to shore up the team with additional strong engineers to support the superstar.

In general, the ratio of strong vs weak really can't dip below 2/1 before the team veered into unhealthy territory. The members of strong teams will support and help each other, but there has to be balance between them; if it gets asymmetrical, it starts to drain everyone.

Strong POs are critical​

We had Product Owners deployed to each squad, usually a PO would cover 2-3 squads. The difference between a great PO and a bad one was very stark. POs ultimately decide what the teams work on, so it seems fairly obvious that they'd have an outsized influence on the team.

At the core of the role is having great instincts about customers, the product, and the problem space. But there are other key factors which are underappreciated in POs:

• Empathy and rapport with developers
• A willingness and interest in understanding technical limitations and tradeoffs
• An understanding that they're not only responsible for the customer experience, but the long-term health of the technology it's built on

Looking back​

I imagine for most people working there at this time, being the guinea pigs in a giant experiment might not have been the ideal work experience. Actually a lot of folks thrived, and did some amazing work. But some reeled from the unrelenting waves of change, and others understandably just threw in the towel.

For me, though, it was very different. I got to see all this from a birds-eye view, but also on the ground, since I spent time with nearly every team. It was a bit like watching a whirlwind from inside- it gave me a sense of how seemingly small, well-meaning and thoughtful inputs can have huge, unintended effects that ripple across the whole system.

Like a lot of younger engineers, I used to occasionally express casual and flippant disregard for out-of-touch upper management. OK, admittedly I still might feel this way from time to time, but this experience left me with a very different understanding of what it takes to manage a large engineering organization, gave me a dose of humility and appreciation for how challenging it is.

I'm incredibly thankful to Erin for taking me with her on this journey- I really couldn't have had more learning jam-packed into a few short years of my life.

When learning a new recipe, especially when dabbling in cuisine from different cultures, I find it really important to make sure one is really precise in their understanding the words used in the recipe. I've had a few unfortunate misunderstandings that resulted in... gastronomic disaster.

Similarly, I find that I can't responsibly use the word "DevOps" without testing that the person I'm talking to know which meaning I'm using. Here's some examples of what someone may think I mean when I say "DevOps":

• The whole category of stuff that happens after you do a git commit, that magically makes your code turn into running software
• A type of engineer that does the cloudy, opsy stuff
• A style of operations where there's lots of automation and infrastructure as code
• A culture where developers and operations people collaborate more tightly (vs the bad old days of "throw it over the wall")
• A style of software development where the software engineers figure out how to deploy and operate everything themselves

The thing I usually mean is similar to those last two, but here's a crisper version:

An engineering management philosophy in which teams are are responsible for operating the software they build, in order to create a virtuous feedback loop which incentivizes the team to make their software highly reliable and operable.

I think this is the most useful meaning of the word, mostly because a big percentage of the other meanings are covered by preexisting words, like "operations", "CI", or "infrastructure as code". I mean, yeah- there's definitely a specific modern flavor of operations, but I find it more useful to use more specific words about those practices.

Th DevOps-flavored stew​

The thing I find really interesting about the engineering-management philosophy definition of "DevOps" is how interdependent it is on a whole bunch of other ingredients that coincided historically with it:

• Microservices
• Cloud
• Automation and Infrastructure as Code
• Containers and orchestrators
• Continuous Deployment
• Shift to automated testing and observability vs manual QA
• Platform Engineering

Microservices​

About 10 years ago, the company I was working for had outgrown our monolith, and we reluctantly started on the journey to microservices, and the journey was still ongoing when I left (about 6 years ago).

The thing that surprised us first was the sheer magnitude of the overhead of managing all the operational stuff that had been solved problems in the monolith. The first teams that started extracting their own services spent many weeks just trying to replicate a small fraction of what we had previously taken for granted: reliable builds, rolling deployments, centralized logs, metrics, alerting, feature flags and associated tooling, etc.

Cloud​

At that point, we started toying with some public cloud (AWS), and found that there was a lot of excitement from teams using it. The fact that their infrastructure could be fully automated through reasonable APIs alleviated a whole bunch of the pain we were feeling trying to automate deployments to on-prem servers.

Infrastructure as code​

After building out some of this cloud automation with shell scripts, we quickly discovered that we needed some more powerful ways to manage infrastructure as code. I think at that point we played with CloudFormation and some early Terraform. We were still struggling though, caught between the low-level (infrastructure-as-a-service) nature of EC2, and the relatively immature platform-as-a-service offerings AWS had at the time. We made a little headway with tools like Spinnaker and Octopus, but deployments were still relatively slow and risky.

Containers and orchestrators​

Around this time, Docker was making waves, and we started experimenting with it and early versions of (pre-EKS) Kubernetes and ECS. The speed and ease of deployments, relative to what we had been doing with hand-rolled automation of EC2 and autoscale groups was game changing. Suddenly, treating infrastructure as immutable felt natural, and deployments were cheap and fast.

Continuous deployment​

The teams that had adopted containers, kubernetes, and ECS quickly discovered the power of continuous deployment. While it was technically possible previously, deployments were slow enough that teams were still batching up changes and doing big-ish releases (maybe a couple times a week). Now, the opportunity presented itself to deploy any given feature the second it was ready.

Shift to automated testing and observability vs manual QA​

As the braver teams started to actually practice continuous deployment, they found that there was an increase in the number of bugs that would remain undiscovered, sometimes for days. In retrospect, our culture had been too reliant on having a QA team, who was organized around doing manual regression tests on big batch releases. Teams began to re-discover the need for some essential ingredients of continuous deployment:

• Robust observability and alerting
• Running automated regression testing in CI

Platform Engineering​

At this point there was a huge, and increasing gap in maturity between teams who had invested significantly in their operational capabilities, and teams that hadn't. It was also clear that to get to even a baseline level of continuous deployment required months of investment from every team... and I don't think we'd even come to grips with the reality of maintenance on all that stuff.

It became obvious we needed to find a way to share the capabilities between teams, so we started experimenting with ways to reclaim some of the abilities we used to have with our monolith- but in a way that worked in a world of autonomous teams and distributed systems.

We quickly realized that some things were a no-brainer to centralize: running Kubernetes clusters, CI/CD, and observability infrastructure, in particular. We also started playing with integrating other opinions and best practices into tooling, and trying to find the balance between operational uniformity and developer freedom. At some point in the past few years, we started calling this "Platform Engineering".

Flavoring is key​

Looking back, it's actually hard for me to imagine, in any practical way, how any of these practices could exist on their own. I mean, you can gnaw on a potato, but it's hard to call that a meal without the rest of the ingredients.

Historically speaking, there's a few more tasty bits sprinkled into this stew:

• Servant leadership and Intent-based leadership
• The Lean Startup varietal of Agile

These are really key to building a modern engineering culture where developers can flourish; you can't have a high-performing team without enlightened leadership.

I should note that I've tasted a version of this stew, but with these flavorings replaced with "command and control" and "waterfall". That stew tasted like garbage.