Let a thousand flowers bloom

I've been reflecting recently on a really formative period in my career, when I had a chance to be part of a massive experiment in progressive engineering management.

About 3 and a half years before I left Vistaprint, I was asked to join the Engineering leadership team by our (relatively) new VP of Engineering, Erin DeCesare (who is now the CTO of EZCater). She was a particularly bold leader in terms of her progressive management ideas, and was rapidly reshaping the organization with a strong set of values around empowerment and servant leadership.

We were responsible for about 200 developers, who were organized into squads (a.k.a. two-pizza teams), who were then loosely grouped into "tribes" (an idea borrowed from Spotify). The real difference from the previous regime was a pretty extreme amount of autonomy given to teams; they could choose their own technologies, work processes, architectures.

On top of this, Erin was pushing farther into some even more progressive empowerment concepts. For example:

• Managers were given a lot of coaching on servant leadership, and the ones who weren't able to evolve were managed out
• Teams would be supported by embedded agile coaches, who helped them optimize team health properties, such as psychological safety and culture of feedback
• Teams were given the freedom to decide what work to pull from our enterprise backlog
• Teams would decide how to distribute bonuses between them (this one went a bit sideways)

All this stuff was a bit mindblowing to me, but I was doing my best to commit to making it all work. It was an extraordinary amount of cat-herding, managing through influence, and general chaos, but it felt crazy enough that it just might work.

We set up some initial organizational mechanisms to attempt to manage all this chaos, since the whole point was to build a better team that could deliver features and products that made our customers happy. We set up feedback loops, elevated thoughtful and visionary people, and set up structures to help make sane architectural decisions.

Then, things got very interesting.

Then we watched the experiments unfold

We had something like 30 squads, each running themselves in almost any way they saw fit. There was a lot of diversity between teams; folks with different backgrounds, different technology preferences, stronger or weaker opinions, different seniority levels, etc.

What I witnessed was 30 different teams running 30 different experiments into what makes a team successful... or not.

Some examples:

• Some teams stuck conservatively to working in our monolith, and did a release every 3 weeks, others spun up Kubernetes clusters in AWS with KOPS and deployed their apps via helm charts several times a day.
• Some were absolutely religious about automated testing, and obsessive about code coverage, others had a more deliberate risk-management strategy.
• Some teams spent a long time getting observability working, and others relied more on signals from external SRE and QA.
• Some worked via consensus and mostly made decisions together, others had one or two very senior folks which set the direction for the team
• Some teams did all their work in pairs, or via "mobbing", while others only interacted with their teammates when they were stuck.

There was also a lot of variation in the adoption of the agile philosophies we were coaching the teams into adopting.

Things I noticed

After running in this mode for about 9 months, there were a few themes I noticed which have really informed my perspective today:

Engineers will fill all available space with engineering work

One of the "tribes" (4 squads, around 25 people) was given an objective, some constraints, and 6-12 months to deliver a new platform for managing our product catalog. We were replacing our "legacy" product system because it had grown too creaky and complex over the years, and we wanted a bunch of new features, especially for marketers to be able to manage content without needing engineering time.

Yeah, I know, this is a classic case of second-system syndrome, but everything just seemed so obvious to us at the time; we were smart people, and it seemed achievable to us. So we did some design, broke the work up and gave pieces to each of the squads.

As it turns out, our teams invented engineering problems to fill all available space. They created an elaborate network of microservices, built SPAs out of hip new JavaScript frameworks, integrated some truly abominable "enterprise" vendor products, and designed massively complex processes and workflows to solve every use case that we failed to account for in the legacy system.

It started to hit me that we had jumped the shark when I noticed that a microservice one of the teams built could have been implemented as a single text file. Once that clicked, I started realizing, to my horror, that everywhere I looked, the entire system was like this. And then, through the haze of the groupthink, it occurred to me:

We'd have been better off giving this entire objective to a team of 5 people for a month.

A small, constrained team would have had no choice but to build something small and simple that worked. Then they would have had to evolve it incrementally, which as we now all know, is the only way to build a working system.

Agile can be so powerful, or so horrible

Some teams got really religious about their agile methodology (usually Scrum), and used agile "rules" as a weapon against their PO, manager, and occasionally each other. They'd spend a lot more time playing games with ticket management, ceremonies, and storypoints than they did thinking about customers, products, or using their common sense.

One of the best defenses against teams going this direction was having great agile coaches (NOT the high priests from big consultancies who spend their time promoting seminars on LinkedIn). A good agile coach can provide a sort of continous intervention for a team; they hold up a mirror, allow them see their own disfunction, and help re-center them on what matters.

From observing these coaches in action, I formed two separate beliefs:

• I've noticed that great agile coaches tend to also be very product-centered, and are genuinely interested in developing technical and domain knowledge from the team they're working with. They ask a lot of questions, and develop insights that are specific to the practical limitations the team is managing. They don't just waltz in and start unloading a bunch of dogma.

• Its interesting that when you see a healthy, high-performing team, agile methodologies are sort of invisible; they're there, but they sort of melt away into the background. The team is just focusing on the actual work: how to meet a customer's need, what hypotheses they should be testing, or thinking critically about business requirements in order to find an 80/20 solution.

The cult of test

We had a few teams that drank the TDD kool-aid hard. One in particular had agreed as a team to take testing really seriously. They set about implementing extensive test suites, held book groups about BDD, and spent hours trying to compile Gherkin files so their business partners could define test cases (which they never actually did). They believed in the singular truth of the test pyramid, and went big on unit tests, shunning higher-level approaches as lacking in virtue. They set up CI/CD to fail builds when code coverage dipped below 95%.

Things started to go sideways fast. Because their test suites were targeting low-level implementation details, every feature they implemented broke a gazillion tests. They were terrified to refactor anything (including the tests) because it would ensure weeks of work and an avalanche of merge conflicts. They'd get nothing done, sprint after sprint. The result wasn't amazing quality, it was utter paralysis.

Deep Thoughts

There's definitely some lessons to learn here specifically about testing practices, but I think this is really a more general case of a failure to apply continuous improvement. The intervention wasn't to parachute in and tell them how to do testing better, it was to have them pause and reflect, re-focus on what they were trying to accomplish, and give them permission to try something new.

With some perspective and coaching, they started to re-think their testing strategy. They began to see that some risks were more important to mitigate than others, and some testing techniques gave you a lot more bang for your buck. They came to the conclusion that 95% on a code coverage report wasn't a business objective, since we were building a freaking e-commerce site, not pacemakers or cruise missiles. They gradually woke up from their fog, and came up with some much more clever testing strategies, such as testing the high-level interfaces and APIs, which were far more stable, and gave them the freedom to refactor implementation details.

Moderately strong teams outperform superstars surrounded by a meh team

We had a few squads where 4-5 (out of 6) team members were solid "A-" or "B+" players. In contrast, there were other teams held together by one very strong "superstar" lead surrounded by "B"s and "C"s. It was strikingly obvious to everyone that the teams with the more homogenous, moderately strong members significantly outperformed the teams with the superstar.

My take on the underlying cause was that the superstar would get randomized trying to support the rest of the team, and didn't have enough time to do anything innovative. If they ever did manage to spend some time on something interesting, it was generally too advanced for the team to run with, and the momentum fizzled. The rest of the team became helpless and dependent on the superstar to make decisions.

Of course, there was occasionally the more enlightened superstar, who would spend their energy trying to elevate their lower-performing team members. The effectiveness of this was highly dependent on the latent potential of the rest of the team, and usually didn't work without management intervening to shore up the team with additional strong engineers to support the superstar.

In general, the ratio of strong vs weak really can't dip below 2/1 before the team veered into unhealthy territory. The members of strong teams will support and help each other, but there has to be balance between them; if it gets asymmetrical, it starts to drain everyone.

Blinding flash of the obvious

Who knew, you need good people to have a good team.

Strong POs are critical

We had Product Owners deployed to each squad, usually a PO would cover 2-3 squads. The difference between a great PO and a bad one was very stark. POs ultimately decide what the teams work on, so it seems fairly obvious that they'd have an outsized influence on the team.

At the core of the role is having great instincts about customers, the product, and the problem space. But there are other key factors which are underappreciated in POs:

• Empathy and rapport with developers
• A willingness and interest in understanding technical limitations and tradeoffs
• An understanding that they're not only responsible for the customer experience, but the long-term health of the technology it's built on

Looking back

I imagine for most people working there at this time, being the guinea pigs in a giant experiment might not have been the ideal work experience. Actually a lot of folks thrived, and did some amazing work. But some reeled from the unrelenting waves of change, and others understandably just threw in the towel.

For me, though, it was very different. I got to see all this from a birds-eye view, but also on the ground, since I spent time with nearly every team. It was a bit like watching a whirlwind from inside- it gave me a sense of how seemingly small, well-meaning and thoughtful inputs can have huge, unintended effects that ripple across the whole system.

Like a lot of younger engineers, I used to occasionally express casual and flippant disregard for out-of-touch upper management. OK, admittedly I still might feel this way from time to time, but this experience left me with a very different understanding of what it takes to manage a large engineering organization, gave me a dose of humility and appreciation for how challenging it is.

I'm incredibly thankful to Erin for taking me with her on this journey- I really couldn't have had more learning jam-packed into a few short years of my life.