Using AI to promote divergent thinking

TL;DR

A possible explanation for why we observe such a discrepancy between the productivity gains from AI tools of seasoned senior software engineers versus their less-experienced counterparts:

• Inexperienced problem solvers have a tendency to get stuck when using AI tools to solve complex problems because they bias towards convergent thinking: narrowing possibilities and looking for a specific solution which fits within their preconceptions.
• Experienced problem solvers who have adopted AI tools tend to use it to augment a scientific process, which includes building mental models, formulating hypotheses, and performing experiments. In this process, they use AI to accelerate divergent thinking: exploration and discovery of possibilities.

I've been thinking a lot lately about the significant discrepancy I've observed in how effective different people are at using AI for problem solving. I'm certainly not the only one; there seems to be a strong consensus building in my field (software engineering) that AI tools disproportionally augment the effectiveness of experienced people (e.g. those who developed their craft before AI tools became omnipresent), and that providing AI to people early in their career can actually slow them down- and impede their growth.

This certainly fits the observations I've made personally, but raises a lot of questions about what experienced people are doing so differently with AI. I've spent some time intermittently reflecting on my own use of AI, and also paying attention to how some of my younger colleagues are using it, and I think I have a working theory.

Behold: Science!

Like many people who spend their careers solving technical problems, I employ a version of the scientific method:

1. I start with my existing mental model of how a system works (based on previous observations). Like any scientific model, this is always varying degrees of incomplete/incorrect.
2. I use my model to formulate hypotheses about what changes to the system will result in a solution to the problem I'm working on.
3. I decide which of the hypotheses I should test first. This incorporates several factors:
   • the hypothesis's likelihood of being correct
   • how difficult and time consuming it will be to validate
   • what side-effects or tradeoffs come along for the ride
4. I test the best hypothesis and use the findings to improve my mental model.
5. Problem solved... or... rinse and repeat with my refined model.

This isn't just for problems of a certain granularity; it's fractal, and works at any level. I approach both multi-month and 5 minute problems this way (with varying degrees of rigor, depending on how well caffeinated I am). As I break down big problems into smaller ones, recursively, these tools (mental model, hypotheses, running tests) maintain their value.

The forking road

I think of problem solving like traveling a road that branches into infinite forks. Each fork has different probabilities of success, different risks, and different tradeoffs. Some have hidden dangers, or they might be really exceptionally long and difficult to travel. On top of that, any of them could be dead ends.

Following any fork- even the dead ends- could reveal valuable information, which you can use to refine your mental model (which in this metaphor would be something like an aerial map). But it costs time and energy to take any one, so you need a methodology to make each decision as efficiently as possible.

If there's a better way than the scientific method, I'm not aware of it.

What experienced problem solvers do differently

Putting aside AI completely, the biggest advantages experienced problem solvers have in any given domain are:

1. They have a more accurate, well-developed mental model of the problem space, and they invest time in continuously refining that model.
2. They use this model to formulate better hypotheses, prioritize based on chances of success, and estimate the effort required to test them.
3. They design and execute these tests efficiently- they focus on building understanding and reducing uncertainty with the minimum necessary time/effort.
4. They have a general awareness of when it would be more productive to double back instead of plowing forward. After invalidating a hypothesis, they're willing to retrace their steps past multiple previous forks in the road and use their newly refined model to reevaluate a previous decision.

Novice engineers are pretty good at noticing how AI impacts #3 (executing tests on a specific hypothesis). In fact, I don't think they're usually thinking of what they're doing in terms of a scientific method with discrete parts; they're just instinctively taking guesses and charging ahead. The test result is often binary: "problem solved" or "problem not solved".

AI is so freaking powerful that this approach can be enough to solve a lot of kinds of problems. This is especially true if you've got an adequate mental model out of the gate, and you can craft a prompt with sufficient context. If you're dealing with a problem in a well documented space, there's a good chance that an AI can land you a hole-in-one on your first try.

But for an inexperienced problem solver faced with a problem that falls slightly out of this ideal zone, they can easily become lost in our branching road of possibilities. Or, they get stuck on a particular branch and keep trying different variations of a flawed approach.

Avoiding forks entirely

Seasoned engineers often get a gut feel that they can't be the first person facing a particular problem they've encountered. And it's often true- depending on the question you're asking, the answer (or relevant data that can help you answer it) might be readily available on the internet.

As it turns out, AIs are exceptionally good at rapidly scouring the internet, ingesting huge swaths of text, analyzing and summarizing it. These types of questions can require orders of magnitude less time and effort to answer with AI than by a human doing the research.

Experienced engineers get so much more out of AI in this realm because they have the instinct to ask the question before starting any work- and AI reduces the cost of answering many of these questions dramatically.

Scouting the fork ahead more quickly

When initial research leaves you with open questions, you're left with investment decisions: which experiments do you want to spend time on? For a software engineer, this often means writing enough working code to try out an idea- which can require hours/days of your scarce and valuable time. This can be a risky bet on a hypothesis- one that may bear no fruit.

With AI, you gain the remarkable ability to execute tests like this with superhuman speed. If you can articulate your experiment in a prompt (and usually guide it through some iterations), a decent AI agent can take care of the mechanics and gruntwork to generate the minimum viable code to answer a whole lot of questions- lickety-split. The fact that AIs tend to write code that's less than production quality is of little consequence when your goal isn't writing production code- it's answering questions.

The AI speed difference in this phase of the process isn't just incremental, it's transformative. An AI-assisted human can evaluate exponentially more hypotheses than a human alone. Experienced engineers are much more likely to utilize AIs in this way, and better equipped to interpret the results of each experiment.

Discovering which forks exist

The two most common failure modes I observe when inexperienced software engineers get their hands on AI tools:

• they very rapidly execute fully formed solutions that carry very significant (and often unacceptable) tradeoffs they didn't recognize or understand.
• they fail to solve the problem, and get stuck trying to find ways to solve it within the constraints they assume apply, but don't fully understand.

Using our forking road analogy, they had gotten stuck after choosing a path without finding out what other paths existed.

In both cases, I would generally guide my younger colleagues through some divergent thinking. Experienced problem solvers deliberately invest time in idea generation, exploration, research, and brainstorming- to expand the realm of possible solutions they can consider- before they begin converging on a specific solution.

Divergence is a core element of human creativity- and applies especially to difficult and novel problems which often require diversity of perspective and fresh approaches to solve.

Using AI to promote divergence

The first step in divergent thinking usually involves expanding the scope of your existing mental model by surveying preexisting knowledge. For software engineers, it might mean reading up on some relevant computer science concepts, product documentation, or open-source code- and letting this open up possibilities beyond the constraints you previously perceived.

This is where I think AI is drastically underutilized by inexperienced users- for exploration, learning, and idea generation in a broad problem space. AI is incredibly useful when used in this way. In our forking road analogy, it's like having the ability to fly a drone to scout ahead so you can see all the forks available within a few hours' walk, eliminate the ones that obviously result in dead ends, and discover new ones that look very promising.

AI's current flaws, like hallucination and sycophancy, are a lot less problematic when you're using it to augment divergent thinking- since any idea generated in this phase of problem solving (including by humans) can be flawed. It's part of the game- you take early ideas with a grain of salt.

We all get stuck in our own ideas sometimes

Persistence, I'm told, is one of my more admirable qualities... meaning I'm stubborn AF. I occasionally get stuck trying to make a particular idea work, only to fail repeatedly. Even with AI assistance, I'll sometimes find myself trying variations on the same approach until I get maximally frustrated.

When I'm my more enlightened self, I remember this is a great time to step back, open a new prompt with an empty context window, and treat the AI like a colleague who can give me a fresh perspective. I start asking questions that broaden the range of possible solutions.

I focus on prompts that specifically elicit divergence:

• I ask for multiple options and/or crazy ideas
• I ask it to challenge my assumptions
• I ask it questions that help build my general knowledge of the system or problem space
• I ask it to look for examples of similar problems

Even untenable solutions, or ones which may not apply in my current case, might spark an idea that leads to something promising.

Build your mental model, then go nuts

Lately I've been putting deliberate effort into using AI in this way: to build my own understanding of problem spaces. I've been using AI agents as research assistants that I send off on missions to educate me on some topic, at some level of abstraction that I'm interested in. This is usually fairly broad and shallow to start, and from there I can explore the space interactively by peppering them with follow-up questions.

This is a process I used to do with Google searches, O'Reilly books, and a whole lot of reading in bed. It's a process of synthesis- sifting through large amounts of information to build a mental model, so there's a large amount of waste involved, by definition.

If it had been an option, I would have much rather spent an hour talking with an expert on a topic and grilling them with questions. Having to grind through hundreds of pages of source materials on my own, when 95% of the content wasn't relevant, or wasn't at the level of abstraction I was aiming for, wasn't an efficient way to learn.

Current LLMs are incredibly good at this mode of exploration, especially if you keep your skeptic hat on all the while. Applying a little Socratic method to any aspect of their findings that triggers my spidey-sense often helps them self-correct. It helps that I have years of experience engaging in this same process with humans- who, I have noted, are also sometimes wrong.

Having an "expert" give me a broad overview of a problem space also makes me a whole lot more efficient at doing my own manual research. Armed with concepts, vocabulary, and starter sources, I can get right in the weeds and begin reading primary sources on a topic I'm a complete beginner in.

Once my mental model is sufficiently filled out, I find I'm way more effective at the subsequent phases of problem solving. I've developed some vocabulary around key concepts and their relationships, and my intuitions allow me to formulate much better prompts to guide AIs through code generation. I'm also much better at recognizing subsequent hallucinations or other bad ideas.

AIs don't grant problem solving skills

Many non-engineers hold the misconception that software engineers spend most of their time and effort writing code: the kind of code that LLMs are getting increasingly good at writing every day. It is true that we spend a good amount of time staring at code on a screen, but obscures the truth that our time is absolutely dominated by more general problem solving: designing, debugging, troubleshooting, and analyzing systems.

Engineers who understand how to problem solve using the scientific method, who value mental models, and who practice deliberate divergent thinking, have always outperformed those who don't. Augmenting them with powerful AI tools is going to drastically accentuate this discrepancy.