TWH#50: Simple Explanations For Complex Phenomena
Our brains are playing tricks on us and we're none the wiser.
“It is not the strongest species that survive, nor the most intelligent, but the ones most responsive to change.”
—Charles Darwin, “On the Origin of Species”
Isn’t it interesting that for all the talk in startups about “innovation”, “figuring things out” or “change is the only constant”, we end up invariably falling in love with neat things like this:
And this:
And this:
Interesting, but not surprising. Our brains evolved to crave certainty, order, and structure because these were the qualities that kept our ancestors alive. They relied upon them to identify and avoid threats, and to find sources of food and shelter.
Additionally, our brains are this paradox of being capable of incredible feats of imagination and creativity while, at the same time, having a very limited capacity to process information. Order and structure help us simplify complex information and reduce cognitive load. It helps us conserve mental energy that we can allocate towards more important things like, say, survival and reproduction.
It was a huge collective of these brains that, over millennia, got us here. Step by step we created a ton of technology that accelerated everything, created abudance where there was scarcity, solved many problems, and created a lot of new ones. Things became, undoubtedly, a lot more complex.
The issue is that our brains evolved over millions of years, while this complexity explosion happened, for all intents and purposes, yesterday. So we are wired to seek out simple explanations for complex phenomena, relying on stereotypes and 150+ cognitive biases as heuristics to navigate this brave (really) new world.
That means we are, inevitably, wrong a lot while at the same time resisting a lot the idea of being wrong. Why?
Because the idea of being wrong is encoded in our ancient brains as failure, which leads to the fear of being rejected, which leads to the idea of being alone, which back in the day literally meant death. Not great. So our survival instincts take over and override everything.
And yet… at the same time we strive for accomplishment. We want to be part of something bigger than ourselves. We want to self-actualize. We want to go where no one has gone before. We want to create businesses, and build products, and solve customer problems, and change the world!
All of it riddled with uncertainty. Oops.
Simple Explanations
Let’s zoom into the reality of product & engineering teams as a means to explore how this plays out practice.
In my experience, one of the top wishes of software engineers is “better requirements.” They want to do a good job, and their biggest pain point is that “the requirements keep changing.” In other words, if the target keeps moving you beg it to stand still. The reality is that most of them have never worked in a true agile environment. They do standups and retrospectives and all that, but they still want better requirements—a hallmark of the “waterfall” way of working.
On the other side, product managers and other stakeholders demand more accuracy in estimations. How long will it take? Can we really trust it this time? Are you sure?
Both stances, while well-intentioned, are ultimately rooted in the idea that the work is known and predictable. But that’s not true, a fact that is either tacitly not accepted or simply not understood. Our need for predictability and control is so strong that we avoid facing the music. And our busyness is such that we have no time to bridge the knowledge gap by educating ourselves.
The result is a vicious cycle of eroded trust. Developers feel like they don’t have control over the requirements so they eventually shy away from being on the hook for their own estimates. And stakeholders put even more pressure on estimates, further exacerbating the problem.
Ultimately developers make up their minds that stakeholders don’t know what they want or what they’re doing, and certainly that they don’t “get” engineering. Stakeholders make up their mind that developers are not good enough, don’t care enough, or don’t understand business. This resentment often lurks and festers under the surface, leading to slow and silent attrition, etc.
Above the surface, though, the answer is typically the same: do more and better planning. And then everyone is frustrated that planning takes forever, that there’s no time to focus on coding, and delivery gets delayed.
All because of simple explanations for complex phenomena.
Embracing Reality
The reality is that the work is far from predictable, and we ignore this fact at our own peril.
Most companies barely know their customers, let alone deeply understand what their pain points are. And the ones that have a decent grasp of that still need to figure out what best solves for those pain points. Yet, we still manage to outcomes hoping that it’ll all work out in the end—if only we build this and this and that.
Even when it comes to coding itself, while some stuff is indeed predictable and gets abstracted away in frameworks and the like, custom solutions can still be implemented in multiple different ways, all with their own trade-offs and quality outcomes. And, inevitably, we learn new things and that code isn’t so good anymore.
To make it worse, the larger the organization, the more this complexity explodes. No wonder that there’s so much dissatisfaction at work in the tech industry.
Can we do better?
Yes, we can. But it requires embracing the reality that most knowledge work is inherently uncertain and complex, and therefore cannot be predicted and controlled. Lean and Agile consultant Liz Keogh puts it this way:
If you’ve done it before, requirements are known.
If someone else has done it before, requirements are knowable.
If it’s never been done before by anyone, requirements will change.
Product work most often falls into that last category, particularly when you’re trying to create a new product, enter a new market, ultimately attempting to do something that hasn’t really been done before. If you can’t confidently say your company hasn’t yet found product-market fit, then you’re definitely in that space most of the time.
That’s why requirements can’t be perfect—they actually need to be discovered as you go. And that’s why being asked for estimates for something you’ve never done before is terribly frustrating—and a fool’s errand designed to create an illusion of control and to break hearts.
Keogh again:
The code written in both BDD and TDD relies on a clear understanding of the outcome and how to reach it – when requirements are known, or knowable (you might have to look for a better expert).
Seeking clear outcomes works well in a complicated or obvious space – one in which cause and effect are strongly correlated and predictable – but we know that if it was possible to analyse everything up-front, Waterfall would work.
When you have conversations around the scenarios, sometimes the business won’t be certain about the outcomes they want. They might know about a higher-level outcome – gaining more customers, or more advertising revenue, for instance – but they might not know whether the feature they’re suggesting will actually work.
If you’re a developer, a product manager, a designer, an engineering manager, or a stakeholder—what would you need to shift in order to embrace this reality instead of being at odds with it?
How I Learned to Stop Worrying and Love the Complexity
I might be painting too dark a picture of the tech industry. I may across as if I believe that no one knows what they’re doing. At some deeper, more philosophical sense that’s true, but I assure you that denigrating everyone’s best effort is not my intention. In fact, the question that keeps me up at night is: how come so many smart, talented, well-intentioned people end up within a system where the outcome is burnout, disengagement, and general unhappiness?
My hope with this week’s piece is to establish, for those who aren’t yet aware of these themes, that part of the answer to that question lies in how our brains work. We are programmed to default to less than helpful ways of working. And, unfortunately, although we are for sure feeling the pain, we struggle to find the solutions because they can be quite counter-intuitive.
There is, however, a growing body of knowledge, practictioners, and communities around this. There is increasingly more awareness and proficiency in new ways of working that are better suited to dealing with complexity. And, importantly, these practices bring humanity into the work.
This is all nice and good but you might be wondering (and rightfully so): “Alright, Paulo, but if not more planning, better requirements, and less changes in priorities, what then?”
The short answer is: less predicting & planning, more probing, sensing & responding.
When we know we want more customers and revenue but are not sure how to get there, analysing the situation to death won’t make much of a difference. We need to accept our ignorance, and experiment to learn. And get really good at it.
It requires a momentous shift in mindset, and then small practical adjustments here and there that add up and compound over time. The good news is that there’s an abundance of intelligence, creativity and curiosity in all human beings. And, luckily, those are exactly the base ingredients required for complex work to… work.
Enough with the philosophy, already. Starting next week, we’ll get practical and explore different ways of working you can apply today, no matter your starting point or your authority. Amidst uncertainty and in the absence of predictability, we can safely experiment our way to better outcomes, both business and organizational, tapping into the full potential people have to offer.
Isn’t that a world worth striving for?
Thanks for reading. If you enjoyed this post, please consider hitting the ❤️ button, and sharing it using the button below.
Also — I have a couple of openings for executive and leadership coaching. If you’d like to explore working with me, drop me a message on LinkedIn or Twitter.
Until next week, have a good one! 🙏
The problem statement and the way it was described resonated very much with me. Can't want to see the next few week's essays.