Challenging the status quo
Why it's essential for innovation
Introduction
In the early stages of our careers, we often lack the experience and knowledge to question the practices and processes within an organization. Our thoughts and opinions are not yet well-formulated, and we are still learning the ropes. We do not know what is good or bad, nor do we understand whether an organization would function better with a different structure or set of best practices. This is expected, as we are still in the learning phase.
However, as we gain experience and spend more time in the industry, we start to see certain patterns. Little by little, we form our own opinions about what works and what doesn't. The danger lies in becoming too dogmatic in our beliefs. Even worse, we might learn to live with the status quo and not question it.
In this article, we will discuss the importance of challenging the usual approach and how it can help us in our careers and benefit our organizations.
The trap
Many companies, especially those that have been successful for a long time, fall into the trap of not questioning their processes. Over time, they build large systems and become industry leaders by leveraging their initial success. Initially, they might write small scripts to handle specific tasks and offer their users valuable services. Being early movers in their industry, they quickly establish themselves as leaders in their field.
Like many companies, they never stop to question their processes. They are content to ride the wave of success, focused on growing their business as quickly as possible. Fast forward a few years, their systems become more and more complex. They add more scripts to handle more tasks, build interfaces to help users interact with their services, and even start to offer subscription services. Little do they know that as they grow their user base, the more they have to satisfy their users' needs.
Their monolith grows so large that it becomes impossible to even run it on a single developer's machine. They form a dedicated team to manage virtual machines where developers log on to run the application. Not only does this mean that their time-to-market drops drastically, but it also greatly reduces the developer experience.
They are so focused on growing their business that they never stop to think about whether they are doing things the right way. To be honest, back then there were no industry-standard best practices to follow either, so they don't know any better.
Today, we live in a world of various technologies such as microservices, serverless, and containers. They go on hiring the best engineers to help them build better systems to solve the problems they are facing. However, they are still stuck in their old ways of thinking.
Why we don't stop to question
Every new engineer they hire is shown how these virtual machines work and how they can run the application on those machines. Once they are certain that the new hires can become productive, they ask them how they would improve the system. All these bright minds come up with interesting ideas:
Building a system to easily provision these virtual machines
Building a system to easily sync the code they are working on locally with the virtual machine
Building a logging system to help debug issues replicated on these virtual machines
Building a system to monitor the health of these systems
The list goes on and on. But do you see the pattern? All these engineers are thinking inside the box. They have been trained so well in how the systems work that they are not able to look beyond the box in which they have been placed.
Challenging the usual
Dr. Russell Ackoff1 gave an excellent talk about his time at Bell Labs in the seventies, where he shared a story of how he unexpectedly found himself in a room with some of the company's engineers. These engineers were told that the company was facing a huge problem. They were told that, overnight, all the landlines of the country went down and that there was no way of fixing it. Since Bell Labs, back in the day, was in the telecom business, this was a huge problem.
The engineers were given the task of coming up with a solution to this. They were all confused and didn't know where to start. This was the first time ever in the many years they had spent at the company, iteratively improving what they already had. Never had it occurred to them that there was a whole world of possibilities out there, which they had never bothered to wonder.
The problem here does not lie in the intelligence of the engineers—they were some of the brightest minds in the whole world. Nor was it a problem induced by the company. The problem is basically human nature.
Human nature
Humans are creatures of habit. Imagine being put into a prison. The first couple of years, you definitely will not feel comfortable with your freedom being taken away from you. But after a while, you will get used to it. You have to get used to it since there is nothing in your power to change it.
Similarly, the famous Stockholm Syndrome2, where during a bank robbery in Stockholm, people were held hostage and ended up having positive feelings towards the robbers. They went as far as defending their motives. These are examples of how we, as humans, adapt to our circumstances. After all, we're all just trying to survive.
The vicious cycle
The cycle of thinking inside the box is a vicious one. We need to spend some effort to break out of it. That effort might come externally or internally, but independent of where the push comes from, we need to make sure to seek that activation energy.
Not always will there be a way to break out of the box. There will always be certain constraints which we need to adhere to. However, making a conscious effort to try to break the cycle will potentially help us discover better ways of doing the things we do.
In the case of the engineers at Bell Labs, the situation they were put into helped them invent the push-button telephone instead of the traditional dial phones (which caused us to hate our friends with nines and zeros in their phone numbers).
In our case, we realized that by leveraging containerization, we were able to set up a working development environment for our engineers to help us overcome the issues we were facing with having to provision virtual machines in which we develop and debug remotely.
It is apparent that we fear taking risks. We fear that by trying to improve the situation, we might actually make it worse. Something we've previously experienced feels to be a safer place than the unknown. But when we do that, we're actually comparing experiences to perceived facts. Therefore, the importance of applying the scientific method to our daily lives.
None of it would have been possible if we had not stopped to think about it.
What can you do?
As with many aspects of life, this is challenging to achieve. As previously mentioned, adaptation is so deeply ingrained in our nature that it's extremely difficult, if not impossible to resist.
However, there are certain methods which you can bake into the company to help you break out of the cycle.
Research
You might remember from your university years how, when you are reading a scientific paper, the first thing you do is read the abstract where they talk about the previous research which has been done in the field.
This is also important to remember in our fields. Many other companies have faced the same problems which we are facing. They have come up with different solutions to the problems. Taking some time to do some research will help us discover what our options are.
Second opinion
One of the most effective ways to break out of the cycle is to bring in an outsider who has not yet formed any opinions on the status quo.
New Recruits
When onboarding new engineers, instead of telling them exactly how the organization works, try giving them a month or two to discover things on their own. This will help them struggle with some of the problems which the organization might have become numb to. Also, ensure to have a check-up with them at the end of this period, asking about their opinion on how things are done. You would be surprised with the amount of insights they will bring to the table. It might even make you feel stupid about some of the things you have been doing for all those years without ever noticing that it could have been done differently.
In some cases, those concerns might not be valid, or there might be a good reason why it was set up like this in the first place. But that's also fine since it will help you explain the reasons behind some of those decisions to the new recruit and therefore help them with their onboarding.
Even if you manage to get one or two good ideas out of it, you can have a big impact on the whole organization.
Consultants
One (mostly dreaded) way to get an outsider's perspective is to hire a consultant. An expert in a specific field who can help you assess the way you have been doing things. Of course, good consultants exist, those who first understand the problem and help you develop solutions, and bad consultants, who spend their days watching YouTube videos and then offer solutions they've merely seen elsewhere. It's crucial to find someone with the right expertise.
Hackathons
Another great way to help your engineers push the boundaries of your methods is to have hackathons. You could even invite external engineers (which also would be a great tool for recruiting new talent). Since during hackathons, people tend not to feel the pressure of the day, it will allow them to look beyond what they have been doing and try to come up with new ways of doing things.
We've seen, again and again, great ideas being bred during hackathons which eventually become the tool which the company uses going forward (and even in some cases, such tools have become the company's main product).
Artificial barriers
You can also artificially induce some more barriers. Since humans are like water, where they'll come up with finding the path with the least friction, such barriers will force them to think differently.
Imagine a company having a CI pipeline where they run E2E tests on every commit to the master branch. As the company grows, so will the codebase (both in production code as well as more tests). This will mean that the CI pipeline will start getting slower and slower. The way engineers might go around the problem might be that they decide to write fewer E2E tests, or that they try to optimize the code to help make the tests run faster. If you would go in and add an artificial delay to the pipeline, rendering the whole pipeline useless (which will cause the pain to be felt today instead of months from now), then engineers might come up with the idea of extracting the pipeline into smaller pipelines where the unit tests would still run on every commit but the E2E tests might be running on a certain cadence (hourly, daily, etc.).
Only when the pain becomes unbearable, then we feel the urge to do something about it. Otherwise, we just learn to live with it.
Slack time
One great idea which we've seen being used in several companies is the concept of Slack Time3. Where engineers are given a certain percentage of their time to work on whatever they want. Similar to hackathons, this allows them to work on things which they might not have had the time to work on during their normal work hours. This has been the birthplace of many great ideas which have been implemented in many companies.
Conclusion
It's important to have systems in place to help your organization explore unconventional solutions on a continuous basis. While it may seem difficult, establishing methods like onboarding new recruits, hiring consultants, organizing hackathons, introducing artificial barriers, and allocating slack time can go a long way in helping your engineers and teams overcome the status quo and come up with innovative solutions to the problems they're facing.
As all these different methods to what you have been doing in the past, you can rely on the commitment device to help create these new habits.
By implementing these strategies, companies can create an environment where challenging the usual approach becomes a habit, leading to continuous improvement and innovation. This not only benefits the organization but also helps individuals grow in their careers by fostering a culture of curiosity and continuous learning.