The Science of Chaos to Clarity

What do you do when your life is in Chaos? Perhaps you call a close friend, a family member, a mentor or an advisor and seek help. When things get to a state where the word chaos can be used, stopping to reflect and reach out may not be a luxury that that person or company has, but it is the best chance to survive.

As a trusted advisor to hundreds of companies, we have run into this so often that we created a methodology called "Chaos to Clarity". Some businesses may not think that they are in chaos, but they know that things are not right, and they just want it to get better. Chaos is very disruptive to companies, and some of the more sophisticated businesses out there actually have a process to inject chaos into the business on a regular basis to make sure that they are resilient and can handle disruption. Netflix has a process called Chaos Monkey that tries to cause havoc in their infrastructure to make sure they are ready for anything.

At iuvo Technologies, because we help so many businesses navigate away from chaos, we even made T-shirts for our employees showing our company protecting a data center from Godzilla, the ultimate generator of Chaos. Let us dig into this a bit more to help you understand how to avoid chaos.

What Are The Most Common Causes Of Chaos?

Chaos comes in many forms but for the sake of this blog article, let’s rule out Godzilla, asteroids and most natural disasters and focus on the type of chaos caused by humans. We will also focus on what happens inside businesses, although there can certainly be parallels to individual situations. Chaos is what happens when complexity goes awry. It is almost always something that is built over time, often via a series of short cuts or work arounds. Those work arounds could happen because of lack of knowledge, experience, resources (human or financial), or in some cases gross negligence. It often starts with a complex workaround because a simple solution is not apparent, then other complex solutions are built on top of it until there are layers of complexity.   You might be asking yourself, why would someone not take the time to construct simple and elegant solutions? The answer is that it is difficult to do. Simple and elegant solutions to business or technical needs require experts who know how to do it quickly and correctly the first time. Those experts are often not available or perceived to be too expensive and thus the work around needs to be "good enough." More sophisticated businesses will track metrics like the Cost of quality (good and bad), yet it typically takes a serious business impacting event for leadership to take action.

Today's world is run by computers and networks, so when you read about a major service outage, a security breach or a company that went out of business because of technology, there was likely some element of chaos at play.

Clarity to Chaos - How Does It Work?

Going from Chaos to Clarity takes true experts who are also great generalists with the ability see solutions by using different perspectives that were learned over time.   To help you understand how to avoid Chaos in the first place, let's talk about the framework used for critical thinking and how a company would get from simplicity to chaos.

A useful framework to use is the Cynefin Framework developed by Dave Snowden when he worked for IBM Global Services. It draws on research into  systems theory, complexity theory, network theory and learning theories. To help you understand how it works, we will use an IT related business example.

Phase 1 - Simple

Installing a new computer for a small company of 3 people. You may buy a PC laptop from your favorite computer vendor and turn it on. Microsoft guides you through the steps to set it up, connect to wireless and to the Internet. There are a few steps to get you essentially going that will loosely be the same for every new computer as long as you remember to record your steps and create processes.   You know what to expect and the results should be consistent each time.

Phase 2 - Complicated

When a business or IT system becomes complicated, there are elements that require expertise because there are unknowns that are introduced. The expert in this phase will use the known facts and apply best practices. For the above business, they have now grown to twenty people, they have twenty uniquely configured machines and now need centralized authentication, email, collaboration, and data secured according to business requirements. This is now complicated, but to an expert it is standard practice to create a simple and elegant solution even under complicated situations. It is important that the business leader seeks out the expertise because likely they are spending too much time managing a sub-optimal configuration instead of focusing on growing and scaling their business. Outside of the pure technical realm, a business becomes complicated when they require different departments, sites and experience growing pains. Leaders can rely on expertise from trusted business advisors on growing the company while keeping the elements of the corporate culture that maintains quality.

Phase 3 - Complex

Complexity is something that you can only see if you take it apart to see how it works. There can be processes that are complex yet documented and repeatable. We see this often in DevOps, where there are complex layers of web, application and database servers and software to provide the company's service. The online SaaS services that run today’s businesses are complex. They require more experts and a more rigorous process to keep from moving to chaos. One of the biggest challenges with companies in this phase can still be that the larger the service or product becomes; the more difficult to control and keep expertise in place. Technical work arounds are still very possible. If an employee introduced unnecessary complexity into the environment and then leaves, it may seem more intuitive to build upon an unknown step vs. unraveling it and keeping it elegant.   This is commonly referred to as technical debt. As consultants, when we are introduced to a company to help them correct a complexity problem, there are often so many steps and interdependencies that it can be difficult, though not impossible to identify and correct issues.  This is where an expert generalist become even more valuable because complexity requires someone who can look at something from different lenses and experiences. This is not only true of technology, but complex things like corporate culture.  

Phase 4 - Chaos

Chaos is the phase you seek to avoid. When chaos happens, it requires quick and decisive leadership to "stop the bleeding", look for patterns, and communicate actionable steps to keep people moving towards a solution. Ultimately their goal is to move it from Chaos to Complex. Chaos can be an event like an outage where the complexity had grown to the point that it is not clear what is causing the issue. This is the area where the cost of poor quality will have the greatest adverse effect on a business. Chaos, however, does not require an outage or traumatic event to be chaotic. We have worked on a chaotic situation that occurred at a software company that had a complex IT architecture where employee turnover resulted in problematic software that was not well understood. The trigger event was when the IT Director unexpectedly died with all the critical information and passwords in his head. Nothing was documented in a place that was well known and the infrastructure was full of equipment with abundant failure lights. The technical debt and chaos was merely steps away from a business failure. It is possible to move from Chaos to Clarity, but it becomes difficult for the senior leadership to move the business forward while they are concerned with failure.

Chaos to Clarity

In the four phases previously mentioned, where does clarity fall? Simplicity is the state where everything is known and perhaps simple. Clarity does not have to be simplicity, but perhaps somewhere between simplicity and complexity. The key is that clarity is supported by a process that ensures that the business is resilient, the complexity is well-understood and the risks are manageable. Two examples of building Clarity in complex businesses are around deploying Microsoft 365 and DevOps technologies.

Microsoft 365

Experienced IT staff can set up a scalable, repeatable, and secure business platform that allows a company to grow seamlessly from a handful of people to thousands while following a straightforward and repeatable process. Such a platform can provide the ability to grant or remove an employee’s access to multiple applications and arbitrary collections of data with a single step. This kind of management can be effective in complex environments that cover multiple locations.

In a regulated environment, a properly built infrastructure can ensure that all data is encrypted and only accessible by the right people. There are many ways to accomplish this, an excellent example of such a platform and one that we encourage our clients to consider is Microsoft 365. As a business grows, complexity will be introduced as a natural course of growth. Meeting requirements will necessarily result in a business adopting new processes. Having the necessary experts with critical thinking skills is a key differentiator for any changing business. The knowledge to properly set up tools like Microsoft 365 will yield gains that would more than justify the cost of good quality.

DevOps

The majority of the newest applications available for business and personal use are in the cloud, which means they are in a securely controlled data center and available via a web browser or specific application. There is a radical difference between a proof-of-concept application and one that scales out to tens of millions of users like Slack and Amazon Prime, or even billions of users like Facebook and YouTube. The ability to scale seamlessly and economically is largely a function of the DevOps methodology chosen. Imagine thousands of web, application, database, load balancers and firewalls all working in concert to serve the content to users. The ability to scale dynamically, consistently and to provide consistent results requires properly planning for and managing complexity. It is fair to say, that there is also a large dependency on architecture, code, and design that can introduce complexity very quickly. When poorly managed, the complexity can accelerate quickly into chaos. Unravelling chaos can take significant expertise and skill. In the below case, we were asked to take on a large customer whose income is derived directly from the products hosted on their web servers. If their servers were down, they were losing revenue. When we arrived, they were seeing 97.73% availability across all their publicly available services. This translates to eight days and seven hours of downtime per year due to massive complexity and chaos. This type of complexity had to be understood, tackled, and resolved while keeping the core functionality operational. After four years of work, the site was at 99.98% availability which translates to one hour and forty-five minutes of unplanned downtime and eventually down to six minutes per year. This was truly a valuable chaos to clarity story. The below table also shows that chaos can cost real money. Whether the loss was actualized or not, a product that is unavailable or unreliable will certainly struggle to maintain paying customers.

Client DevOPS Sample

$80M Annual Revenue = $152.21/minute

Where Do You Start?

Whether you’re trying to recover from a chaotic situation or avoid getting into one in the first place, the first thing you need to do is make sure you’ve got the best people available to help. Hiring the right people is absolutely critical to building a resilient and robust business. Many of the most successful businesses started by hiring trusted advisors to establish the proper framework, mindset and help the burgeoning entrepreneur avoid common pitfalls to ensure success. Today more than ever, regardless of industry, computing and cloud technology are a critical requirement for any business to compete. To get the right assistance, do your research, formulate critical questions and success criteria, and ensure that you find consultants who have your success as a priority. Keep in mind that all consultants, IT services and managed service providers are not created equal. It’s important to understand that less expensive services often won’t have the long-term experience needed to avoid chaos. The right trusted advisor can find a better solution faster, thereby saving you time, money and agony.

The Role of Critical Thinking

Critical thinking is a skill that is needed more than ever in todays automated service-oriented culture. Just like using GPS causes most people to turn off their directional brains and just follow the GPS, businesses can also be lulled into the marketing of how easy it is to use the Cloud. Both Apple and Microsoft make it easy to set up a new computer and get started quickly and most of their amazing, automated processes are aimed at the consumer rather than the business. When it moves from a single consumer to a business it moves into the Phase 2 complicated realm and it becomes more important than ever to use critical thinking. The challenge is that there are many unknowns and often the unknowns are unknown. The only way to expose the unknowns is to surround yourself with experts in those domains. HR, IT, legal, engineering, compliance or anything that you need to know, but are not an expert in. As a business leader the most important thing to understand is you and your team’s strengths and weaknesses and make sure you shore up the weaknesses while you build that capability. Keep experts on Tap, not on Top and continue to think critically and apply your own wisdom.

Contact us if you are interested in learning more about our chaos to clarity methodology in order to keep your business running.