Mission CR-IT-ical - Turning Chaos to Clarity - Episode 1

Welcome to a new series we call Mission CrITical that is essentially a discussion among our fellow iuvonauts on topics that are really important to the community at large and need more discussion than a regular blog. It is essentially the type of discussion that happens inside of iuvo Technologies all the time. Although this first one is posted in a discussion format, future episodes may be in different formats including video, podcasts and maybe even a song.

This discussion is moderated by Erica Thompson and includes Derek Murphy, Noah Krohn , Stephen Kleine and Jesse McCain and for more information about them check out our team profile pages.

Todays topics is about Named Storms, which are storms that are big enough to have a name and could have serious impact to your business. What do you need to know so your business can be resilient in the face of natural disasters. The ever-present threat of blizzards in the winter, the new threats of tornadoes, and of course storms like Super Storm Sandy are real events that should give us good insight.

Grab a cup of coffee and follow this discussion on what these IT experts think about proactive approaches to potential natural disasters.

Transcript:

Erica Thompson

Good morning all. Let’s commence the discussion.

We’re here with Derek Murphy, Noah Krohn, Jesse McCain and Stephen Kleine to discuss Named Storms and how they affect IT. I think it is not too controversial to say we currently live in a world where storms are likely to be more frequent, stronger and happen in areas where they don’t traditionally.

What are the biggest challenges you see regarding a named storm impacting a client’s area?

Derek Murphy

Keeping the lights on is the biggest - often it's completely outside of your control. For example, even if you have a UPS, a generator etc... something could happen to disrupt that as well. Coastal flooding could completely flood underground generators, or disrupt gas lines. For business continuity in a disaster you need to have failover that is outside of your region – on-prem or cloud.

Noah Krohn

What I see many times is lack of true DR...there are backups and maybe even enough resources to prop up a few systems offsite, but not the ability to say, spin up an entire infrastructure if your primary data center gets flooded.

Derek

+1

Jesse McCain

and when there is a DR solution, there is often not a tested plan to use to implement it.

Derek

In cloud based infrastructure - that's where things like terraform and designing/provisioning resources in an infrastructure as code based fashion is invaluable - you can provision entire infrastructures in different regions if it's all architected to be scalable/fluid (edited)

Jesse

How available is that to the SMB?

Derek

terraform? It’s all open source.

Noah

Absolutely. It's also where container tech starts to really shine too as you can fairly seamlessly uplift your application stack into a cloud provider and for the most part it's cloud-agnostic.

Derek

you just need to learn how to use it

Stephen Kleine

Something else as well; your work site can be up and running, but debris etc., can prevent people from getting there

Jesse

Asking more from a resource perspective…

Derek

Depends on the SMB. A manufacturing/non-tech company - they wouldn't have engineers. A software startup - software engineers could easily take on managing a code based infrastructure deployment if involved in the implementation.

Stephen

As for how available it is to a SMB... it really depends on the business. A shipping company needs their warehouse up and running, but the sales and order-taking sides of the business may be able to be done from anywhere with a cloud-based strategy.

Noah

I think Derek might be hitting on one of the core problems too...that traditional IT sometimes hasn't adopted the practices have started to become ubiquitous in the DevOps space.

Derek

GUI's are sooooo 2000.

Noah

As is doing things by hand

Or I guess more accurately I should say, doing things manually without any automation backing it

Erica

This sounds like fantastically neat technology. But is it really feasible for infrastructure conversion. I.e. could iuvo walk into a client and say, yup, for disaster preparedness we’re going to implement terraform?

Noah

Absolutely, because we can largely leverage code we'd written before.

Derek

No. Terraform on its own won't fix anything. Terraform is a technology for provisioning 1 piece of the whole picture.

Terraform + ansible + vault + a cloud provider +...

Erica

That’s starting to sound expensive.

Derek

All free!

Except the cloud.

Jesse

To buy, not to implement.

(unless we've changed our business model)

Noah

Hrmm...not sure I agree entirely. If you've developed scripts that deploy a fleet of nginx servers...other people utilize nginx. You might just need to tailor what you have a bit to their environment to make it usable, but the initial investment to develop the scripts is already done.

Stephen

"Expensive" is relative, and a lot of folk tend to get sticker shock without looking at the total costs.

Derek

Sure - so in a small 20 person company who’s tech needs are simple - all of that could be over the top. Maybe they have 2 servers and a small datto does everything they need because nothing changes.

I agree with Noah around the establishing a reusable framework. We can say (for companies who’s business would fit that deployment framework) - "here's how we do it - let us work magic" - if they have a reason to do things completely different - it might be more difficult - but not impossible.

Having standard ways of deploying things I think matters a lot with organizations we manage.

And naming things. Reusable, standardization, familiar conventions.

Noah

Definitely, even down to naming conventions. If I'm trying to recover a site, I don't want to sit there for 20 minutes trying to figure out what function lol-myserver_rocks-23 brings to the party.

Erica

On the flip side, what options might we have if the infrastructure is up in the client’s building, but the building is cut off? Perhaps roads are closed, iced or flooded?

Noah

A lot of this really comes back to some basic best practices - self-document as much as possible (e.g., use system names that make sense), make your infrastructure resilient, those types of things.

Derek

I think that's client specific. A manufacturing company - production halts - there's no way you're going to get your million dollar laser cutter elsewhere.

Stephen

If they're hosting all of their computing inside the walls, they're going to need secure remote access. Whether this is possible depends on the data; some clients have specific needs with regard to HIPAA, PCI, and other compliance.

Noah

But hopefully in that situation we can at least say, "Hey, your email is in Office 365 so at least you can email your customers and let them know what's up right now."

Derek

An organization where everything is remote accessible - VPN - ensuring you have enough user licenses to support all employees, and bandwidth (maybe burstable capable) that you can have to support full user load remotely

and as Noah mentioned - commoditized services hopefully in the cloud - email, file storage etc...

Buying into ecosystems has value. Going office/microsoft365 offers a lot of value in having services co-exist peacefully with ease.

Stephen

I was having this discussion on Office 365 with someone just yesterday; their insistence was O365 isn't a cost savings over just purchasing Office 2016 Professional licenses. In my opinion, they're using the wrong value marker because O365 doesn't require you to have a mail server, and has OneDrive for personal storage, SharePoint for shared storage, and other useful Microsoft offerings under one umbrella.

Noah

It's funny because a year ago I was adamantly against using cloud services and could have seen myself making that argument too.

Stephen

The ability to relatively seamlessly move from one product to another has an efficiency value all its own.

Derek

Same for cloud based infrastructure. If you can design your app to work with AWS services - vs rolling your own - it becomes more valuable. You can easily plan for DR within AWS. If an issue happens that takes down both USEast1 and USWest1 - there's a big problem that will likely be an issue regardless of where you are.

Erica

A lot of the discussion keeps coming back to the cloud and hosted solutions. Is the cloud now an inevitable necessity for businesses?

Derek

I would argue yes.

If I were to start my own company - there is a 0% chance I am buying any hardware.

At least not for the first couple years - unless there was a huge defining reason to.

Noah

Unless you have pretty deep pockets to create your own private cloud

Erica

Wow. I think that might be a topic for a future discussion.

Stephen

Cloud offerings have really matured over just the past few years, and I agree with Derek - if I was rolling up a new client there wouldn't be a local install of anything but office networking, if they even needed that.

Noah

I'm still a dinosaur in that regard...some services I will keep out of the cloud if I have the budget to do so.

Stephen

My daughter runs her own small company. Her office has an all-in-one printer and a Comcast line. Everything else is on a Mac and backed up to the cloud. Location, for the most part, doesn't matter.

Derek

I think it does depend on what your business model is. If you are doing huge data processing and need petabytes of data - perhaps cloud is hard at first and you need a ton of local cheap storage. If you have a couple servers/web/app stack with a small DB layer - cloud all the way.

Erica

Bringing the conversation back to storms; when do preparations begin for a weather event. What methods are used to monitor possible disruptions?

Noah

Now.

Right now.

Before there is another super storm.

Stephen

When the announcement is made that the storm's coming, planning time is OVER.

Noah

If you're waiting to think about it when the rain starts coming, it's waaaaayyyy too late.

Derek

Exactly. The time to plan (for anything) is today.

Stephen

The nightmare scenario is always "When the business is wiped out" - everything in the building is gone. Where are the backups? How quickly can you source hardware? How long will data recovery take? How much data can you afford to lose?

Noah

In practice though, it's a difficult sell to get people to stop thinking about their next code release or whatever it is they do and plan for what will happen if their data center ends up a swimming pool.

Derek

Right. Preparation costs money - and everything is working fine now - it'll never happen to me.

Stephen

That's human nature.

Erica

Planning is one thing, but preparation the actually start of implementing the plan, is what I am talking about here. Of the two major hurricanes we had this year, Florence came in hot, then slowed down and finally arrived with devastating effect, about a week after the first news reports. Michael appeared out of nowhere essentially and steamrolled FL. What is the line between starting too early and having a storm miss you, vs not early enough and not getting the plan fully implemented?

Noah

I don't think there *is* a line. If you're running without a DR solution, you're driving without a seatbelt and hoping you don't hit anything.

Derek

I think the implementation of the plan and testing of the plan is outside of any storm timeline. It needs to be done way before.

Storm. UPS catching fire. War time attacks. Etc... anything disruptive.

Stephen

The smaller things matter. Backups need to happen, and need to be off-site as soon as possible after they're taken (if they're not just sent offsite immediately). But, as the maxim goes, your backups are only as good as the their last restore.

Noah

And the thing is, many times you don't need to prop up *everything* for DR, just the essentials. You need your mail server but maybe not the company media server housing half a petabyte of Jean Claude Van Damme singing hits from Broadway.

Stephen

And while the cycle of plan, implement, test, analyze, and plan again has time and money costs, they're likely very minimal compared to having to rebuild on the fly, if a rebuild can happen at all.

One of the lessons from 9/11 is that companies that can't get up to full steam in a short time - if I remember the study correctly it was two business weeks - aren't around in five years.

Erica

So what’s the call to action here? If you each had to sum up an elevator pitch for DR/DP what would you say?

Noah

Start planning now, and test your solution.

Derek

What is the base need for your business to operate?

Identify must need systems and nice to have systems.

Email/code repositories might be critical - but the 10TB of data that you use to run tests against - maybe not.

Noah

Nevermind, I like Derek's better

Derek

and also - that goes into DR strategy.

Stephen

Disaster is unavoidable, but their effects can be mitigated. You don't buy a snow shovel AFTER three feet of snow's fallen from the sky, you buy it before the storm arrives.

Derek

A software engineer might feel that 10TB is business critical - but the CTO might decide - nope - we can spend a week recreating it.

I would argue Stephen that you buy a snow blower and a shovel.

I spent a winter with 1 shovel - and then it broke and I was sad.

Stephen

That's actually both humorous and a good point - redundancy matter too.

Erica

Great discussion, folks. Do you have any final thoughts?

Noah

Circling back to something we talked about earlier, having the ability to recreate what you have is pretty crucial too. Let's say something happens and your backups are all corrupted because you didn't take the time to test your solution. Having everything documented and having an idempotent way to create your infrastructure again is critical.

Derek

I agree. Also - something that serious can also happen outside of a natural disaster.

i.e. AWS root account compromised and nefarious person deletes all your EC2 instances - and backups.

Noah

Heck, it might even remove the need for a more expensive DR solution. If your data layer is completely abstracted from your application stack then you just back up your data and push a button to recreate your application stack

Derek

Containers

Noah

My favorite

Stephen

Document, document, document. Don't rely on that one person in the company who has all the information in their head.

Erica

Awesome. Thank you all.