INSUBCONTINENT EXCLUSIVE:
Kolton Andrus
Contributor
Share on Twitter
Kolton is co-founder and CEO of Gremlin, the
chaos engineering company helping the world build a more reliable internet.
I recently had a scheduled video conference call with a
Fortune 100 company.
Everything on my end was ready to go; my presentation was prepared and well-practiced
I was set to talk to 30 business leaders who were ready to learn more about how they could become more resilient to major
outages.
Unfortunately, their side hadn''t set up the proper permissions in Zoom to add new people to a trusted domain, so I wasn''t able to
We scrambled to find a workaround at the last minute while the assembled VPs and CTOs sat around waiting
I ended up emailing my presentation to their coordinator, calling in from my mobile and verbally indicating to the coordinator when the next
slide needed to be brought up
Needless to say, it wasted a lot of time and wasn''t the most effective way to present.
At the end of the meeting, I said pointedly that if
there was one thing they should walk away with, it that they had a vital need to run an online fire drill with their engineering team as
Because if a team is used to working together in an office — with access to tools and proper permissions in place — it can be quite a
shock to find out in the middle of a major outage that they can''t respond quickly and adequately
Issues like these can turn a brief outage into one that lasts for hours.
Quick context about me: I carried a pager for a decade at Amazon
and Netflix, and what I can tell you is that when either of these services went down, a lot of people were unhappy
There were many nights where I had to spring out of bed at 2 a.m., rub the sleep from my eyes and work with my team to quickly identify the
I can also tell you that working remotely makes the entire process more complicated if teams are not accustomed to it.
There are many
articles about best practices aimed at a general audience, but engineering teams have specific challenges as the ones responsible for
keeping online services up and running
And while leading tech companies already have sophisticated IT teams and operations in place, what about financial institutions and
hospitals and other industries where IT is a tool, but not a primary focus? It often the small things that can make all the difference when
working remotely; things that seem obvious in the moment, but may have been overlooked.
So here are some tips for managing incidents
remotely:
There were many nights where I had to spring out of bed at 2 a.m., rub the sleep from my eyes and work with my team to
quickly identify the problem… working remotely makes the entire process more complicated if teams are not accustomed to it.
