Automate Incident Management with Code

An incident in a production environment is never a good thing to deal with. However, those things happen. Even though not very common. Once it happens, you have to be ready for it. Better planning is key to handling incidents. What if a support model includes a prompt for the system involved to automatically handle tasks such as triage and incident response to an incident. If that is the case, I suppose your collaborators will find themselves working peacefully and with resilient systems and subsequently add more value to the company, probably through better customer services and quality products or services.

Response-as-code

Response-as-code (RaC) is a term that I came across, and it is part of the evolution of software development. Software development has brought about new techniques and “products.” We have Infrastructure-as-Code, where the infrastructure is described in machine-readable language and definition files and store these files in some code repository together with your source code. This gives you a single source for your application and the needed infrastructure for its deployment.

Similarly, with Response-as-Code, plan, you check-in tools and solutions together with your code. The code provides the foundation to automatically identify incidents and resolve problems without involving people such as engineers.

The RaC components include generic components to use across all projects to identify and resolve common problems, project-specific components for issues specific to an individual project. These components are then connected for a comprehensive system.

RaC process starts with a playbook. From the playbook, identify common production problems for your team. Determine if the problem is unique to a system or service or more generic across different services — for example, high network traffic and slows performance. Design a programmatically automatic response. The programmatic response should have an escalation path if the programmatic response fails or does not work in all scenarios. Or if the problem reaches a certain threshold and needs a human to intervene. You could leverage APM application performance monitoring to alert based on specific criteria. You can use webhook or triggers to API to activate a script to fix a problem. For example, AWS Lambda can be used to automatically resolve infrastructure problems(Mackory, 2020).

Identify and design patterns to detect issues.

Identify the critical technology stacks across the organization and compile a library of code solutions that automatically detect common problems. Referencing past issues will help in the future identification of similar issues programmatically.

Build your solutions

You have a collection of potential problem identification and troubleshooting tools and a library of possible solutions by now. For example, an AWS Lambda solution could be used to resolve many AWS related issues. The lambda logic could also be used in other clouds and for on-premise solutions.

Continuously Improve

The RaC process is a continuous one. Share your solutions and continuously improve, adding new features, maintain the code, etc.

Also, there is a lot of free and paid tools to automate the incident process. Some of these free tools include, but are not limited to, TheHive, AlienVault, GRR Rapid Response, Cyphon, SANS Investigative Forensics Toolkit(SIFT), Volatility, CrowdStrike CrowdResponse, and Cyber Triage.

Reference

Mackrory, M (2020). How to Automate Incident Management with Code and Get Better Results. Retrieved from https://thenewstack.io/how-to-automate-incident-management-with-code-and-get-better-results/

--

--

--

Love podcasts or audiobooks? Learn on the go with our new app.

Recommended from Medium

Challenges.re Linux e12 Writeup — Reverse Engineering a 64 bit Linux Binary with Ghidra

《猫和老鼠》 完整版 【Tom & Jerry】 (2021) 新电影动画 在线观看 HD

CKAD Exam Overview

4th week, learning more than coding.

How to embed a Slack channel in a web page

Apache Hudi EMR set up

Tips for creating an iOS Framework

They structure entire half local.

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store
Celestin Ntemngwa

Celestin Ntemngwa

More from Medium

Move to the cloud to build a better business, not more servers

Exploring The Possibilities of Infrastructure as Code

Where did my GitHub contributions go ?

Hard and Symbolic links — Same or different?