How to Assess and Audit DevOps Security to Improve Business Value

How to Assess and Audit DevOps Security to Improve Business Value

The ISACA guidelines for auditing DevOps recognize the potential business value of DevOps, but they stress that there are assurance, governance, and security factors to consider. “ This guidance outlines these considerations: the risks associated with DevOps (in both adoption and non-adoption), controls which can help mitigate key risk areas, and specific actions which practitioners can take to increase the transparency of their DevOps practices, as well as improve overall assurance levels around the effectiveness of those practices,” reads the report.

The ISACA guidelines separate DevOps into several components with specific considerations: culture, processes, collaboration, communication, and automation. Some fundamental principles that support a secure DevOps environment include the adherence to accepted security standards, such as the ARIS (Automated Reference Information for Security) taxonomy and NIST cyber-security framework. To properly support DevOps practices, these standards must be maintained throughout a company’s development processes.

The emergence of DevOps has created new challenges and opportunities in software development. It has required greater transparency in code review and vulnerability discovery that necessitates tools to provide maximum visibility into source code. A vital tool in a DevOps environment is continuous integration, which allows for automated testing and builds.

“How do you continue to develop secure applications when developers are often not aware of changes made by others? Security automation can help address this challenge by enabling effective testing and validation of software before it is released into production,” states the ISACA report.

Automated security tools can be used to identify potential vulnerabilities in code at all stages in its life cycle. Unit tests can also be automated as part of the continuous integration process, providing an additional layer of protection against defects that could lead to security issues.

Security monitoring should also be automated to ensure no security breaches and that the system is not compromised from outside sources. Automated alerting can provide visibility into potential issues and vulnerabilities before they become a problem. In addition, effective logging of DevOps activities provides visibility into how changes are made throughout the development lifecycle.

DevOps does not have to be a security nightmare. The ISACA guidelines provide a set of principles for ensuring DevOps implementation in a secure manner that improves business operations and reduces risk exposure.

“Risk management must now extend into the development process, which is facilitated by DevOps automation,” said Mark Weatherford, member of the Technical Steering Committee for the NIST Cybersecurity Framework. “ISACA’s guidance provides recommendations for risk management activities that will help enterprises improve how they manage information security risks in a DevOps environment.”

Let’s explore some controls:

Automated scanning of software

To keep up with the more rapid release cycle of a DevOps environment, ISACA recommends an automated scan to find security configuration issues in code. The document suggests that auditors observe application code scanning tools and log files to confirm that scans occur.

Automated scanning for vulnerabilities

In addition to scanning software, automated vulnerability scans are also recommended. The document discusses how DevOps platforms such as Chef or Puppet automate configuration management and

May introduce changes and vulnerabilities dynamically in the environment. To help ensure security, it’s a good idea to trigger some automated scan before you release new software onto your system for deployment through the DevOps pipeline.

The benefit of automation: The ability to utilize automated scanning tools helps operationalize the security process and reduces the risk and workload involved in discovering and patching vulnerabilities before they can cause problems. In addition, using an automated scanning tool means that you are consistently running scans throughout your DevOps life cycle, which is a good thing for both developers and those responsible for security.

Web application firewall

It is common for automated scans to uncover vulnerabilities that are not immediate priorities. Installing a firewall or some other type of filtering device on the network will keep internet traffic out while you address these underlying issues and critical vulnerabilities.

Using this technology also addresses an increased attack surface as the number of applications and services increases in a DevOps environment.

Automated tools such as WAF can be used to block incoming attacks or prevent them from spreading throughout your network.

The benefit of automation: Automating the implementation of a web application firewall ensures that security is not an afterthought but rather a critical component of the DevOps pipeline. Also, WAF technology is likely to be needed from day one and must be put in place right away or made part of a post-launch remediation plan.

Developer training on application security

The importance of developer training on secure coding techniques and how to avoid common security configuration issues are not unique to DevOps. An ISACA auditor will assess evidence of developers attending or participating in appropriate training courses. For example, the SANS Institute offers a course in secure coding.

A DevSecOps team should also have some requirements or guidelines for developers to attend training and ensure evidence of compliance with these guidelines.

Benefits of training: Training is critical to the success of the DevOps movement. Not only does it raise awareness and give developers the tools they need to create a secure environment, but it also helps build a safer culture throughout your organization. Training can also reduce an organization’s long-term viability if it incurs a data breach or other security incident.

Managing software dependencies

When you combine DevOps, containers, open-source tools, and faster app deployment, one of the risks is software dependency. Apps are built on APIs, open-source libraries (often also used by other apps), and other middleware, all of which can introduce vulnerabilities at some point. Therefore, ISACA recommends that organizations keep track and manage supporting libraries to control their application environment.

Benefits of dependency management: Managing dependencies allows you to track vulnerabilities based on your use of specific third-party software libraries. It also helps ensure that security is factored into the DevOps pipeline from inception and that Vulnerability Management is part of your ongoing operations.

Logging access and activity

DevOps plays a significant role in preventing data breaches, and to do this, they need some form of identity and access management. ISACA suggests that when using DevOps to implement IAM, it needs a developer activity log with information about changes made and who made them. Hence, there is always an audit trail.

Benefits of logging access and activity: Having an activity log is essential in DevOps. Without it, you can’t know if your security was compromised by someone modifying code or data without authorization. Log management also provides evidence for compliance with many regulations such as PCI-DSS.

Other benefits of log management include faster detection of security incidents, reduced risk exposure and compliance failures, reduced investigation time and resources, easier prioritization of investigations, and more. Log Management also allows a platform view across all applications in your environment, essential for effective governance processes.

Documentation of policies and procedures.

Establishing clearly defined policies and procedures is a standard best practice for any environment, but the dynamic nature of DevOps makes creating and maintaining these policies even more important. Therefore, ISACA recommends auditors review all relevant policies to ensure companies are implementing them effectively.

Benefits of documentation: Documentation is key to building a successful DevOps program. All policies and procedures must be documented so they can be easily communicated across the organization. Having them in place also helps ensure compliance with governance, regulatory, and internal controls requirements.

Continuous Monitoring (CM) / Continuous Adaptation (CA)

Continuous Monitoring and Continuous Adaptation answer one of DevOps’ most significant challenges — security controls that must be effective for long periods. Since there is no fixed release schedule in DevOps projects, it’s not feasible to wait until new feature development is complete before implementing security controls. ISACA recommends organizations address this ongoing challenge by implementing a Continuous Monitoring and Continuous Adaptation program.

Benefits of continuous monitoring: CM provides an opportunity to identify gaps in security controls early before they become problems. It can also provide better visibility into production issues, leading to faster resolution time for IT teams.

Benefits of continuous adaptation: CA ensures that risks are addressed quickly, and mitigations are tested to ensure effectiveness. In addition, CA allows companies to take advantage of new attack vectors that were previously unknown, as well as fix vulnerabilities more efficiently.

DevOps should be considered when determining an organization’s risk posture. Therefore, ISACA recommends organizations conduct proper risk assessments for potential DevOps implementations. That vulnerability assessment should consider the level of collaboration, accessibility (and exposure) to system controls, as well as any new risks or vulnerabilities that DevOps may introduce.

If your organization is solely focused on information security, you’re likely missing out on one of the most extensive opportunities for cost savings — implementing DevOps practices. Begin by focusing on security tools and processes that help create a better product. The more value you provide for your customer, the more they’ll be willing to spend on your organization’s products. ISACA recommends companies find ways to improve the quality of software and information to prevent errors and faults in products.

Managing the performance of applications.

Application performance management is an important tool for developers that can help anticipate and resolve any issues in the development process. If apps are not running correctly or are performing poorly, it impacts productivity. Therefore, ISACA recommends organizations use APM and metrics to proactively address issues during the app creation process.

Benefits of APM: With APM, users can gain insight into how apps perform in production and address issues before impacting the user experience. This can help create more stable functions that run at optimal performance levels minimizing downtime for application failure.

Asset management and inventory

Any organization must maintain a current and accurate asset inventory, including the asset owner, business purpose, physical or virtual location, and other relevant details. DevOps accelerates the pace of development and app deployment, making it more challenging to stay on top of this. The ISACA control calls for an automated tool or manual process to maintain a current and accurate inventory.

Benefits of an asset inventory: This can help organizations ensure assets are secure by knowing the exact location, state, use, and owner of each item. An accurate inventory can also help reduce costs through standardization and automation; it’s common for multiple teams to own the same assets.

Continuous audit and monitoring

With continuous integration, automation testing, and deployment, organizations should also perform auditing and monitoring to ensure the validity of their processes. The ISACA control calls for tools to help test and monitor the effectiveness of controls.

Benefits of continuous monitoring: The ability to test and review security processes gives organizations a competitive advantage by ensuring compliance. It also ensures that development teams are consistently maintaining, testing, and improving their software and products. This can help prevent any security issues that may arise from poorly developed or tested code.

Because DevOps teams work on software continuously and release new versions more frequently, they need to effectively audit changes without spending too much time analyzing them. Thus, DevOps controls should help with continuous flow monitoring, auditing, and other practices.

Benefits of continuous flow monitoring: Applying application security testing as a continuous flow can help determine if there are any flaws in the software, and if so, where. The earlier you find these flaws, the more time you have to address them before it reaches production. This can help identify potential issues with new code before production deployment.

Benefits of continuous security: The speed at which new features and changes are deployed can sometimes make it hard to test for vulnerabilities — especially in ICS environments with a long-life cycle for updates or patches. This control applies to DevOps because it requires testing within the production environment as much as possible, thus improving test coverage and efficiency.

Benefits of continuous risk: The DevOps approach is about continual improvement; it provides an opportunity to constantly update and improve security processes to address vulnerabilities as new threats emerge. This can help prevent any security breaches from introducing new code before testing for potential issues.

What Security Risk Analysts and Auditors can do to improve DevOps

Code analysis and security reviews

Auditors should look into how application code is reviewed and validated, including reviewing the test cases to confirm they are effective. In addition, all changes made as part of a release should be systematically documented, with unified change management across the organization. Changes should also be communicated promptly to handle any issues before they become a problem.

Coding standards and code review

Developers should have a coding standard to help them avoid errors or vulnerabilities in the application code. Security policies and procedures should be implemented early in the development lifecycle (for example, at the architecture planning stage), and they should also be included in the code review process. This provides visibility into how the security standards are being implemented.

Auditors should also look into how the code review process is conducted to ensure effectiveness and follow a standard methodology. For example, does the organization use a formal peer review (e.g., red team), or is there more ad hoc feedback from others on development activities? In addition, auditors can check whether the code review process is part of continuous integration so that it takes place automatically as changes are being developed.

In addition to reviewing the code and testing for defects, auditors should also confirm how security vulnerabilities are identified. For example, does a vulnerability management program exist? If so, who is responsible for handling reported issues?

Security configuration management

Auditors should consider whether proper configuration management practices are being used to modify the application and operating system, including any changes made as part of a DevOps release process. In particular, auditors need to confirm that security settings are not modified outside of the standard configuration management processes for the organization or application. In addition, any additional configurations should be thoroughly documented.

Auditors should also confirm that the organization uses a centralized configuration management database to capture, track, and manage changes throughout the software development life cycle. In addition, they should examine how any access controls are configured to permit proper file system permissions and protect data during batch processing or in-memory transactions, and address issues related to logging and auditing.

Auditors should also confirm that testing of application security configurations is performed, including any system-level security controls (e.g., logging, network controls). All changes made to the code or configuration should be validated using automated tools and manual methods before releasing them into production. This helps ensure that the release was performed successfully, and vulnerabilities have not been introduced along the way.

Auditors should also confirm how security configuration items are controlled, including who is responsible for making changes to the system and ensuring that they are implemented successfully. Any additional configurations should be thoroughly documented and made available to other members of the organization. Configuration management software tools can help ensure application integrity throughout the software development life cycle. For example, tools such as Microsoft’s System Center Configuration Manager, Apache Tomcat’s mod cluster, and Google App Engine’s Cloud Build Service help developers manage application component versions across the entire lifecycle of an application, including configuring policies for automated updates and changes to configuration settings.

Auditors should confirm that security configuration management is performed on the application components used by developers during testing, including any third-party components. For example, auditors should determine where software libraries with known vulnerabilities are being used. This helps address the risk that a vulnerability in one component may affect other applications using that same component.

Auditors can help uncover these vulnerabilities by analyzing security configuration management changes to the application. For example, auditors should confirm that any changes made to a library during development are consistent with the library vendor’s secure coding guidelines and recommendations.

Auditors can also help uncover issues related to data protection by examining how security controls are configured on servers used for batch processing or in-memory transactions (e.g., data warehouses, in-memory databases).

Auditors should also confirm that logs are generated to document any changes made to security settings or application configurations and provide a level of detail sufficient for reconstructing the events leading up to an incident. This can be implemented using change-managing systems, configuration management tools, software development tools, and repository managers.

Auditors should confirm how the application is configured (e.g., access controls) to protect data during batch processing or in-memory transactions, such as whether the data is encrypted at rest or in transit and what controls are implemented on any intermediate systems used for transmitting the data. In addition, auditors can help address this issue by reviewing any data transfer agreements in place (e.g., Data Transfer Documents) and confirming that sensitive data is not released to unauthorized parties, such as partners or subcontractors, during the software development life cycle.

Executing tests following the implementation of application security controls help ensure that they are adequately implemented and operating as intended. Too often, applications are pushed into production without adequate testing. Testing also serves as a form of validation for security configuration management controls, which can help prevent issues that may otherwise go undetected until launched or released into production. As part of their overall review strategy, auditors should determine if the appropriate tests were performed on all software components before being deployed into production, particularly those with known vulnerabilities.

Auditors should also confirm that the application developer has established and is following a process for reporting, tracking, and fixing security issues discovered during testing to verify that any applications in production or pre-production were patched as soon as possible. Where manual processes are used to report vulnerabilities, such as sending emails to a central mailbox (e.g., a ticketing system), auditors should confirm that any security incidents are identified and tracked.

Auditors can help uncover these issues by reviewing how vulnerabilities are discovered, reported, evaluated, patched, and communicated to the development team. They should also determine whether the application developer has established internal processes for reporting potential vulnerabilities during testing before any code is pushed into production. A lack of documented processes can represent a significant risk to an organization because it increases the likelihood that vulnerabilities will be missed and the application will remain unpatched, putting sensitive information at greater risk of compromise.

Auditors should review any security incident response procedures to determine how critical systems are protected from attack by insiders or external attackers in an incident. The procedures should include responding and containing the attack, restoring service as quickly as possible, and identifying any evidence that the attacker may have left behind.

Auditors can also help confirm that security monitoring tools are implemented to detect attacks on critical assets and possible breaches, anomalies, and incidents across the entire attack surface. For example, application developers should ensure they can monitor logical (e.g., network-based) and physical (e.g., host-based) security controls. The effectiveness of an organization’s security monitoring program depends on having a continuous focus on identifying new threats based on computer code analysis or community reports. Also, identifying known vulnerabilities in the application development kits and frameworks used. And common attack methods (e.g., SQL injection, cross-site scripting) can be detected using existing security controls such as firewalls and intrusion prevention systems and automated tools deployed to deliver alerts about potential issues with logical and physical infrastructure.

Auditors should also evaluate how an organization’s continuous monitoring program can detect and alert on common attacks, such as cross-site scripting (XSS), SQL injection, and directory traversal. For example, security monitoring tools that focus primarily on network flow analysis to identify potentially malicious traffic might not detect a SQL injection or XSS attack because the attack originates from within the application and not from a third-party source such as an external attacker.

Auditors can assess how quickly security incidents are identified and contained by testing whether web applications are deployed with protection to prevent XSS attacks. For example, auditors can test whether applications automatically encode or escape special characters that could result in an XSS vulnerability.

Auditors should review any documentation about testing the application to determine whether all components of an organization’s infrastructure have been tested for vulnerabilities. Applications residing on systems under different administrative control could represent a single-point-of-failure if they are not patched and patched simultaneously. To mitigate this risk, application developers should ensure that they use a system with access to the infrastructure used to deploy their web applications so that security controls are in place and protected.

Auditors can help determine if an organization has established an effective incident response process by reviewing any testing evidence. For example, auditors can consider some of the scenarios outlined above:

what would happen if a hacker gained access to sensitive information or broke into the application? What security controls, such as firewalls and intrusion detection systems, would be used to detect an attack from a third-party source, and how quickly would they react? If network monitoring tools detect a possible attack, which decides whether to block traffic based on the risk posed by a particular attack? Are there consequences for making decisions based on false positives or other security events that turn out to be benign?

What would happen if an insider launched an attack and steals sensitive information from the application or causes damage to the system? What steps have been taken to identify employees who might pose a risk of misusing their privileges?

Identifying Where to Implement Security Controls

Security controls are most effective when they are implemented appropriately. For example, when conducting network-based monitoring, organizations need to ensure that the correct traffic is monitored. In addition, one is more likely to observe malicious traffic in a location with direct access to sensitive data than in another area unlikely to receive sensitive data. Therefore, security controls should be implemented where they are most likely to detect malicious traffic.

A practical application security program reduces an organization’s potential exposure from all types of common attacks, including SQL injection, XSS, directory traversal, and cross-site request forgery (CSRF). In addition, by identifying the attack vectors and vulnerabilities that are most likely to result in an exploit, organizations can focus their efforts on addressing these threats first.

Auditors should determine whether web applications have been tested for the following:

· SQL injection

· Cross-site scripting (XSS)

· Ciphertext injection

· Cross-Site Request Forgery (CSRF)

· Common web application flaws

Auditors can use the OWASP top 10 list of most critical web application security risks as a guide: http://www.owasp.org/index.php/Top_10_2010-A3 . However, because this is very high-level guidance, auditors should not rely on it to determine whether a specific application is secure.

Mitigating Common Web Application Vulnerabilities

The following sections describe steps organizations should take to help mitigate common web application vulnerabilities, including SQL injection, XSS, and CSRF:

Control network access to the database server. Because most attacks targeting web applications will start with an entry point on the network, database servers should be segmented from the public network where possible. In addition, organizations should implement firewalls to prevent access to sensitive information and restrict which hosts can access the database server from a particular network. Finally, organizations need to ensure that only authorized users have access to their databases. For example, auditors can check if the database software supports access controls to restrict access using hostnames and IP addresses.

Prevent direct external access to application endpoints. Organizations should implement several layers of defenses in front of web applications, including proxies, firewalls, and intrusion detection systems (IDSs) to prevent direct connection into an application or a database through the Internet. In addition, because some attacks, such as SQL injection, do not require direct access to the application or the database server, intermediaries can help prevent these types of attacks by enforcing authentication and authorization.

Attack detection systems should be deployed in front of applications. Attack detection systems can detect suspicious activity (e.g., an increase in outgoing communications or traffic to unusual destinations). It is often more effective to reduce the attack surface by preventing access into an application than attempting to detect malicious activity after an organization has been compromised.

Implement input validation mechanisms. Input validation involves filtering out unexpected characters and malformed data from user inputs before processing them for their intended use. For example, if an application processes a user’s first name and last name, the input validation mechanism should check to ensure that the values are not greater than expected. Input validation mechanisms can effectively mitigate SQL injection attacks by preventing attackers from entering unexpected or malformed data that could trigger unintended commands on the database server.

Implement output encoding mechanisms. Output encoding mechanisms help prevent attackers from injecting malicious characters that could be interpreted by application logic as legitimate commands, thereby avoiding a SQL injection attack. For example, most web applications can serve HTML encoded input data (e.g., characters < > & ‘. / “) if the output encoding mechanism is correctly configured.

Implement cross-site scripting countermeasures. The goal of cross-site scripting (XSS) is to inject malicious scripts into an application that the user can execute. Although XSS attacks have been known to cause high impact incidents, there are several countermeasures organizations can implement to mitigate these risks depending on the organization’s security strategy and risk tolerance:

Change default file extensions. For example, an application may store files with the .html extension, which attackers could use to launch a cross-site scripting attack. Changing the default file extension (e.g., from .html to .txt) can help prevent this type of attack.

Enforce strict input validation for user-supplied input. Input validation mechanisms can help prevent malicious scripts from being injected into application-controlled data.

Enforce strict output encoding for user-controlled output. Output encoding mechanisms are designed to prevent attackers from injecting malicious characters that could be interpreted as legitimate commands, thereby avoiding a cross-site scripting attack. For example, an application should not encode output that is used as part of a query parameter.

Implement application-specific countermeasures. In some cases, organizations may develop custom defenses to mitigate the risk associated with malicious script injection. For example, suppose an application does not correctly validate user input or interpret it for its intended use (e.g., treating it as an HTML directory). In that case, it may be possible to modify the application source code.

Implement password brute force attack mitigation mechanisms. While password complexity and lockout policies can help prevent users from using easily guessed passwords, they do not stop attackers who try to guess passwords more systematically. For example, password “dictionary” attacks use pre-built word lists to make brute-force attempts. To mitigate this type of attack, an organization can implement a password lockout policy that limits the number of failed authentication attempts allowed before the attacker is blocked from initiating any additional login attempts.

Implement user account lockout policies. Account lockout policy refers to the maximum number of times a user can incorrectly attempt to authenticate before a system blocks the user from attempting another login. This policy is usually combined with periodic account audits that remove users who no longer need access to the system.

Implement multi-factor authentication or secondary authentication factors as part of an overall defense-in-depth strategy. While passwords are still vulnerable to guessing and brute force attacks, a multi-factor authentication mechanism that uses something you know and something you have can help increase the level of security for your organization.

Implement web application firewall protections. Web application firewalls (WAFs) proxy HTTP requests from users and evaluate them for malicious activity before passing them on to underlying applications. If WAFs are configured in line with your security strategy, they can help reduce the risk associated with some of the attacks described in this chapter. For example, WAFs can inspect HTTP requests and responses for malicious payloads and URL strings related to SQL injection.

Implement transport layer protection mechanisms (e.g., SSL/TLS). Using HTTPS to encrypt communication between web servers and clients can help prevent attackers from intercepting traffic flows for use in other attacks.

Implement secure coding practices. Web applications that are vulnerable to attack may have been created with insecure code, or they may have been developed without security specialists or security teams (i.e., developers who are responsible for the application’s security) on the development team. Secure coding practices such as following security-related industry best practices, performing threat modeling and threat analysis activities, identifying potential attack vectors and implementing countermeasures, using secure coding techniques (e.g., sanitizing input data to prevent code injection attacks), understanding how different languages natively handle input validation, and other mitigation techniques can help prevent vulnerabilities from being introduced into applications.

Properly configure and maintain web servers, application servers, database servers, firewalls, routers, switches, load balancers, and other network infrastructure components to mitigate unauthorized access or malicious activity. For example:

* Enable logging capability so that you can audit activity and detect anomalous behavior.

* Use security controls to help reduce the risk of exploiting known vulnerabilities in software, services, or configurations (e.g., default credentials).

* Use a network firewall as an additional layer of defense to help prevent unauthorized network traffic from reaching servers.

Other general information security practices that all organizations should follow:

Implement a solid information security policy, which may also be an acceptable use policy or a code of conduct. In addition to outlining minimum standards for behavior and conduct, the document can provide information about users’ expectations concerning protecting organizational resources and data.

Communicate the information security policy to all employees, contractors, and other individuals who access internal systems or data.

For more information contact Celestin Ntemngwa on LinkedIn