In the previous article, we created and ran Automated Security Tests on Visual Studio Team Services. One shortcoming of the security tests we wrote was that there was no way of failing the security tests if the result of the test exceeds a certain threshold. The test only executed the security scan and then gave the result of the test as HTML, XML and Markdown files.

To effectively have the tests on a Continuous Integration or a Delivery/Deployment pipeline, we should have the ability to stop continuing if a certain threshold we define exceeds when it comes to security testing. We will address this issue with this article. we can easily do this by inspecting the XML report file that is generated. The idea of what we are about to do is to parse the XML report returned by the security scan and compare the result with security criteria we defined.  To start off we need to represent the XML report that is returned by the security test with C# classes. You can inspect the XML document and create the classed by yourself or you can simply use a tool to generate the Classes by parsing through the XML document. You can use an online tool Xml2CSharp.com  to do this. Include the generated classes in the unit test project.

I’ve added all the classes that represent the XML document into the ZapReportModels folder in the unit test project.

Next, I’m going to add RiskCode class to represent the Risk Codes return in the Alerts with in the XML Report to make it easy when we are comparing risk codes. Add the following class to the unit test project.

namespace OwaspZapSecurityTesting.Tests
{
  public class RiskCode
  {
    public const string Informational = "0";
    public const string Low = "1";
    public const string Medium = "2";
    public const string High = "3";
  }
}

Next add another class SecurityScanResult to represent the number of alert types (Low alerts, Medium alerts etc.) returned by the report. And we will be using this to represent the security threshold we defined and to compare with the results returned by the security scan.

namespace OwaspZapSecurityTesting.Tests
{
  public class SecurityScanResult
  {
    public int InformationalAlerts { get; set; }
    public int LowAlerts { get; set; }
    public int MediumAlerts { get; set; }
    public int HighAlerts { get; set; }
  }
}

Next, we need to parse the XML document returned by the security scan to take out the security alerts. To do that add the following code segment into the SecurityTests.cs file.

private List<Alertitem> GetScanAlerts()
{
  OWASPZAPReport report = null;
  using (var memoryStream = new MemoryStream(_zapClient.core.xmlreport(_zapApiKey)))
  {
    var xmlDoc = new XmlDocument();
    xmlDoc.Load(memoryStream);

    var serializer = new XmlSerializer(typeof(OWASPZAPReport));
    report = (OWASPZAPReport)serializer.Deserialize(new XmlNodeReader(xmlDoc.DocumentElement));
  }
  return report.Site.Alerts.Alertitem;
}

This method will create a MemoryStream from the bytes returned by calling the _zapClient.core.xmlreport() method that returns the scan result in XML. Then we are using the MemoryStrem to generate a XmlDocument object and serialize it a type of OWASPZAPReport (Class we generated to represent the XML document) using a XmlSerializer. And finally, we return the AlertItems contained within the report.

And to finally get the count of the types of Alerts returned we need to process the AlertItems returned by the GetScanAlerts() method. Add the following code segment into the SecurityTests.cs test class.

private SecurityScanResult GetScanResults()
{
  var result = new SecurityScanResult();
  var scanAlerts = GetScanAlerts();

  foreach (var alert in scanAlerts)
  {
    switch (alert.Riskcode)
    {
      case RiskCode.Informational:
        result.InformationalAlerts++;
        break;
      case RiskCode.Low:
        result.LowAlerts++;
        break;
      case RiskCode.Medium:
        result.MediumAlerts++;
        break;
      case RiskCode.High:
        result.HighAlerts++;
        break;
    }
  }

  return result;
}

This is a simple method where we take the list of AlertItems and iterate through the items count the Alert types and finally return the results.

Next, we’ll remove the 2 test method we had earlier and instead add a single test method with the proper assertions to ensure we are meeting the security criteria we define. Remove the ExecuteSpider() and ExecuteActiveScan() test methods and replace it with the following test method.

[TestMethod]
public void ExecuteSecurityScan()
{
  // Spidering
  var spiderId = StartSpidering();
  CheckSpideringProgress(spiderId);

  //  Active Scan
  var activeScanId = StartActiveScan();
  CheckActiveScanProgress(activeScanId);

  // Security Criteria We define
  var expectedResult = new SecurityScanResult
  {
    InformationalAlerts = 0,
    LowAlerts = 3,
    MediumAlerts = 2,
    HighAlerts = 0
  };

  var actualResult = GetScanResults();

  Assert.IsTrue(actualResult.LowAlerts <= expectedResult.LowAlerts, "Low Alerts have exceeded the expected values.");
  Assert.IsTrue(actualResult.MediumAlerts <= expectedResult.MediumAlerts, "Medium Alerts have exceeded the expected values.");
  Assert.IsTrue(actualResult.HighAlerts <= expectedResult.HighAlerts, "High Alerts have exceeded the expected values.");
}

Here we are including the spidering and the active scan in the same test. In addition to that, we have defined security criteria by creating an object from SecurityScanResult class. The security criteria we define allows the application to have 2 Medium Security Risks and 3 Low Security Risks. Then we get the actual scan results and have 3 Asserts to make sure the actual security risks are less than or equal to the security criteria we defined.

Now the test is complete and it will break if the security criteria fails. Let’s push the code to Visual Studio Team Services and trigger a build. The security criteria we defined for this build allows No High Risks, 2 Medium Risk, and 3 Low Risks. And we know the application returns exactly 2 Medium and 3 Low security risks. So, this build should PASS.

As you can see the test passed since the security scan results were within the criteria we defined. Now let’s define stricter criteria and run the build again. This time, the new security criteria allow No High Risks, No Medium Risks and 3 Low Risks. And we’ll trigger a build again.

And now, you will see that the build FAILED. This happened because the security scan result exceeded the security criteria we defined.

If you go the detailed test report, you can see that the assertion failed for medium security risks since our threshold was 0 medium risks and the scan returned 2 medium risks. Now you can see this way you can define security thresholds for your application and include these security tests in a build pipeline or a release pipeline and stop continuing if the tests fail.

Summary

In this example, we have the security tests included in a build pipeline. But in a real-world scenario, these security tests will be included in a release pipeline. Since these tests require a running web application hosted in your environments. So normally these tests are running after the application is deployed to an environment.

So, in a scenario where we have a release pipeline deploying to 3 environments Dev, Test and Staging we can include the automated security tests in the middle of Dev and Test deployments. The security tests will be running against the Dev environment and if the tests fail, the application will not be deployed to the Test and Stage environments, preventing security issues from propagating to later environments in the release pipeline. So you can see, having automated security tests included in your release pipeline is a good line of defense against security issues creeping into your production environments and ensure the quality of your web applications.

The complete solution can be found on GitHub (branch ‘test-with-assert’)

Automated Security Testing with OWASP Zed Attack Proxy -  All Articles

• #1 Installing & Configuring OWASP ZAP on an Azure Virtual Machine
• #2 Creating & Running Automated Security Tests on Visual Studio Team Services
• #3 Working the Result of ZAP Security Scan to Pass or Fail the Security Tests (This Article)