In the previous article, we installed and configured OWASP ZAP on an Azure VM and added a reverse proxy to access it over the internet. In this article, we’ll discuss on how to use the OWASP ZAP API and Visual Studio Unit Test project to create Automated Security tests and then run them in a Visual Studio Team Services build pipeline.

Create the Security Tests

To start off we’ll create an ASP.Net MVC web application project and a Visual Studio Unit Test project to store the Security tests. Web application is the application under test and it’ will be hosted on Azure App Services. We’ll only focus on creating the security tests.

Open up Visual Studio and Create a new project by navigating to File > New > Project…

Select ASP.Net Web Application and put in a name for the Web app and click OK. In the next dialog select MVC template and click Ok. I have Authentication set to Individual User Accounts.

Now the Web application is created. Next, add the unit test project by right clicking on the solution in the Solution Explorer and Add > New Project…

Select Unit Test Project and add the name for the project and click OK to include it in the solution. After creating the unit test project go the NuGet Package manager and update the MSTest.TestAdapter & MSTest.TestFramework NuGet packages to its latest version.

Then you need to add the NuGet package to access the OWASP ZAP API. In the NuGet Package Manager, search for OWASP.

The search results will include OWASPZAPDotNetAPI NuGet package, select it and click on Install to add it to the Unit Test project. Now it’s time to add the security tests.

Rename the UnitTest1.cs file to SecurityTests.cs. Then add the following variables to the Test Class.

using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using OWASPZAPDotNetAPI;

namespace OwaspZapSecurityTesting.Tests
{
  [TestClass]
  public class SecurityTests
  {
    private readonly static string _zapApiKey = "62simrufj37n98f5r0dmj68q8q";
    private readonly static string _zapUrl = "zap.k2vsoftware.com";
    private readonly static int _zapPort = 80;
    private readonly string _targetUrl = "http://k2vowasptestsite.azurewebsites.net/"; // Web App Hosted on Azure

    private static ClientApi _zapClient;
    private IApiResponse _response;

    [TestMethod]
    public void TestMethod1()
    {
    }
  }
}

The variables include API Key for OWASP ZAP, URL of the ZAP API, the port that the ZAP API is running on and the URL of the web application that needs to be tested. (The web app is hosted on Azure). Next, we need to have the ClientApi coming from the OWASP ZAP Nuget package which is used to access the ZAP API and the variable for the API response.

Note: You can get the API Key by opening up OWASP ZAP Application and navigating to Tools > Options… and on the Options dialog box click ok API menu item on the left. Then you can see the API key. Copy it and paste in as the value for the API key variable.

Next add the Initialize method with the ClassInitialize attribute where the client is instantiated by supplying the ZAP Url, port and the ZAP API Key.

[ClassInitialize]
public static void Initialize(TestContext context)
{
  _zapClient = new ClientApi(_zapUrl, _zapPort, _zapApiKey);
}

Next, we need to add methods to start spidering the web application and check the progress of spidering. Add the following code to the test class.

private string StartSpidering()
{
  _response = _zapClient.spider.scan(_zapApiKey, _targetUrl, "", "", "", "");
  return ((ApiResponseElement)_response).Value;
}

private void CheckSpideringProgress(string spideringId)
{
  int progress;
  while (true)
  {
    Thread.Sleep(10000);
    progress = int.Parse(((ApiResponseElement)_zapClient.spider.status(spideringId)).Value);
    if (progress >= 100)
    {
      break;
    }
  }

  Thread.Sleep(5000);
}

StartSpidering() method is initiating a spider scan by supplying the ZAP API key and the target URL for the rest of the configuration options that are passed as parameters,  pass empty strings. And then it will return the id of the spider scan to identify this particular scan, which we can use to check the progress of the scan.

CheckSpideringProgress() method does just that, the id of the spider scan is passed into the method and its used to query the progress of the scan and break out when the scan is done.

Next, we’ll change the default test method name to ExecuteSpider and add the code to execute the spider scan and progress check.

[TestMethod]
public void ExecuteSpider()
{
  var spiderId = StartSpidering();
  CheckSpideringProgress(spiderId);
}

For the rest of the scans that ZAP provides, it’s the same pattern we need to follow. Next, we’ll initiate an Active Scan of the web application. Let’s add the methods for executing the active scan and progress check for that.

private string StartActiveScan()
{
  _response = _zapClient.ascan.scan(_zapApiKey, _targetUrl, "", "", "", "", "", "");
  return ((ApiResponseElement)_response).Value;
}

private void CheckActiveScanProgress(string activeScanId)
{
  int progress;
  while (true)
  {
    Thread.Sleep(10000);
    progress = int.Parse(((ApiResponseElement)_zapClient.ascan.status(activeScanId)).Value);

    if (progress >= 100)
    {
      break;
    }
  }

  Thread.Sleep(5000);
}

StartActiveScan() method initiates an active scan by passing in the API key and the target URL and it will return the id for that active scan. Then CheckActiveScanProgress() method takes that scan id and checks the progress. Then we need to add the following test to execute the test

[TestMethod]
public void ExecuteActiveScan()
{
  var activeScanId = StartActiveScan();
  CheckActiveScanProgress(activeScanId);
}

This will call the methods we created and start an active scan and check for its progress.

Now we need to add the methods to generate the report after the tests are complete. ZAP API supports XML, HTML and Markdown reports at the moment. You can easily generate the reports by calling the zap client with the API key. Add the following code to the test class that generates the reports.

private static void GenerateXmlReport(string filename)
{
  var fileName = $"{filename}.xml";
  File.WriteAllBytes(fileName, _zapClient.core.xmlreport(_zapApiKey));
}

private static void GenerateHTMLReport(string filename)
{
  var fileName = $"{filename}.html";
  File.WriteAllBytes(fileName, _zapClient.core.htmlreport(_zapApiKey));
}

private static void GenerateMarkdownReport(string filename)
{
  var fileName = $"{filename}.md";
  File.WriteAllBytes(fileName, _zapClient.core.mdreport(_zapApiKey));
}

We need to supply the name for the report and the methods will write the report to each of the files. Finally, we’ll add the cleanup method to generate the reports and clean up the tests. Add the following code segment to the test class.

[ClassCleanup]
public static void CleanUpAndGenerateReport()
{
  _zapClient.Dispose();

  var reportFilename = $"{DateTime.Now.ToString("dd-MMM-yyyy-hh-mm-ss")}_OWASP_ZAP_Report";
  GenerateXmlReport(reportFilename);
  GenerateHTMLReport(reportFilename);
  GenerateMarkdownReport(reportFilename);
}

The test cleanup method will dispose the zap client and generate a report for the scan for all 3 types. And we are done. At this moment, your entire Test Class should look something like this.

using System;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using OWASPZAPDotNetAPI;
using System.Threading;
using System.IO;

namespace OwaspZapSecurityTesting.Tests
{
  [TestClass]
  public class SecurityTests
  {
    private readonly static string _zapApiKey = "62simrufj37n98f5r0dmj68q8q";
    private readonly static string _zapUrl = "zap.k2vsoftware.com";
    private readonly static int _zapPort = 80;
    private readonly string _targetUrl = "http://k2vowasptestsite.azurewebsites.net/"; // Web App Hosted on Azure

    private static ClientApi _zapClient;
    private IApiResponse _response;

    [ClassInitialize]
    public static void Initialize(TestContext context)
    {
      _zapClient = new ClientApi(_zapUrl, _zapPort, _zapApiKey);
    }

    [TestMethod]
    public void ExecuteSpider()
    {
      var spiderId = StartSpidering();
      CheckSpideringProgress(spiderId);
    }

    [TestMethod]
    public void ExecuteActiveScan()
    {
      var activeScanId = StartActiveScan();
      CheckActiveScanProgress(activeScanId);
    }

    [ClassCleanup]
    public static void CleanUpAndGenerateReport()
    {
      _zapClient.Dispose();

      var reportFilename = $"{DateTime.Now.ToString("dd-MMM-yyyy-hh-mm-ss")}_OWASP_ZAP_Report";
      GenerateXmlReport(reportFilename);
      GenerateHTMLReport(reportFilename);
      GenerateMarkdownReport(reportFilename);
    }

    private string StartSpidering()
    {
      _response = _zapClient.spider.scan(_zapApiKey, _targetUrl, "", "", "", "");
      return ((ApiResponseElement)_response).Value;
    }

    private void CheckSpideringProgress(string spideringId)
    {
      int progress;
      while (true)
      {
        Thread.Sleep(10000);
        progress = int.Parse(((ApiResponseElement)_zapClient.spider.status(spideringId)).Value);
        if (progress >= 100)
        {
          break;
        }
      }

      Thread.Sleep(5000);
    }

    private string StartActiveScan()
    {
      _response = _zapClient.ascan.scan(_zapApiKey, _targetUrl, "", "", "", "", "", "");
      return ((ApiResponseElement)_response).Value;
    }

    private void CheckActiveScanProgress(string activeScanId)
    {
      int progress;
      while (true)
      {
        Thread.Sleep(10000);
        progress = int.Parse(((ApiResponseElement)_zapClient.ascan.status(activeScanId)).Value);

        if (progress >= 100)
        {
          break;
        }
      }

      Thread.Sleep(5000);
    }

    private static void GenerateXmlReport(string filename)
    {
      var fileName = $"{filename}.xml";
      File.WriteAllBytes(fileName, _zapClient.core.xmlreport(_zapApiKey));
    }

    private static void GenerateHTMLReport(string filename)
    {
      var fileName = $"{filename}.html";
      File.WriteAllBytes(fileName, _zapClient.core.htmlreport(_zapApiKey));
    }

    private static void GenerateMarkdownReport(string filename)
    {
      var fileName = $"{filename}.md";
      File.WriteAllBytes(fileName, _zapClient.core.mdreport(_zapApiKey));
    }
  }
}

Next, it’s time to run the tests locally. Build the solution so that the Visual Studio Test Explorer can detect the tests. Then run the tests.

The tests should pass successfully. Since now we have verified that the tests are running correctly, we can add these to a build pipeline on Visual Studio Team Services.

Creating the Build Pipeline on Visual Studio Team Services

Now we will create a new project on Visual Studio Team Services and add the solution into a GIT repo. Then we’ll create a build pipeline to build and run the security tests.

Navigate to Visual Studio Team Service and create a new project. Give the project a name and click Create.

Then push the Visual Studio solution we created to the git repository and navigate to the code section of Visual Studio Team Services for the project you created. There you should see a button to create a new build definition.

Click on Set up Build button to create a new Build Definition.

We’ll create the build definition from scratch, so click on the Empty process on the next screen to continue.

Add a name for the build definition and select Hosted VS2017 as the Default agent queue. The repository and the branch are set automatically. (as you can see in the definition). Click on the Add Task button below and add the NuGet Task.

In the NuGet task configuration options, select the restore as the command and click on the browse button and selecting the solution. Then click on Add Task Button and add the Visual Studio Build task.

Again, select the Solution by clicking on the browse button and set the Visual Studio version to Visual Studio 2017. Then set the Platform and the Build Configuration. (they are set by using the Variables set in the Variable section)

Click on the Add Task button and add the Visual Studio Test task.

In the configuration options for the Visual Studio Test task, Set the Search Folder to the default working directory by using the built-in variable. The test assemblies section already has a matching pattern what will pick up the tests assemblies since we have tests in our Unit Test project name. Finally set the Test Platform version to Visual Studio 2017

Next, Add the Copy Files task to the build definition.

This task will grab the test result files using the pattern we supply and copy them to a folder we specify. Set the Source folder to the default working directory using the built-in variable. For the contents, add the pattern ***OWASP_ZAP_Report.* that will match the report file names we gave in the tests. For the Target Folder, create a folder with the name TestResults by combining the built-in variable with the folder name.  Then select the options to Clean Target Folder and Flatten Folders by checking the checkboxes and you are done.

Then to add the final task, click on Add Task button and add the Publish Build Artifact task to publish the test reports.

With the Publish Build Artifact task, we can have the test results file generated by the security tests, available for download for inspection. For the Path to Publish configuration option we provide which folder to be selected for publishing and then we provide a name for the artifacts. I’ve used the Build.BuildId built-in variable to generate the artifact. Then select the Artifact type as server and you are good to go.

Click on Save & Queue drop-down menu and select Save to save the definition.

Add a comment to the save options and a destination folder if necessary and click save to save the definition.

Click on the Queue button to queue a new build and you will see a link to the newly queued build. After the build completes, you should hopefully see that all build tasks completed successfully and the tests have passed as well.

In the build summary page, you can see the number of tests executed and the results of it. There you can see a link to download and explore the artifacts that were created by the build.

Click on the artifacts link and then click on Explore button to see the generated test results files.

You can see under the security-test-results folder, 3 reports are generated, one for HTML, XML, and Markdown. Click Close and click on the download button to download the artifacts as a compressed .zip file. If you extract and open the HTML report you should see something like this.

Here the security tests performed an active scan and uncovered some issues with the web application including 2 medium level issues and 3 low-risk level issues. You can see the details related to each of the report items below it.

Summary

This article focused on creating and executing security tests using OWASP ZAP, OWASP ZAP.Net API NuGet package, Visual Studio Unit Test Project and Visual Studio Team Services. One limitation of this approach is that we don’t have the ability to fail the tests if there are issues passing a certain security threshold. We can only run the tests and when it completes, we can download the test results. We will look into a way to fix this issue in a later article. I hope you enjoyed this second article of the Automated Security Testing with OWASP Zed Attack Proxy series, and I’ll see you in the next one.

The Sample Code is available on GitHub.

Automated Security Testing with OWASP Zed Attack Proxy -  All Articles

• #1 Installing & Configuring OWASP ZAP on an Azure Virtual Machine
• #2 Creating & Running Automated Security Tests on Visual Studio Team Services (This Article)
• #3 Working the Result of ZAP Security Scan to Pass or Fail the Security Tests