I’ve been creating extensions for Azure DevOps (formerly VSTS) for a few months now and am quite fond of it. And recently, my Slack Notification Extension for Azure DevOps reached 1000+ installs which I am so excited about and it also made me realize that more people are using it means, larger the impact if I mess things up. I really needed to think about the quality of the extensions I write.

There was a lot to think about and many things to improve. I will write about what exactly I felt that was needed to improve on another article. I’ll focus on the issues I faced when releasing the extension and what I did to improve that process. Some of the pain points I faced was;

• Testing the extension at its final form, where I can download it from the marketplace and run some real-world usage tests
• I had to release the extension to the marketplace and then run the tests.
• If for some reason, the extension is not working properly, I had to redeploy the previously working state as a new version.

I had an idea about what needs to be done to restructure the way I deploy the extension but never got around to do that. Then recently, there was this awesome episode of RadioTFS where Jesse Houwing talked about Azure DevOps, Extensions and Release gates which pushed me over the edge so to speak.

https://twitter.com/jessehouwing/status/1052905578677329920

And the conversation that followed with Jesse helped me a lot to get to the point where I am now with releasing the Slack Notification extension.

So, let’s dive into the details of how things are set up now.

Building the Azure DevOps Extension

The extension is a cross-platform task written in TypeScript. The code is hosted in GitHub. The task is built and tested using npm scripts. The build process looks something like this in Azure Pipelines.

The first 2 tasks are for replacing the Sentry related configuration options, where I set the Sentry DSN and the Release Version of the task using another task I created called RegEx Match & Replace. We’ll talk about How to configure Sentry to capture exceptions from your Azure DevOps Extensions on a separate article.

Then I install the npm dependencies using the npm task and then build and run the unit tests for the extension. The next 3 Copy Files tasks will copy the Extension Manifest file, the markdown documentation, marketplace icons, the screenshots and the transpiled JavaScript code to the staging directory where I publish the build artifacts.

Finally, I use the production version of the Slack Notification task to send me a notification depending on the success/failure of the build. These 2 tasks are encapsulated into a Task Group since I am reusing the same setup in my other build and release pipelines.

These steps are pretty straight-forward when it comes to building TypeScript based Extensions. But next is the interesting part

Releasing the Azure DevOps Extension

This is where the major changes happened during the revamp. I am using Azure DevOps Extension Tasks to package/publish the extensions I make to the Visual Studio Marketplace. Earlier I was releasing the new version of the task to the marketplace and then once its published then I run the tests to make sure it’s working.

This was not ideal since I might release a version that did not work as expected and it is released to the entire user base. And If I had issues in the new version that needs to be reverted and re-released to make sure the users are not affected.

With the revamp I did things differently, this is how the new release pipeline looks.

Now I release the extensions, with a slightly different Name and a different Task ID to the marketplace and also mark its Extension Visibility as Private. And then share the extension with my own Azure DevOps account so I can test it. This was the new version of the task is not visible for the public and I can run the tests I want in any new version I release before I release the public version.

Before I try to use the newly release Dev version of the task, I need to make sure that the task is published successfully to the Visual Studio Marketplace. The marketplace is running scans on the extensions to make sure the extensions does not contain any unwanted stuff in there. This validation usually happens quickly, but it can take time and delay the publishing of the extension or even not publish the new version of the extension.

So, if that happens you need to wait to see if the extension is successfully published before you run the tests on the newly released version of the extension. To make sure this happens, I used the Release Gates feature of the Azure Pipeline Releases and the Is Valid Extension task (to Check Marketplace Validation Status) which is part of the Azure DevOps Extension Tasks.

I can set the Evaluation Options to check the status every 5 mins to make sure the task is published to the marketplace.

If the publishing fails, the pipeline will fail, and I will be notified. As soon as the publishing is completed, I can start running tests.

As the next step after the “Dev” release, I have an environment where I run some real-world usage tests to cover specific scenarios. This uses the newly released Dev version of the task.

Once all the usage tests have passed, the task is ready to be published. Now all I do is override the task version, and then use the default Extension name and the Task ID and set the Extension Visibility to Public and release the extension.

After this, the extension is released to the public and I can be sure that the extension is working as intended. The Azure Pipeline release overview looks something like this after the revamp.

This is how I modified the release pipeline for my Slack Notification Azure DevOps extension to improve the quality of the extensions I release. I plan to roll out the same type of a setup for all my Azure DevOps extensions soon. Let me know what you think. Would you have done things differently? Anything that I can improve? I am open for suggestions.