Azure DevOps agent architecture

I’ve been an active member of the stackoverflow community for almost 10 years now, recently I was browsing through my filtered tags and I noticed a question asking about the difference between Build and Release agents in Azure DevOps.

My answer was a fairly brief, high level, summary of how agents work in Azure DevOps but as I was writing the answer, it occurred to me that the topic of agents and how they are organised and managed is quite nuanced so I thought I’d put something more comprehensive together.

Azure DevOps pipelines are a great way to create automated builds and application deployment pipelines. Automated build and deployment jobs need somewhere to run and this is where the agent comes in to play.

So what is the difference between a Build agent and a Deployment agent? Well on a technical level they are the same thing. Microsoft provide agent software that runs on Windows, Linux and MacOS. Microsoft themselves host agents in Azure to allow you to quickly get up and running, or you can host the agents on your own infrastructure. Self hosted agents are where things get interesting.

There are 3 usage models for agents in Azure DevOps. When you install the agent software on a server it needs to be configured to be part of one of the following groupings.

  • Agent Pools
  • Deployment Groups
  • Environments

Agent Pools

As the name suggests Agent Pools are way of grouping agents together. This is a logical grouping and there are a few ways do decide how to group agents together. Ideally you want to make your agents as generic as possible so that they can be fully utilised. Whilst the agent software is free, there are licencing restrictions on the number of jobs that can run in parallel, not to mention the hosting costs of running many agent VM’s. So theoretically you can just have one Agent Pool full of agents that can do all the things. However, there might be other considerations that mean you want to create pools for different purposes.

Agent Pool with 4 windows agents
  • Operating System: This might make sense if you have some jobs that can only run on Linux, Windows or MacOS.
  • Capability: Pools could be separated by what types of jobs they can run, you might want to have dedicated agents for running tests, or separate out your Java and .net builds.
  • Function: You may want to have dedicated pools for builds vs tests vs deployments.
  • Network Access: You may want to limit the number of agents that have access to particular network resources, or agents in particular network segments.
  • Team or Product: Giving a dedicated build capacity to a particular team or product group.
  • All of the above. A mix and match approach might be appropriate depending on your circumstances. Be aware though that the more segmented your approach to pools is, the greater likelihood that some agents will be siting idle whilst others are over subscribed.

Build vs Deploy

Once you have one or more agents in a pool then you can use them to build and deploy software. Builds should be pretty straight forward, for example if you want to build a Java application then install Java and Maven on your agent. Now you can build, unit test and package your application using the agent.

Deployments are a bit more complex. If you use an agent in a pool to deploy software then your deployment scripts need to be able to run remotely against your target servers, or you need to copy your scripts to the target machine and execute your scripts over SSH or WinRM. You could use some additional tools such as Ansible to run tasks remotely as well. Whist this is doable it adds complexity to your scripts and means you have to have inbound traffic to ports open to allow the agent to connect. This leads us neatly on to the 2 other usage models for agents.

Which of the other 2 models you use will depend on how you’ve created your pipeline. Azure DevOps allows you to create pipelines in 2 different ways: Releases (also called Classic mode) are a GUI driven way of creating a build and release pipeline. Multi Stage Pipelines are a way of building pipelines as code, and are expressed as YAML. If you’ve created a “Classic” release pipeline then Deployment Groups are the model to use, if you’ve created a “YAML” pipeline then Environments are the appropriate model.

Deployment Groups

If you’ve built your pipeline in classic mode then you can use Deployment Groups to create collections of servers. Typically these will be servers that make up an environment. When you create a deployment group you can add targets, the Azure DevOps UI will give you a script to run on the target server which will install and configure the agent. You can add tags to the target at installation time or after in the UI. Tags allow you to differentiate between server types or roles such as web, app, database. This is useful when you configure your jobs in the pipeline so certain jobs will run against certain targets.

A Deployment Group for a Development environment with targets and tags

Environments

Environments are very similar to Deployment Groups, but they are only supported in YAML pipelines. When you create an environment you can add resources such as Virtual Machines or Kubernetes clusters and then control the deployment of software to those resources. As with Deployment Groups you can tag resources to refine your deployments based on type or role

An Environment for a Development environment with resources and tags

There are 3 main advantages to using Deployment Groups and Environments

  1. The agent is local to the machine its deploying to, meaning your release process doesn’t have to cater for working over a network.
  2. Security, the agent talks to Azure DevOps over port 443 which means that you can have pretty strong rules on inbound traffic to the server, as only outbound traffic over 443 is required for the agent to work.
  3. Cost, Agents in pools are subject to rules around parallel jobs. Deployment Groups and Environments* are not part of this limit

Azure DevOps agents provide a flexible way to build and deploy your software. Whilst the agents themselves all use the same code they can be used in 3 modes. Agent Pools are the generic work horses of your build and deployment pipelines. Deployment Groups and Environments allow you to use agents for specific deployment targets.

*Environments and YAML pipelines are currently in preview and the pricing model may change when they go in to general availability

One thought on “Azure DevOps agent architecture

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: