Azure DevOps Workflow Identity Federation - A scripted solution

Videos

This is an older version about this where I first used the new feature:

TODO: Upcoming video-chat with Yashas himself

Summary

Some time ago Microsoft introduced a new typo of service connection available for Azure DevOps in order to authorize operations against Azure Entra ID. Formerly you had to define service principals using secrets which naturally needed to expire at some point. I already made videos about the advantages of using your own SPs instead of letting Azure DevOps create them automatically.

The new federated service principals now offer an automatic conversion which brings us back to the state where Azure DevOps creates and names the associated SPs in Entra ID (which is bad).

In this article I want to explain how to do this manually again using pre-defined Entra Service Principals and thus hold control over your tenant.

At the end I will give some insights on how the MS team screwed the new availabilities up a little bit and how you can get back to normal.

Recap

The old service connection with secret

First lets recap what we where left to before all those changes. We are talking about so called Service Connections which can be managed in Azure DevOps in the project settings. If you wanted to create a new connection to Azure you needed to create a new one and select Azure Resource Manager as the type. Because I have no way to showcase the old way completely the following screenshot shows you with what options you would create this old service connection which is fully controlled by Azrue DevOps:

Fig. 1: Creating a manual service connection with secret

Keep attention to the newly appearing warning when you select “Secret” stating that this will then needed to be re-done after 1 year and pointing you to use identity federation. Anyways, if you create such a service connection, Azure DevOps will create a corresponding service principal in Entra ID on behalf of you naming it {ADO_ORG_NAME}-{ADO_PROJECT_NAME}-{GUID}:

In Azure DevOps you are now presented with this service connection:

Fig. 3: Automatic service connection in ADO

I highlighted the new Convert option which I’m not going to use in this article.

The problem with this now is that if the obligatory secret behind the Entra SP will expire you are forced to delete and re-create this from Azure DevOps side. This is arguably bad. The arguments for this beeing mostly:

Not having control over the name leaves you with a bunch of SP names which are hurting any probable naming conventions your organization might (should) have.
The person creating the service connection must have rights to create a service principal in Azure and give it access to the needed resources which is and should be unlikely.
Fine grained access needs to be defined by hand later and this needs to be repeated if the service connection is stalled. So all of this hurts automation totally.
You have no chance of moving this over to a fully automated approach.

The old workaround: manual created SPs

So instead of leaving the SP-creation to ADO you could also create a service principal yourself in Azrue DevOps. Then you would give it the required role assignments on the required scopes, have control over its name and could later update the secret in ADO when necessary when it expires.

I’m not covering this in detail here. Look at the first linked video at the top if you are interested.

New experience

As I showed you in Fig. 1 the new ADO throws out warnings. More importantly though it offers a direct creation (instead of conversion only) of those new federated service connections.

Using the automatic created one

So lets start by selecting the default options on new Azure Resource Manager service connections:

Fig. 4: Creating federated service connection in ADO

I highlighted the important settings. If you execute this, the SP in Entra ID can be created in 2 separate ways.

You get an SP where the subject identfier looks like sc://{ADO_ORG_NAME}/{ADO_PROJECT_NAME}/{ADO_SC_NAME}.
Your subject identfier looks like /eid1/c/pub/t/{ADO_GENERATED_ID_1}/a/{ADO_GENERATED_ID_2}/sc/{SOME_GUID_1}/{SOME_GUID_2}

Breakout: Entra issues, Microsoft rollbacks and preview stuff

If you get the second one you will have trouble in automatic generation of the service connection (see below). You cannot predict those IDs and GUIDs but they seem to be important. This was introduced by MS silently and after we created a support ticket we learned the the ADO team had not been aware that people tried to use automation on this process.

My collegue Yash detected it and addressed it in VS Dev Community thread.

So what you can do is to create a support ticket and ask Microsoft to rollback your ADO Organization back to the original predictable behavior!

Looking into Entra ID I just recently found another interesting option which is currently not reflected in Azure DevOps:

There seems to be an upcoming option to define claim matches so that you probably will be able to define the matches yourself later. Because there is no corresponding setting on Azure DevOps side I cannot tell you if this is going to be useful there.

Breakout ends: Back to our track

So anyways. From this moment on I assume that you get those to settings on ADO service connections:

The issuer will be created in the pattern https://vstoken.dev.azure.com/{ADO_ORG_ID}. The ADO_ORG_ID could be obtained using MS Graph against your Azure DevOps graph endpoints.

All of this still leaves you with an automatically created and named service principal in Entra:

I highlighted the created name and the GUID part which just is self-referencing in the name. So again we are back to this bad situation which I explained above.

Creating federated service connections manually using graph and PowerShell

We could not find a way currently to directly create federated service connections But it is possible to first create a classic service connection with secret and the convert it into a federated one using MS Graph.

So what we will do now is:

Create a service principal in Entra and give it the rights it needs.
Use PowerShell and MS Graph to
1. Create a classic service connection
2. Convert it into a federated one.

So lets start!

Talking to Azure DevOps with PowerShell

You can skip this section to The PowerShell script to use if you want to have a look at my final function based approach.

So first it is important to learn something about the Graph API when it comes to ADO. We will start by retrieving service connections from ADO using the hints from the docs.

There are actually 2 endpoints you can talk to:

https://dev.azure.com with token url and
https://app.vssps.visualstudio.com

Lets define some variables first:

$tenantId = ''
$orgName = ''
$projectName = ''

So lets retrieve an access token to talk to Graph. The token URL for this is a constant GUID which tells Entra to give you a token for talking to ADO:

$token = Get-AzAccessToken -ResourceUrl 499b84ac-1321-427f-aa17-267ca6975798 -TenantId $tenantId
$accessToken = ConvertFrom-SecureString -SecureString $token.Token -AsPlainText

Now use this to create a header for our request to ADO Graph:

$headers = @{
    Accept        = "application/json"
    Authorization = "Bearer $accessToken"
}
$uri = "https://dev.azure.com/$orgName/$projectName/_apis/serviceendpoint/endpoints?api-version=7.1"
Invoke-RestMethod -Method GET -Uri $uri -Headers $headers

If you execute all of the above in from a single script it should spit out the list of current service connections which are present in the given ADO-project:

This is good news because it means we possibly also could send POST requests to create some stuff.

Retrieving the ADO project id

We can also retrieve a list of all projects in the current organization by leaving out the project name from the URI and ask the projects endpoint using the same credentials:

$uri = "https://dev.azure.com/$orgName/_apis/projects?api-version=7.1"
Invoke-RestMethod -Method GET -Uri $uri -Headers $headers

At this time keep track of id property of the project you want to create a service connection in.

Create a classic service connection

To create a new service connection you have to send a POST request to the same Graph endpoint using the following JSON body:

$body = @"
{
  "name": "$serviceConnectionName",
  "data": {
    "creationMode": "Manual",
    "environment": "AzureCloud",
    "scopeLevel": "Subscription",
    "subscriptionName": "$subscriptionName",
    "subscriptionId": "$subscriptionId"
  },
  "type": "AzureRM",
  "serviceEndpointProjectReferences": [
    {
        "name": "$serviceConnectionName",
        "projectReference": {
            "id": "$projectId",
            "name": "$projectName"
        }
    }
  ],
  "url": "https://management.azure.com/",
  "authorization": {
    "parameters": {
      "servicePrincipalKey": "$clientSecret",
      "tenantId": "$tenantId",
      "servicePrincipalId": "$clientId",
      "authenticationType": "spnKey"
    },
    "scheme": "ServicePrincipal"
  },
  "isReady": true
}
"@

Of cause you need to replace the variables. In this sample I assume that you want to a service connection which can deploy in a subscription. Here is an explanation for them:

serviceConnectionName: The display name you want the service connection to be in Azure DevOps.
subscriptionName: Display name of your subscription where the service connection should deploy to.
subscriptionId: The unique GUID of the Azure subscription.
projectId: The GUID of your ADO project (see the step before).
projectName: The display name of your project as it appears in ADO.
clientId: The client id of the service principal.
clientSecret: The current secret of the service principal.

By simply passing this as the body the endpoint with

$uri = "https://dev.azure.com/$orgName/$projectName/_apis/serviceendpoint/endpoints?api-version=7.1"
$response = Invoke-RestMethod -Method POST `
    -Uri $uri `
    -Headers $headers `
    -Body $Body `
    -ContentType 'application/json'
$endpointId = $response.id

should do the job. Afterwards you should check if there is a corresponding service connection in your ADO project settings.

Converting to a federated service connection

Now in order to convert it to the desired state use the following code:

$uri = "https://dev.azure.com/$orgName/$projectName/_apis/serviceendpoint/endpoints/$endpointId?api-version=7.1&operation=ConvertAuthenticationScheme"
$body.authorization.scheme = "WorkloadIdentityFederation"
$body.data.PSObject.Properties.Remove('revertSchemeDeadline')
$body.authorization.parameters.PSObject.Properties.Remove('authenticationType')
Invoke-RestMethod -Method PUT `
    -Uri $uri `
    -Headers $headers `
    -Body $body `
    -ContentType 'application/json'

So basically we are replacing values from the previous step and add an URL parameter to the request. This seems a little odd because why not create the connection this way in the first place, right? At least for us this wasn’t working but the conversion did work. Probably this will be fixed eventually and then getting obsolete.

The PowerShell script to use

So bringing all the former stuff together I decided to make up a script which should be easily re-usable. It can be found in this Gist. This brings functions that you can simply call. Those function are a little bit more complicated than shown before but you should get it.

Disclaimer: Do not use this code in production! It needs severe improvements and represents a cut-down version of our own scripts. It for instance stores secrets in clear-text in local memory which is nothing what you should do!

First thing to do is to create a new service principal with appropriate permissions in Entra ID. If you want to create one quickly here is the code to do it with PowerShell:

$sp = New-AzADServicePrincipal -DisplayName demo
Write-Host "Name:           $($sp.DisplayName)"
Write-Host "ClientId:       $($sp.AppId)"
Write-Host "ClientSecret:   $($sp.PasswordCredentials.SecretText)"

You will need these values later!

If you need to remove this SP after your test (which you should) use Remove-AzADServicePrincipal -ApplicationId $sp.AppId!

So lets start with the download:

mkdir sample
cd sample
curl https://gist.githubusercontent.com/codingfreak/9db97a4db83a718b72c2e8a4f233a268/raw/ce6ff723bf026ed557d605533f95e361f6b47e98/sample.ps1 -o sample.ps1
code .

Now in VS code (or whatever editor you prefer) head down to the bottom and find the TODO-section. Fill in the required values:

# ...
# FILL IN THOSE VALUES!
$orgName = 'TODO'
$projectName = 'TODO'
$serviceConnectionName = 'TODO'
$spId = 'TODO'
$spName = 'TODO'
$spSecret = 'TODO'
#...

Than back in your terminal simply execute

./sample.ps1

That should do the job and both the service connection and Entra service principal should be good.

Ensure to use Connect-AzAccount correctly before running this stuff!

Sometimes it can happen that the timeout in the script (10s) is not enough. For brevity i left out sophisticated retries here.

Afterwards you can check your results in Azure DevOps and Entra ID. In Azure DevOps in the service connections of your project you should see the one you defined in $serviceConnectionName. In Entra ID your should go the the SP your chose under $spName and look into its Certificate & secrets section. Check the Federated credentials tab. You should see a credential corresponding to your ADO project. Thats it.

Final thoughts

So now we are finally able to create password-less credentials which allow Azure DevOps to pair up with existing service principals which we can create beforehand. This is more like a “normal” process in enterprises would work. One guy is responsible for Entra ID creating and authorizing the service principals there. Another person can create the service connection and pair it with an existing SP without needing to be allowed to tinker around in Entra. I think this is great.

codingfreaks

Experiencing Microsoft