The Data Factory Module: Terraform Applies, the Pipeline Finishes

Most modules in this catalog end where Terraform ends: apply succeeds, the resource exists, done. The Data Factory template is the exception, and the reason it earns a full article. When its apply finishes, the factory is standing — but it is not compliant. Public network access is still on and encryption is still Microsoft-managed, because the azurerm provider version in play simply had no arguments for either. So the pipeline picks up a wrench: an Azure CLI step locates the factory, a PowerShell step acquires a token and PATCHes the ARM resource directly. It's the canonical specimen of a pattern every mature estate eventually needs — closing a provider gap with a post-apply REST call — and this article is the specifics.

This is Part 7 of the Terraform Module Catalog, the companion to the main Terraform on Azure series; the shared stack chassis — thin root, workspace-keyed locals, pinned module tags — is that series' Part 1 and won't be repeated here.

The declarative 95 percent

Before the wrench, the HCL. The wrapper's surface is Data Factory plus its heaviest optional organ: an SSIS integration runtime, for estates lifting existing SSIS packages into the cloud rather than rewriting them as native pipelines. The runtime is described the way you'd describe a small VM fleet, because that is what it is:

node_size_list = {
  dev     = "Standard_D2_v3"
  staging = "Standard_D4_v3"
  prod    = "Standard_D8_v3"
}
number_of_nodes_list     = { dev = 1, staging = 1, prod = 2 }
edition_list             = { dev = "Standard", staging = "Standard", prod = "Enterprise" }
license_type_list        = { dev = "LicenseIncluded", staging = "LicenseIncluded", prod = "LicenseIncluded" }
executions_per_node_list = { dev = 1, staging = 1, prod = 2 }

license_type is a line item worth a code comment in any real deployment: LicenseIncluded versus BasePrice is the Azure Hybrid Benefit switch, and on an eight-core nightly-ETL runtime it is real money. Two richer inputs ride along as objects — a catalog_info block (the SSISDB catalog: server endpoint, admin login, pricing tier) and a custom_setup_script block (a blob container URI plus SAS token, for installing drivers onto runtime nodes before packages run). In the template both arrive as objects whose attributes are commented out, a documented menu you enable when needed.

Networking follows the library's standard posture with one addition: alongside the usual private-link subnet, the platform JSON carries a subnet fact dedicated to Data Factory — the delegated subnet its managed VNet integration runtime joins, so that pipeline activities reach data stores over private addresses. App stacks never learn the address space; they look up a name, as always.

The imperative 5 percent

Now the pipeline. After the standard install → init → workspace → validate → apply spine, two steps run only when the operation was apply. First, an Azure CLI step finds the factory and exports credentials:

- task: AzureCLI@2
  displayName: Locate factory, export SP
  condition: and(succeeded(), eq('${{ parameters.terraformOperation }}', 'apply'))
  inputs:
    azureSubscription: $(pvar-pipeline-service-connection)
    scriptType: pscore
    addSpnToEnvironment: true
    inlineScript: |
      $name = az datafactory factory list --resource-group $(pvar-adf-resource-group) `
        --query '[-1].name' -o tsv
      Write-Host "##vso[task.setvariable variable=adf_name]$name"
      Write-Host "##vso[task.setvariable variable=ARM_CLIENT_ID]$($env:servicePrincipalId)"
      Write-Host "##vso[task.setvariable variable=ARM_CLIENT_SECRET;issecret=true]$($env:servicePrincipalKey)"
      Write-Host "##vso[task.setvariable variable=ARM_TENANT_ID]$($env:tenantId)"

Then PowerShell acquires an AAD token with those credentials and PATCHes the factory's ARM resource:

$body = "grant_type=client_credentials&client_id=$clientId" +
        "&client_secret=$clientSecret&resource=https://management.core.windows.net/"
$token = Invoke-RestMethod -Method Post `
  -Uri "https://login.microsoftonline.com/$tenantId/oauth2/token" `
  -Body $body -ContentType 'application/x-www-form-urlencoded'

$patch = @{ properties = @{
  encryption = @{
    VaultBaseUrl = "$(pvar-cmk-key-vault-uri)"   # e.g. https://kv-platform-dev.vault.azure.net
    KeyName      = "$(pvar-cmk-key-name)"
    KeyVersion   = "$(pvar-cmk-key-version)"
  }
  publicNetworkAccess = "Disabled"
} } | ConvertTo-Json -Depth 3

Invoke-RestMethod -Method Patch -Headers @{
    Authorization  = "$($token.token_type) $($token.access_token)"
    'Content-Type' = 'application/json'
  } `
  -Uri "https://management.azure.com/subscriptions/$sub/resourceGroups/$rg/providers/Microsoft.DataFactory/factories/$($env:adf_name)?api-version=2018-06-01" `
  -Body $patch

Same identity Terraform just used, same resource Terraform just created, two properties Terraform could not reach. Customer-managed-key encryption and publicNetworkAccess = Disabled were governance requirements, not preferences; the alternative to this PATCH was “don't deploy Data Factory”. I wrote about why these escape hatches are structural rather than shameful in the main series (Part 9, Terraform's Escape Hatches Are Load-Bearing); this is the artifact that earned the pattern its section.

Where the wrench slips

I admire this pipeline and I would still change four things about the version I found in an estate I worked on.

Discovery by list-and-take-last. The real script located “the factory Terraform just made” by listing every factory in the resource group and indexing the last element. With one factory per group it works forever; the day a second factory appears, the CMK PATCH silently lands on the wrong one — and both keep applying green. Terraform knows the name it created; emit it as an output, read the output, never guess.

Secrets leaving the vault of the task. addSpnToEnvironment hands the service-principal secret to the script, which re-exported it as a pipeline variable — in the original, without marking it secret, so it was eligible for expansion in later logs. If a credential must cross task boundaries, issecret=true is the minimum toll (the version above pays it); better is doing the token dance inside the same task and exporting nothing.

-lock=false, everywhere. Both init and apply disabled state locking — almost certainly a leftover from debugging a stuck blob lease. On a manually triggered pipeline collisions are rare, which is precisely why the one that eventually happens will be a mystery. Locks are cheap; archaeology isn't.

A pinned key version. The pvars carry vault URI, key name — and key version. Pinning the version means key rotation requires a pipeline-variable edit and a redeploy per environment; pointing at the versionless key lets the platform's rotation policy work. Whether ADF's API accepted that at the time is a fair question. Whether anyone would remember to re-run three pipelines after rotating a key is not.

And the catalog's recurring defect makes its appearance: custom_setup_script was a list holding one object in dev, but a bare map in staging and prod — the same shape-mismatch-across-environments trap as Part 6's empty tuples, dormant until the first non-dev plan. For good measure, the template's README still linked to the Cosmos DB sample it was copied from. Templates are copied far more often than they are read; their smallest mistakes have the largest fan-out.

The lesson

The Data Factory module teaches the catalog's most transferable move: when the provider lags your governance requirements, the resource's API is still there. Declare everything the provider can express; PATCH the remainder in the same pipeline run, under the same identity, gated on the same success condition — so no human ever sees the resource in its non-compliant intermediate state. Today the azapi provider would bring even the PATCH inside the graph, with drift detection included; the shape of the solution doesn't change, only how much of it Terraform gets to see.

Next in the catalog, the counterweight to all this machinery: the smallest module in the library, and what its hundred lines say every module must carry — Part 8, Redis.