Terraform's Escape Hatches Are Load-Bearing

Every Terraform tutorial implies a world where the provider has a resource for everything you need. Every production estate I have touched contains the other world: the places where azurerm simply cannot do the thing, and someone reached for local-exec, an external data source, or a raw REST call to bridge the gap.

The polite name for these is escape hatches, which suggests something you use once, in an emergency. In the estate this series describes, three of the nine glue modules from Part 5 and Part 6 — including the two most-consumed modules in the entire platform — were built around escape hatches. They were not emergencies. They were load-bearing structure, executed on every apply of dozens of stacks for years. This part catalogues the three patterns, then pays the bills they ran up.

Pattern 1: local-exec because there is no resource

Azure Functions have host keys and function keys, and for years azurerm had no resource to set one. But the platform's whole APIM-fronting design required Terraform to control the key — mint it, install it, share it with the gateway. The workaround:

resource "random_password" "function_key" {
  length  = 32
  special = false
}

resource "null_resource" "install_key" {
  triggers = {
    always_run = timestamp()   # remember this line
  }
  provisioner "local-exec" {
    interpreter = ["pwsh", "-Command"]
    command     = <<-EOT
      az functionapp keys set `
        --name ${var.function_app_name} `
        --resource-group ${var.resource_group_name} `
        --key-type functionKeys --key-name apim-access `
        --key-value '${random_password.function_key.result}'
    EOT
  }
}

Terraform generates the secret (so it lives in state and can flow to the APIM named value and the Key Vault record), then a null_resource shells out to az to install it. Declarative intent, imperative enforcement. Roughly ninety stacks depended on this seam.

Pattern 2: external data source because there is no data source

The inverse problem: a value that only the platform can tell you, with no data source to ask. A Logic App workflow's trigger URL is SAS-signed and only retrievable via a management-plane call. The estate wrapped it:

data "external" "callback_url" {
  program = ["pwsh", "-File", "${path.module}/get-callback-url.ps1"]
  query = {
    logic_app_id  = var.logic_app_id
    workflow_name = var.workflow_name
  }
}

# get-callback-url.ps1 - contract: JSON in on stdin, flat JSON out on stdout
$query = [Console]::In.ReadToEnd() | ConvertFrom-Json
$raw = az rest --method post `
  --uri "$($query.logic_app_id)/hostruntime/runtime/webhooks/workflow/api/management/workflows/$($query.workflow_name)/triggers/manual/listCallbackUrl?api-version=2022-03-01"
$url = ($raw | ConvertFrom-Json).value
$uri = [System.Uri]$url
@{ base_url = "https://$($uri.Host)$($uri.AbsolutePath)"; sas_query = $uri.Query.TrimStart('?') } | ConvertTo-Json -Compress

The script splits the URL into a base path (which becomes an APIM backend) and the SAS query (which becomes a secret named value) — string surgery on a credential, performed at plan time, on every plan. It works. It is also a data source that talks to the network during plan, which means plans now require connectivity, permissions and a healthy management plane just to render a diff.

Pattern 3: the post-apply PATCH

The most instructive specimen lives in the Data Factory stack's pipeline, not its HCL. The provider version in use could not set customer-managed-key encryption or publicNetworkAccess = Disabled on a factory. Governance required both. So the pipeline finishes the job after Terraform:

- script: terraform apply -auto-approve   # ...the declarative 95%
- task: AzureCLI@2
  displayName: Post-apply hardening      # ...the imperative 5%
  inputs:
    scriptType: pscore
    inlineScript: |
      $token = az account get-access-token --query accessToken -o tsv
      $body = @{ properties = @{
        publicNetworkAccess = "Disabled"
        encryption = @{ keyName = "$(cmkKeyName)"; vaultBaseUrl = "$(kvUri)" }
      } } | ConvertTo-Json -Depth 5
      Invoke-RestMethod -Method Patch `
        -Uri "https://management.azure.com$(factoryResourceId)?api-version=2018-06-01" `
        -Headers @{ Authorization = "Bearer $token" } -Body $body `
        -ContentType 'application/json'

Acquire a token, PATCH the ARM resource directly, done. The pattern generalizes: when the provider lags the platform, the resource's API is still there. Today you would reach for the azapi provider and keep even this inside the graph — but the shape of the solution (declare what you can, PATCH what you must, in the same pipeline) is evergreen.

Now the bills

Escape hatches are not free, and this estate paid every one of the standard invoices.

Idempotency dies first. Look back at triggers = { always_run = timestamp() }. Its purpose was to make sure the key gets installed even after out-of-band changes. Its effect is that every apply of ninety-odd stacks rotated a production function key — apply a tag change, rotate a credential. The team half-embraced it ("free rotation!"), and it is true that regularly rotated keys are better than immortal ones. But rotation-as-side-effect means rotation is unscheduled, unlogged as such, and simultaneous with unrelated changes — when a consumer breaks, was it your infra change or the rotation that rode along? A plan that is never clean is a plan nobody reads carefully. This single line did more damage to plan-review culture than anything else in the estate.

Portability dies second. Three modules shelling out to PowerShell and az means every agent needs PowerShell, the CLI, the right versions, and a logged-in identity — which is a big part of why the whole estate ran on self-hosted Windows pools (Part 4 covered the identity half). The hosted-agent option, and with it most of the ecosystem's assumptions, was foreclosed by three small scripts. Escape hatches constrain your fleet, not just your module.

Visibility dies third. terraform plan renders a beautiful diff of declarative changes and says nothing about what the scripts will do — null_resource diffs are just “will run”. Secrets flow through state (the random_password), through command lines (visible in process listings on the agent), and through script output. The blast radius of a debug set -x or an over-chatty script is a credential in a build log.

Rules I now carry

Having run on load-bearing escape hatches, I do not conclude “never use them” — I conclude they need engineering standards stricter than ordinary resources, because Terraform is not checking your work:

1. Exhaust the declarative options first — a newer provider version, the azapi provider (which gets you CRUD semantics, diffs and drift detection even for unsupported properties). Half this estate's hatches would be plain resources at today's provider versions; schedule the migration when the provider catches up, or the workaround outlives its reason.
2. Make triggers honest. Trigger on the inputs that should cause a re-run — the key name, the function app id — never on timestamp(). If you need rotation, build rotation: a scheduled pipeline, an expiry-tagged secret, anything with its own audit trail.
3. Scripts get the same rigor as resources: a defined JSON contract, $ErrorActionPreference = 'Stop', no secrets in echoed commands, and a comment stating which provider gap this fills and how to check whether it still exists.
4. Declare the fleet dependency. If a module needs PowerShell + az, its README says so; better, a pipeline step asserts it before Terraform runs.

The escape hatches carried the platform's most dangerous cargo, and mostly carried it well. Next, the series visits the place where the cargo was heaviest: the estate's richest single artifact — Part 10, SQL as a Governed Blueprint. And for the per-module detail behind every pattern in this series, the Terraform Module Catalog companion series walks the template library one module at a time.