As Macro has grown, so has the number of services we run to deliver the product experience our users expect. One recurring headache has been keeping each service’s configuration requirements synchronized with the environment variables provided to its container at runtime.
Environment Desync
A very common pattern (and one that we originally adopted) is to use infrastructure as code to orchestrate all the environment variable values required by our services. We originally followed this pattern using Pulumi and TypeScript. It became quite easy to add a new required config value to our Rust service and forget to update the Pulumi infra to provide this. A typical flow would be: we would deploy to development, learn that the deployment had failed, and dig through the logs to identify the missing environment variable. The same failure could then occur in production if the corresponding configuration change was not applied there.