In a typical development workflow, there's often a need to maintain multiple environments such as development, staging, and production. Each of these environments might have its own set of configuration values: API endpoints, database connection strings, third-party secrets, and more.
Hardcoding these values or keeping them inside source code is a security risk and makes managing configurations complex. Pulumi ESC (Environments, Secrets and Configuration) offers a centralized store to manage configuration data, plain-text data, and secrets.
In this tutorial, we’ll demonstrate the power of Pulumi ESC in managing configuration across multiple environments.
## Prerequisites
You will need the following tools to complete this tutorial:
Pulumi ESC is a service of Pulumi Cloud that can be used with or without Pulumi IaC. This means that if you already have the Pulumi IaC CLI installed, you do not need to install the Pulumi ESC CLI, and you may substitute `pulumi env` anywhere you see the `esc env` command in this guide.
{{</notes>}}
- An [Amazon Web Services](https://aws.amazon.com/) account
Imagine you're building a system that integrates with a third-party service such as:
- a payment provider
- weather data provider
- a content-management system
In the development environment, you might be integrating with the sandbox or development endpoint of the third party service, while in the testing environment, you might be integrating with the production endpoint. Each of these third party services could potentially have different API endpoints and API keys, and having this separation of concerns enables developers to be able to interact with their environments without affecting real users, data, or API limits.
In the next steps, you will deploy an example application that will simulate a third party service with both a dev and test endpoint.
### Deploy the Application
Start by cloning the [sample application code](https://github.com/pulumi/tutorials/tree/pulumi-esc-get-started) locally to your machine.
The endpoint URLs as well as the names of the Secrets Manager secrets have been outputted for easy reference. You can also see these values in the `Outputs` tab of your CloudFormation stack in the AWS Console.
#### Verify Application Endpoints
Now that the sample application is deployed, you can test that the `dev` and `test` endpoints are working with a simple curl command. Copy and paste the following command into your terminal, replacing the placeholder text with the values relevant to your own application endpoint:
This is a simple way to quickly test that your endpoints are working for a single application, but when it comes to using and managing endpoints and API keys for real-world workloads at scale, this practice is not sustainable or secure.
Many applications require access to configuration values or secrets, which often means those values need to be directly embedded or read from local environment variables or files. If you have multiple applications that use the same configuration details, and each application is referencing its own local environment files or variables, this can lead to a very error-prone process when it comes to updating or removing configuration values.
With Pulumi ESC, you can centralize these values as a single source of truth because you will only need to make updates in one place instead of multiple places. Further, you can grant your applications permissions to directly fetch these values at runtime, ensuring that:
- applications always have the latest values without manual intervention
- sensitive values are not unintentionally exposed
In an environment file, values are defined as a series of key-value pairs in YAML format. All variables will be defined under a top-level key named `values`. These values can be strings, numbers, or arrays, and they can be manually provided, dynamically generated from external sources, or referenced from other values in the file.
You will be directed to the Environments landing page. To create a new environment, click the `Create environment` button, and then enter a name for your environment (e.g., `app-env-global` for the shared configuration environment).
Repeat the same steps to create the environments for `app-env-dev` and `app-env-test`. Then, click on the name of the `app-env-global` environment to open its definition editor.
In the editor, you will be presented with a split pane view. The left side is where you will write the definition of your environment configuration, and the right side will show a preview of your configuration in JSON format.
The configuration that you will be adding to the `app-env-global` file is the base URL of your application endpoint. Because this URL will be the same across environments, you only need to define it in one place, and you can reference it from other environments accordingly. You will learn more about how to do this using `imports` later in this tutorial.
Using the value of the `ApplicationEndpointUrl` from the CloudFormation output in the previous step, add the following details to the `app-env-global` environment file, making sure to replace the placeholder text with the actual value of your application endpoint URL:
Your `app-env-test` environment file should have a value of "test" for the `ENVIRONMENT` parameter, and the value of `API_KEY` should be the auto-generated value of the `escTestApiKey` secret from Secrets Manager.
Now that you have populated your environment files, you can verify that your values have been successfully stored by retrieving them through the Pulumi ESC CLI. Start by running the following command to log into the CLI:
The CLI has a built-in `get` command that enables you to retrieve a single value from your environment. The format of the full command looks like the following:
esc env get <your-org>/<your-environment-name><variable-key-name>
```
Let's assume that your Pulumi organization is named `acme` and the environment that you want to retrieve values from is named `app-env-dev`. If you want to retrieve the value of the `API_KEY` variable, the command to do so would look like the following:
There may be scenarios in which the value you need to retrieve lives in a different environment file. For example, since your base application endpoint URL will be the same across environments, it would be more efficient to define it once in one place rather than multiple times across multiple environment files.
With Pulumi ESC, you have the ability to import other environments into your environment file and make use of the imported configuration values and secrets. Similar to `values`, `imports` is a top level key you will need to define in the environment file, meaning the syntax to create an import looks like the following:
When you save the file, the environment preview will update to show that `ENDPOINT_URL` is now included in the list of accessible variables in this environment.
For the purposes of this tutorial, you've manually added the values of your API keys to your `dev` and `test` environment files. However, this is not the recommended best practice when it comes to the storing, management, and retrieval of sensitive values.
With Pulumi ESC, you can dynamically retrieve those values directly from a provider. In this next section, you will learn how to configure your environment file to retrieve secret values from AWS Secrets Manager.
- replace the placeholder text for the `roleArn` parameter with the ARN of your OIDC role
- provide a value for `duration` no greater than the [maximum session duration](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use.html) of your OIDC role
- provide the name of the AWS region that you deployed your sample application into
As shown with the `${aws.login}` and `${aws.secrets.api-key}` references above, values within Pulumi ESC environments, as well as its imports, can be referenced through the use of interpolations. Below is an example of what this would look like in the console:
You can verify your provider connection and the retrieval of your secret from AWS Secrets Manager by running the `esc open acme/app-env-dev API_KEY` command.
{{<notes>}}
To see examples for how to accomplish this with other providers, take a look at the [syntax reference documentation for Pulumi ESC](/docs/pulumi-cloud/esc/reference).
{{</notes>}}
### Exposing Values as Environment Variables
The Pulumi ESC CLI also has a `run` command that enables you to run other commands using environment variables (for example the `aws s3 ls` command) without having to locally set the environment variables first. This is great for when you have other components or applications that may need to interact with your endpoints as part of a larger application workflow.
For example, if you have a CI/CD pipeline that will automatically push blog post updates to a content-management system, you can enable it to retrieve the endpoint and API key specific to its environment and run the command to update the post.
However, values in your environment file are not exposed as environment variables by default. You can expose them by adding your key-value pairs under a second-level key labeled `environmentVariables`:
Following the above format, add the `environmentVariables` key to your `app-env-dev` and `app-env-test` environment files. Then, move your `ENDPOINT_URL`, `ENVIRONMENT` and `API_KEY` variables so that they are nested underneath it before saving your file.
As seen above, the `ENDPOINT_URL`, `ENVIRONMENT`, and `API_KEY` values that you defined in your environment file are referenced directly in the command as environment variables, without needing to explicitly set them locally and without needing to use the `esc env get` command to retrieve their values inline.
If you run the command with `app-env-test`, it will state `You have reached the simulated TEST endpoint.` instead.
## Integrating with Pulumi IaC
{{<notes>}}
This section of the tutorial is optional and requires the installation of the [Pulumi IaC CLI](/docs/cli/) and one of [Pulumi's supported language runtimes](/docs/languages-sdks/) as a prerequisite.
{{</notes>}}
Pulumi ESC works independently of [Pulumi Infrastructure as Code (IaC)](/docs/get-started/), providing developers the flexibility to centrally manage their environment configuration regardless of how or where those environments are created.
Pulumi ESC also integrates seamlessly with Pulumi IaC, and this next section will demonstrate how to leverage Pulumi ESC in your own Pulumi programs. This works everywhere, including Pulumi Deployments and GitHub Actions, without any additional work or changes.
To start, [create a new Pulumi AWS project](/docs/clouds/aws/get-started/create-project/) and [ensure it is configured to use your AWS account](/registry/packages/aws/installation-configuration/).
During the creation of your project, you will typically be prompted for some configuration values for the stack.
```bash
$ pulumi new aws-python
This command will walk you through creating a new Pulumi project.
Enter a value or leave blank to accept the (default), and press <ENTER>.
Press ^C at any time to quit.
project name (pulumi-python):
project description (A minimal AWS Python Pulumi program):
Created project 'pulumi-python'
Please enter your desired stack name.
To create a stack in an organization, use the format <org-name>/<stack-name> (e.g. `acmecorp/dev`).
stack name (dev): zephyr/esc-dev
Created stack 'zephyr/esc-dev'
aws:region: The AWS region to deploy into (us-east-1): us-east-2
Saved config
Installing dependencies...
Finished installing dependencies
Your new project is ready to go!
To perform an initial deployment, run `pulumi up`
```
For AWS projects, you are prompted for the AWS region.
This value is stored in a file called `Pulumi.<your-stack-name>.yaml`.
```yaml
config:
aws:region: us-east-2
```
Defining the configuration this way may be fine when dealing with a singular project, but it can become very challenging to maintain securely and consistently across multiple projects. Centralizing these configuration values provides more scalability without increasing administrative overhead along the way.
To centralize this configuration for your Pulumi program, you will need to add a second-level key named `pulumiConfig` in your environment file that will expose the values nested underneath it to Pulumi IaC.
If you view your resource in the AWS console, you will see that your program resources will get created in the region you specified in your environment file.
You can also reference the `pulumiConfig` environment values directly from within your Pulumi program in the same way that you would access them from the project's config file.
Then run the following command from the root of the sample application folder to delete the AWS resources, replacing the placeholder text with the name of your CloudFormation stack as provided in your stack outputs:
In this tutorial, you created and retrieved values from a Pulumi environment. You also exposed and programmatically consumed values as environment variables and as Pulumi configuration.
- Learn more about [environments](/docs/concepts/environments/), [secrets](/docs/concepts/secrets/), and [configuration](/docs/concepts/config/) in the Pulumi documentation.
