This article is the second part of a series on best practices for securely managing AWS credentials on CI/CD. In this article, we go in-depth on providing AWS credentials securely to a 3rd party and introduce a Pulumi program to automate rotating access keys.
In the [first post](/blog/managing-aws-credentials-on-cicd-part-1/) in our series, we created a dedicated IAM User to perform updates to AWS resources within your CI/CD system. The next step is to pass the AWS access keys for that user to your CI/CD system.
We need to take great caution. [AWS's documentation](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html) states, **"Do not provide your access keys to a third party"**.
Providing AWS access keys to any system is dangerous because of the risk that a bad actor could obtain those keys and do something nefarious. Even if your CI/CD service takes great care to protect your secrets, those AWS credentials could be inadvertently exposed in debugging output, system logs, or in some other way.
This is one of the main advantages of performing continuous deployment from within your AWS account. For example, running Jenkins on an EC2 instance you manage, or using the AWS CodeDeploy service. Because when performing a deployment from within an AWS compute environment, you can use built-in mechanisms to obtain credentials securely.
However, there are many reasons to use a hosted service for performing your CI/CD, such as developer productivity, ease of use, performance, or simply because you don't want to recreate your existing deployment workflows.
If you choose to provide your CI/CD with credentials, the most important thing is to mark them as secret. Your CI/CD provider typically has built-in support for handling "secure variables" or other sensitive information. As opposed to general, configuration data or environment variables.
User-supplied configuration values are typically write-only and only accessible by the CI/CD worker jobs at runtime. For example, Travis CI has [encrypted environment variables](https://circleci.com/docs/2.0/env-vars/#overview) or [secure environment variables](https://circleci.com/docs/2.0/env-vars/#overview) in CircleCI.
> Rather than giving your CI/CD provider AWS credentials, why not have your CI/CD system obtain credentials from a specialized "secrets manager" service?
In other words, if you choose _not_ to trust your CI/CD system with this data, can you _instead_ trust some other system dedicated for securely storing and retrieving sensitive information?
Abstractly, the difference here is that the credentials are provided on-demand, rather than being available to the CI/CD job when it starts (and stored via the CI/CD provider). Instead, your CI/CD job would obtain credentials from the "secrets manager" only when needed.
There are some advantages to this approach, such as providing a clear audit trail for access and more control over the distribution of sensitive information.
However, the secrets provider system needs to be presented with some form of credentials. And _those_ credentials need to be available to your CI/CD environment. So using a secrets manager leaves you in the same place you started, i.e., needing to provide sensitive data to your CI/CD provider.
Also, by adding a dependency on a secrets manager, you introduce additional risks. Not only do you need to be even more security-conscious about that secrets manager, but it also needs to be highly available. Any outage for that service would mean that you would be unable to perform deployments!
So it does not seem that using a secrets manager to dole out AWS credentials to your CI/CD system is a good practice to follow. Or, at the very least, makes some tradeoffs without fundamentally making your approach to CI/CD any more or less secure.
When providing credentials to a 3rd party, rather than hoping it will be 100% secure forever (which is impossible), we can make those credentials _volatile_. If we regularly invalidate and rotate the credentials supplied to the CI/CD system, we can dramatically reduce the impact of any accidental disclosure. Even if the IAM User's credentials were leaked, by the time they were discovered and used, they would no longer be valid.
The AWS security blog describes how to [rotate access keys for IAM Users](https://aws.amazon.com/blogs/security/how-to-rotate-access-keys-for-iam-users/), by using the following steps:
That all sounds simple enough, but it's certainly tedious. And if you need to repeat the process across dozens, if not hundreds of different CI/CD pipelines, you need a better solution. Fortunately, Pulumi provides an excellent and extensible way for writing a serverless program to
The next few sections describe a simple infrastructure application for automating AWS IAM credential rotation. You can see the full application on GitHub at
It creates an AWS lambda function that's triggered on a regular schedule, e.g., every 30 minutes, and performs the next step in the sequence for rotating access keys as outlined above.
First, it creates a new access key and pushes the new value out. On the next iteration, it marks the older access key as "inactive." On the following iteration, the inactive key is deleted. The process repeats, generating a new key and removing the inactive key.
The heart of the application is triggering it to execute on a fixed interval. Thankfully this is super-easy to do using Pulumi since it allows you to seamlessly blend your "cloud infrastructure" with "code" in a natural way. The user guide for Pulumi Crosswalk for AWS has more information on [serverless eventing](/docs/clouds/aws/guides/lambda/)
The following snippet is the core part of the credential rotator app. We define a function to handle the logic of key rotation in `rotateIAMUserKeys`. Then we create an AWS Lambda resource
named `lambda`. Finally, the `triggerSchedule` resource invokes our lambda on a fixed schedule, thereby ensuring that the key rotation process goes on indefinitely. You can see the full code
When writing reusable infrastructure components in Pulumi however, it is helpful to organize things into a [custom resource](/docs/concepts/resources#custom-resources).
For example, we can bundle together the AWS Lambda, CloudWatcn schedule, and the associated IAM policies into a single conceptual resource `AccessKeyRotator`. Bundling resources allows for the code reuse.
With the mechanics of updating an AWS access key out of the way, the next step is to notify dependent systems what the new access key should be. The key rotator app has a `CredentialPusher` abstraction to provide a pluggable way for you to send credentials to where they need to be. The example on GitHub only supports updating a Travis CI project (See [`credential-pusher-travis.ts`](https://github.com/chrsmith/pulumi-aws-travis-cicd-demo/blob/master/infrastructure/key-rotator/credential-pusher-travis.ts).), but it could be easily extended to support other CI/CD
[Pulumi stack's configuration](https://www.pulumi.com/docs/concepts/config/), and hard-code the specific set of projects and encrypted environment variables to
To demonstrate the access keys rotation, we can examine the log files generated from AWS Lambda. These can be accessed from the command-line using the [`pulumi logs`](/docs/cli/commands/pulumi_logs) command.
As you can see, the AWS credentials for the IAM User are automatically updated every few hours. And whenever a new access key is created, the value is pushed out to the impacted Travis CI projects automatically.
In this post, we covered some of the things to consider when providing AWS credentials to your CI/CD system. (In short, be cautious and follow best practices.)
We then showed how relatively easy it is to stand up a Pulumi infrastructure application for automatically rotating AWS access keys and updating your CI/CD system. By rotating credentials, you can limit the impact if the value is inadvertently disclosed.
At this point, we've now given access keys to a CI/CD system for a low-privilege AWS IAM User account. But we still can't _do_ anything with it. The IAM User whose credentials we have given to the CI/CD system doesn't have access to your production AWS account.
In the next post, we'll go into the details about IAM Roles, and they can be used to securely and temporarily gain access to additional resources. This is how our low-privilege IAM User can access needed to update production data for your CI/CD pipelines.
