Because Pulumi uses general-purpose programming languages to provision cloud resources, you can take advantage of native tools and perform automated tests of your infrastructure. The full power of each language is available, including access to libraries and frameworks for testing.
This blog post takes a deeper dive into mock-based unit testing of Pulumi programs written in C# and F#.
<!--more-->
## How Unit Testing Works in Pulumi
Let's start with a picture showing how Pulumi components interact during a typical deployment process.
Whenever a new resource is instantiated in code, the program makes a remote call to the engine. The engine receives the resource's input values, validates them, and translates the request to an invocation of a cloud API. The API returns some data, which are translated to resource outputs and sent back to the program.
The remote calls may be slow, unreliable, and non-deterministic, which makes testing of these interactions hard.
The Pulumi SDK provides a hook to replace all the remote calls with mocks. Mocks run in the same process and can respond immediately with hard-coded or calculated on-the-fly data.
We can write fully deterministic and blazingly fast automated tests as all remote calls and uncertainty are eliminated. There is no cloud to respond to resource creation, so it's a developer's responsibility to mimic the cloud behavior with mocks adequately.
This blog post walks you through an example of unit testing and mocking with the .NET SDK.
## Define the Base Structure
In this article, I build a program that deploys a static website to Azure. I use TDD to add new tests and deployment components bit-by-bit.
Let's start with an empty project—go ahead and create a .NET Core Console Application with the .NET CLI or your favorite IDE.
### Install NuGet packages
You are free to choose your unit testing frameworks, mocking and assertions libraries. I'm using NUnit with FluentAssertions, and my program tests Azure resources, so this is my project file:
A Pulumi stack is the "unit" of our testing. Every test instantiates a stack, retrieves the resources that the stack defines, and makes assertions about them.
Here is my starting point in `WebsiteStack` file:
{{<chooserlanguage"csharp,fsharp"/>}}
{{% choosable language csharp %}}
```csharp
using System.IO;
using Pulumi;
using Pulumi.Azure.Core;
using Storage = Pulumi.Azure.Storage;
public class WebsiteStack : Stack
{
public WebsiteStack()
{
// <--Cloudresourcesgohere
}
}
```
{{% /choosable %}}
{{% choosable language fsharp %}}
```fsharp
namespace UnitTesting
open Pulumi
open Pulumi.FSharp
open Pulumi.Azure.Core
type WebsiteStack() =
inherit Stack()
// <--Cloudresourcesgohere
```
{{% /choosable %}}
This class is precisely what Pulumi expects to start deploying your resources to the cloud. Once the test suite is ready and green, you can deploy the stack with `pulumi up`.
### Mocks
As I explained above, unit tests replace the real Pulumi engine with mocks. Mocks get all calls and respond with predefined values.
A mock class has to implement two methods of `Pulumi.Testing.IMocks` interface. `NewResourceAsync` is called whenever a new resource is defined, while `CallAsync` is invoked when our program retrieves information about existing cloud resources. We'll focus on the former in this post, but we still need to define both.
While you are free to use your favorite mocking library, I'll keep it simple and define the mocks as a plain class.
{{<chooserlanguage"csharp,fsharp"/>}}
{{% choosable language csharp %}}
```csharp
using System.Collections.Immutable;
using System.Threading.Tasks;
using Pulumi;
using Pulumi.Testing;
namespace UnitTesting
{
class Mocks : IMocks
{
public Task<(string id, object state)> NewResourceAsync(
// Forward all input parameters as resource outputs, so that we could test them.
let dict = inputs.ToImmutableDictionary() :> obj
// <--We'llcustomizethemockshere
// Default the resource ID to `{name}_id`.
let id = if id = null then sprintf "%s_id" name else id
Task.FromResult(struct (id, outputs :> obj))
member this.CallAsync(token: string, inputs: ImmutableDictionary<string,obj>, provider: string): Task<obj> =
// We don't use this method in this particular test suite.
// Default to returning whatever we got as input.
Task.FromResult(null :> obj)
```
{{% /choosable %}}
Both methods receive several parameters and need to return a result. Notably, they get the list of `inputs`—the arguments that our program defined for the given resource. The goal of the mocks is to return the list of outputs, similarly to what a cloud provider would do.
My default implementation returns the outputs that include all the inputs and nothing else. That's a sensible default: usually, real outputs are a superset of inputs. We'll extend this behavior as we need later on.
### Test Fixture
The next class to add is the container for my future unit tests. I named the file `WebsiteStackTests.cs` and it defines a test container, called a "test fixture" in NUnit:
I defined a helper method `TestAsync` that points Pulumi's `Deployment.TestAsync` to our stack and mock classes.
As we progress through the article, I will add tests to this class one-by-one.
### Retrieve output values
Finally, I need a small extension method to extract values from outputs. Every Pulumi resource returns values wrapped inside an `Output<T>` container. While the real engine runs, those values may be known or unknown, but my mock-based tests always return known values. It's safe to get those values and convert them to a `Task<T>`:
{{<chooserlanguage"csharp,fsharp"/>}}
{{% choosable language csharp %}}
```csharp
public static class TestingExtensions
{
public static Task<T> GetValueAsync<T>(this Output<T> output)
The setup is done, time to write my first unit test! True to the TDD spirit, I write the tests before I define any resources.
Every Azure resource has to live inside a resource group, so my first test checks that a new resource group is defined.
{{<chooserlanguage"csharp,fsharp"/>}}
{{% choosable language csharp %}}
```csharp
[Test]
public async Task SingleResourceGroupExists()
{
var resources = await TestAsync();
var resourceGroups = resources.OfType<ResourceGroup>().ToList();
resourceGroups.Count.Should().Be(1, "a single resource group is expected");
}
```
{{% /choosable %}}
{{% choosable language fsharp %}}
```fsharp
[<Test>]
member this.SingleResourceGroupExists() =
let resources = runTest()
let resourceGroupCount = resources.OfType<ResourceGroup>() |> Seq.length
resourceGroupCount.Should().Be(1, "a single resource group is expected") |> ignore
```
{{% /choosable %}}
This code is great to illustrate the overall structure of each test:
1. Call the `TestAsync` method to evaluate the stack.
2. Find the target resource in the collection of created resources.
3. Assert a property of the target resource.
In this case, the test validates that there is one and only one resource group defined.
I can run `dotnet test` and see the expected test failure:
```
$ dotnet test
...
X SingleResourceGroupExists [238ms]
Error Message:
Expected resourceGroups.Count to be 1 because a single resource group is expected, but found 0.
...
Total tests: 1
Failed: 1
Total time: 0.9270 Seconds
```
I go ahead and add the following definition to the `WebsiteStack` constructor.
{{<chooserlanguage"csharp,fsharp"/>}}
{{% choosable language csharp %}}
```csharp
var resourceGroup = new ResourceGroup("www-prod-rg");
```
{{% /choosable %}}
{{% choosable language fsharp %}}
```fsharp
let resourceGroup = new ResourceGroup("www-prod-rg")
```
{{% /choosable %}}
This change is enough to make the tests green!
```
$ dotnet test
...
Test Run Successful.
Total tests: 1
Passed: 1
Total time: 0.8462 Seconds
```
## Tags
For billing and lifecycle management, I want all my resource groups to be tagged. Therefore, my second test validates that the resource group has an `Environment` tag on it:
{{<chooserlanguage"csharp,fsharp"/>}}
{{% choosable language csharp %}}
```csharp
[Test]
public async Task ResourceGroupHasEnvironmentTag()
{
var resources = await TestAsync();
var resourceGroup = resources.OfType<ResourceGroup>().First();
var tags = await resourceGroup.Tags.GetValueAsync();
tags.Should().NotBeNull("Tags must be defined");
tags.Should().ContainKey("Environment");
}
```
{{% /choosable %}}
{{% choosable language fsharp %}}
```fsharp
[<Test>]
member this.ResourceGroupHasEnvironmentTag() =
let resources = runTest()
let resourceGroup = resources.OfType<ResourceGroup>() |> Seq.head
let tags = getValue resourceGroup.Tags
tags.Should().NotBeNull("Tags must be defined") |> ignore
The test finds the resource group and then checks that the `Tags` dictionary is not null and contains a tag with the name `Environment`. Predictably, the test fails:
```
$ dotnet test
...
X ResourceGroupHasEnvironmentTag [240ms]
Error Message:
Expected tags not to be <null> because Tags must be defined.
...
Test Run Failed.
```
Now, I edit the stack class and change the resource group definition.
{{<chooserlanguage"csharp,fsharp"/>}}
{{% choosable language csharp %}}
```csharp
var resourceGroup = new ResourceGroup("www-prod-rg", new ResourceGroupArgs
{
Tags = { { "Environment", "production" } }
});
```
{{% /choosable %}}
{{% choosable language fsharp %}}
```fsharp
let resourceGroup =
new ResourceGroup(
"www-prod-rg",
new ResourceGroupArgs(Tags = inputMap(["Environment", input "production"])))
```
{{% /choosable %}}
This change makes the test suite green again, and we are good to proceed.
## Storage Account
It's time to add a test for a second resource in the stack: a Storage Account. To make things more interesting, I want to check that the storage account belongs to our resource group.
There's no direct link between a storage account object and a resource group object. Instead, we need to check the `ResourceGroupName` property of the account. The test below expects the account to belong to a resource group called `www-prod-rg`.
{{<chooserlanguage"csharp,fsharp"/>}}
{{% choosable language csharp %}}
```csharp
[Test]
public async Task StorageAccountBelongsToResourceGroup()
{
var resources = await TestAsync();
var storageAccount = resources.OfType<Storage.Account>().SingleOrDefault();
storageAccount.Should().NotBeNull("Storage account not found");
var resourceGroupName = await storageAccount.ResourceGroupName.GetValueAsync();
resourceGroupName.Should().Be("www-prod-rg");
}
```
{{% /choosable %}}
{{% choosable language fsharp %}}
```fsharp
[<Test>]
member this.StorageAccountBelongsToResourceGroup() =
let resources = runTest()
let storageAccount = resources.OfType<Account>() |> Seq.tryHead |> Option.toObj
storageAccount.Should().NotBeNull("Storage account not found") |> ignore
let resourceGroupName = getValue storageAccount.ResourceGroupName
Of course, the test fails. I try to fix it with what I think is the minimal change to make the test pass by adding a new resource to the stack and pointing it to the resource group.
var storageAccount = new Storage.Account("wwwprodsa", new Storage.AccountArgs
{
ResourceGroupName = resourceGroup.Name,
});
```
{{% /choosable %}}
{{% choosable language fsharp %}}
```fsharp
let storageAccount =
new Account(
"wwwprodsa",
new AccountArgs(ResourceGroupName = io resourceGroup.Name))
```
{{% /choosable %}}
However, when I rerun the test suite, all three tests fail. They complain about the missing required properties `AccountReplicationType` and `AccountTier` on the storage account, so I have to add those.
{{<chooserlanguage"csharp,fsharp"/>}}
{{% choosable language csharp %}}
```csharp
var storageAccount = new Storage.Account("wwwprodsa", new Storage.AccountArgs
{
ResourceGroupName = resourceGroup.Name,
AccountTier = "Standard",
AccountReplicationType = "LRS",
StaticWebsite = new Storage.Inputs.AccountStaticWebsiteArgs
{
IndexDocument = "index.html"
}
});
```
{{% /choosable %}}
{{% choosable language fsharp %}}
```fsharp
let storageAccount =
new Account(
"wwwprodsa",
new AccountArgs(
ResourceGroupName = io resourceGroup.Name,
AccountTier = input "Standard",
AccountReplicationType = input "LRS",
StaticWebsite = input (new AccountStaticWebsiteArgs(IndexDocument = input "index.html"))))
```
{{% /choosable %}}
I defined the `StaticWebsite` property as well: I'll leave testing these values as an exercise for the reader.
Two resource group tests are back to green again, but the storage account test fails with a new error message.
```
X StorageAccountBelongsToResourceGroup [84ms]
Error Message:
Expected resourceGroupName to be "www-prod-rg", but found <null>.
I defined the account name like this: `ResourceGroupName = resourceGroup.Name`. However, if we look closely, the `resourceGroup` resource doesn't have an input property `Name` defined. `www-prod-rg` is a logical name for Pulumi deployment, not the physical name of the resource.
Under normal circumstances, the Pulumi engine would use the logical name to produce the physical name of the resource group automatically (see [resource names](/docs/concepts/resources#names) for details). However, my mocks don't do that.
let nameKVs = if inputs.ContainsKey("name") then [] else [("name", name :> obj)]
let outputs = [inputKVs; nameKVs] |> Seq.concat |> Seq.map KeyValuePair
let dict = outputs.ToImmutableDictionary() :> obj
```
{{% /choosable %}}
Note that mocks operate on weakly-typed dictionaries, so I need to get the property name right. Pulumi SDKs are open source, so I looked [here](https://github.com/pulumi/pulumi-azure/blob/1fdcab88065175ced768d900e7dcedf3b1d1b0a7/sdk/dotnet/Core/ResourceGroup.cs#L90) to double-check the exact value.
After this change in `Mocks`, my tests go green again.
```
Test Run Successful.
Total tests: 3
Passed: 3
```
## Website Files
The next step is to upload some files to the static website. Well, instead, to write an automated test that validates the upload with mocks.
Once again, our mocks fail to represent the behavior of the engine accurately. The Pulumi engine knows about the `FileAsset` class pointing to a file on the disk and how to convert it to an uploaded blob. But, the engine doesn't copy this property to outputs. I need to adjust the mocks again.
I'm not particularly interested in testing the binary contents of the files now, so I'll change the `Mocks` class to ignore the `source` property and not to include it into the output dictionary.
{{<chooserlanguage"csharp,fsharp"/>}}
{{% choosable language csharp %}}
```csharp
if (type == "azure:storage/blob:Blob")
{
// Assets can't directly go through the engine.
// We don't need them in the test, so blank out the property for now.
So far, I've been validating the input parameters of resources. What if I want to test something that is usually done by the cloud provider?
For instance, when Azure creates a static website, it automatically assigns a public endpoint. The endpoint is then available in the `PrimaryWebEndpoint` property after the Pulumi program ran and resources are created. I may want to export this value from stack outputs and validate it in a unit test.
{{<chooserlanguage"csharp,fsharp"/>}}
{{% choosable language csharp %}}
```csharp
[Test]
public async Task StackExportsWebsiteUrl()
{
var resources = await TestAsync();
var stack = resources.OfType<WebsiteStack>().First();
var endpoint = await stack.Endpoint.GetValueAsync();
This test doesn't even compile yet, so I have to define the `Endpoint` property and assign a value to it. Here is the complete implementation of the stack with the output property defined at the end.
{{<chooserlanguage"csharp,fsharp"/>}}
{{% choosable language csharp %}}
```csharp
public class WebsiteStack : Stack
{
public WebsiteStack()
{
var resourceGroup = new ResourceGroup("www-prod-rg", new ResourceGroupArgs
{
Tags = { { "Environment", "production" } }
});
var storageAccount = new Storage.Account("wwwprodsa", new Storage.AccountArgs
{
ResourceGroupName = resourceGroup.Name,
AccountTier = "Standard",
AccountReplicationType = "LRS",
StaticWebsite = new Storage.Inputs.AccountStaticWebsiteArgs
{
IndexDocument = "index.html"
}
});
var files = Directory.GetFiles("wwwroot");
foreach (var file in files)
{
var blob = new Storage.Blob(file, new Storage.BlobArgs
then ["primaryWebEndpoint", sprintf "https://%s.web.core.windows.net" name :> obj]
else []
let outputs =
[inputKVs; nameKVs; endpointKVs]
// ...
```
{{% /choosable %}}
And that's it! My static website is ready and tested!
```
Test Run Successful.
Total tests: 5
Passed: 5
Total time: 0.9338 Seconds
```
Note that it still takes less than a second to run my test suite so that I can iterate very quickly.
## Get Started
The tests above cover the basics of unit testing with Pulumi .NET SDK. You can take it from here and apply the techniques and practices that you use while testing the application code. You may also try more advanced practices like property-based testing or behavior-driven development—we believe that the mocks enable many testing styles.
Here are several useful pointers to get started with testing in Pulumi:
- Full code for this blog post: [C#](https://github.com/pulumi/examples/tree/72c9480f4c1240f795f6020f50801733fbef37f2/testing-unit-cs-mocks), [F#](https://github.com/pulumi/examples/tree/72c9480f4c1240f795f6020f50801733fbef37f2/testing-unit-fs-mocks).
