676 lines
28 KiB
Markdown

---
title: "Managing GitHub Webhooks with Pulumi"
authors: ["matt-ellis"]
tags: ["Serverless"]
meta_desc: "Use Pulumi to write a hook that would clean up branches after a pull request got merged."
date: "2018-07-13"
---
At Pulumi, we do all of our development on GitHub, with a workflow built
around topic branches. When a developer wants to make a change, they
push a branch to GitHub, open a pull request and (in theory) once it's
merged, delete the branch. In practice, we'll often forget to delete the
topic branch (I'm probably the worst offender), which means we end up
having topic branches linger on our main repository until they are
explicitly cleaned up. While it's a lot of fun to go a click through the
GitHub UI from time to time, deleting merged branches, it's even more
fun to build automation to do this for us. Since GitHub has a rich set
of [webhooks](https://developer.github.com/webhooks/) and Pulumi makes
it easy to write serverless functions, it felt like it would be natural
to use Pulumi to write a hook that would clean up branches after a pull
request got merged. In addition, Pulumi lets us leverage real
programming languages to build abstractions, which means we can build a
simple framework that hides much of the ceremony behind defining a hook
and lets us focus on the core logic of our hook, without worrying about
how it is deployed and managed.
<!--more-->
## A simple hook
When you register a webhook with GitHub, you provide an HTTP endpoint
and a set of events you'd like to listen for. When one of the events
happens on GitHub, it makes a POST request to your HTTP endpoint with a
payload that describes the event. We'll start by creating a simple
endpoint that just logs when an event happens. We'll start by using the
[`aws-serverless`](https://www.npmjs.com/package/@pulumi/aws-serverless)
package to define an API that just calls `console.log` to write the
`X-GitHub-Event` header that is sent with the request:
```typescript
import * as serverless from "@pulumi/aws-serverless";
const api = new serverless.apigateway.API("hook", {
routes: [{
path: "/",
method: "POST",
handler: async (req, ctx) => {
console.log(`Got event ${req.headers['X-GitHub-Event']}`);
return {
statusCode: 200,
body: ""
}
}
},],
});
export const url = api.url;
```
After running `pulumi update`, we'll have a HTTP endpoint we can give to
GitHub:
Updating stack 'github-hook-blog-dev'
Performing changes:
Type Name Status Info
+ pulumi:pulumi:Stack github-hook-blog-github-hook-blog-dev created
+ ├─ aws-serverless:apigateway:API hook created
+ │ ├─ aws:apigateway:RestApi hook created
+ │ ├─ aws:apigateway:Deployment hook created
+ │ ├─ aws:lambda:Permission hook-980655da created
+ │ └─ aws:apigateway:Stage hook created
+ └─ aws:serverless:Function hook980655da created
+ ├─ aws:iam:Role hook980655da created
+ ├─ aws:iam:RolePolicyAttachment hook980655da-32be53a2 created
+ └─ aws:lambda:Function hook980655da created
---outputs:---
url: "https://t1vyz1x203.execute-api.us-west-1.amazonaws.com/stage/"
Now that we have our Webhook deployed, we'll add it to GitHub. In this
case, I have a little throwaway GitHub repository I use for testing this
sort of stuff. In that repository, I go to Settings -> Webhooks -> Add
webhook and fill in my information:
- Payload URL: The value of the `url` output of my Pulumi program (in
this case it is
`https://t1vyz1x203.execute-api.us-west-1.amazonaws.com/stage/`).
- Content Type: `application-json`. I know we'll be inspecting this
content as we develop the hook and since we're writing the
implementation in TypeScript, it will be easier to interact with
JSON encoded data.
- Secret: Generate a random string and put it here. I used
[random.org](https://www.random.org/) to do so, but any random
string will suffice. We'll use this string to validate that the
request came from GitHub.
- Events: I only care about `pull_request` events, so I picked "Let me
select individual events" and then checked "Pull requests" and
unchecked the other event types.
- Active: Since we want GitHub to deliver events, we'll keep this
checked.
Once that's done, we can test our hook by opening a pull request,
waiting a few moments and then use `pulumi logs` to ensure our hook was
triggered:
$ pulumi logs
Collecting logs for stack github-hook-blog-dev since 2018-07-09T09:06:51.000-07:00.
2018-07-09T10:06:00.610-07:00[ hook980655da-d0462d4] START RequestId: 5aab2d71-839a-11e8-a7ae-93a25d68b5a9 Version: $LATEST
2018-07-09T10:06:00.612-07:00[ hook980655da-d0462d4] 2018-07-09T17:06:00.611Z 5aab2d71-839a-11e8-a7ae-93a25d68b5a9 Got event ping
2018-07-09T10:06:00.630-07:00[ hook980655da-d0462d4] END RequestId: 5aab2d71-839a-11e8-a7ae-93a25d68b5a9
2018-07-09T10:06:00.630-07:00[ hook980655da-d0462d4] REPORT RequestId: 5aab2d71-839a-11e8-a7ae-93a25d68b5a9 Duration: 10.99 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 20 MB
2018-07-09T10:06:23.241-07:00[ hook980655da-d0462d4] START RequestId: 68445127-839a-11e8-80a6-759a29eb6005 Version: $LATEST
2018-07-09T10:06:23.243-07:00[ hook980655da-d0462d4] 2018-07-09T17:06:23.243Z 68445127-839a-11e8-80a6-759a29eb6005 Got event pull_request
2018-07-09T10:06:23.250-07:00[ hook980655da-d0462d4] END RequestId: 68445127-839a-11e8-80a6-759a29eb6005
2018-07-09T10:06:23.250-07:00[ hook980655da-d0462d4] REPORT RequestId: 68445127-839a-11e8-80a6-759a29eb6005 Duration: 1.32 ms Billed Duration: 100 ms Memory Size: 128 MB Max Memory Used: 20 MB
Here we can see our hook got two events, the first is the
[`ping`](https://developer.github.com/webhooks/#ping-event) event, which
GitHub sends to verify the hook. The second event is from when I opened
a pull request in the repository.
Before we go any further, let's start using the secret we set to
validate the Webhook payload. When GitHub sends an event to a Webhook,
it sets a special header
[`X-Hub-Signature`](https://developer.github.com/webhooks/#delivery-headers)
which is an HMAC digest of the request body. The value we placed in the
Secret field when configuring the Webhook is the key. So I'll take that
secret value, set it as a configuration value in my project:
$ pulumi config set --secret hookSecret [READACTED]
Once that's done, I can use this value to validate the signature. We'll
use some functions from node's `crypto` library at runtime to handle
this. If the signature we computed doesn't match what was provided with
the request, we'll return error code 400. Our program now looks like
this:
```typescript
import * as pulumi from "@pulumi/pulumi";
import * as serverless from "@pulumi/aws-serverless";
const cfg = new pulumi.Config(pulumi.getProject());
const hookSecret = cfg.require("hookSecret");
const api = new serverless.apigateway.API("hook", {
routes: [{
path: "/",
method: "POST",
handler: async (req, ctx) => {
// Compute the HMAC of the body, using `hookSecret` as the key and `sha1` as the algorithm
// First, grab the body. It may be base64 encoded, depending on if `req.isBase64Encoded` is set. If so, we
// should decode it.
let body = req.body;
if (req.isBase64Encoded) {
body = Buffer.from(body, 'base64').toString();
}
// Now compute the HMAC.
const crypto = await import("crypto");
const hmac = crypto.createHmac("sha1", hookSecret);
hmac.update(body);
const computedSignature = `sha1=${hmac.digest("hex")}`;
// Compare the signature that came in with the request to what we computed.
if (req.headers['X-Hub-Signature'] === undefined ||
!crypto.timingSafeEqual(Buffer.from(req.headers['X-Hub-Signature']), Buffer.from(computedSignature)))
{
console.log(`error: bad signature ${req.headers['X-Hub-Signature']} !== ${computedSignature}`);
return {
statusCode: 400,
body: "bad signature"
}
}
console.log(`Got event ${req.headers['X-GitHub-Event']} and signature ${computedSignature} matched`);
return {
statusCode: 200,
body: ""
}
}
},],
});
export const url = api.url;
```
After deploying, we can trigger another event by closing an existing
pull request or opening a new one, and then use `pulumi logs` to confirm
that our validation is working as intended:
$ pulumi logs
Collecting logs for stack github-hook-blog-dev since 2018-07-09T12:43:05.000-07:00.
2018-07-09T13:43:00.630-07:00[ hook980655da-d0462d4] START RequestId: ab363ec7-83b8-11e8-a7ae-93a25d68b5a9 Version: $LATEST
2018-07-09T13:43:00.693-07:00[ hook980655da-d0462d4] 2018-07-09T20:43:00.693Z ab363ec7-83b8-11e8-a7ae-93a25d68b5a9 Got event pull_request and signature sha1=b83320205edc7cb9aeea1a845b85de0c83092e1d matched
2018-07-09T13:43:00.731-07:00[ hook980655da-d0462d4] END RequestId: ab363ec7-83b8-11e8-a7ae-93a25d68b5a9
2018-07-09T13:43:00.731-07:00[ hook980655da-d0462d4] REPORT RequestId: ab363ec7-83b8-11e8-a7ae-93a25d68b5a9 Duration: 128.75 ms Billed Duration: 200 ms Memory Size: 128 MB Max Memory Used: 20 MB
## Building an abstraction
We now have a nice little skeleton that we can use when writing a
webhook on GitHub. Let's start to leverage some other Pulumi features to
make things simpler and provide a nicer interface for writing hooks.
Let's start by abstracting away the code that handles validating the
request and returning a response behind a helper method. We'll pass a
function to this helper which will be called after the validation has
completed. We'll also parse the body into a JSON object that our handler
can use.
```typescript
import * as pulumi from "@pulumi/pulumi";
import * as serverless from "@pulumi/aws-serverless";
const cfg = new pulumi.Config(pulumi.getProject());
const hookSecret = cfg.require("hookSecret");
function createWebhook(name: string, handler: ((req: serverless.apigateway.Request, body: any) => Promise<void>)) {
return new serverless.apigateway.API(name, {
routes: [{
path: "/",
method: "POST",
handler: async (req, ctx) => {
// Compute the HMAC of the body, using `hookSecret` as the key and `sha1` as the algorithm
// First, grab the body. It may be base64 encoded, depending on if `req.isBase64Encoded` is set. If so, we
// should decode it.
let body = req.body;
if (req.isBase64Encoded) {
body = Buffer.from(body, 'base64').toString();
}
// Now compute the HMAC.
const crypto = await import("crypto");
const hmac = crypto.createHmac("sha1", hookSecret);
hmac.update(body);
const computedSignature = `sha1=${hmac.digest("hex")}`;
// Compare the signature that came in with the request to what we computed.
if (req.headers['X-Hub-Signature'] === undefined ||
!crypto.timingSafeEqual(Buffer.from(req.headers['X-Hub-Signature']), Buffer.from(computedSignature)))
{
console.log(`error: bad signature ${req.headers['X-Hub-Signature']} !== ${computedSignature}`);
return {
statusCode: 400,
body: "bad signature"
}
}
// Call the handler after parsing the body as JSON
await handler(req, JSON.parse(body));
return {
statusCode: 200,
body: ""
}
}
},],
});
}
const api = createWebhook("hook", async (req, body) => {
console.log(`Got event ${req.headers['X-GitHub-Event']} with action ${body.action}`);
});
export const url = api.url;
```
The shape of the JSON object for this event is
[documented](https://developer.github.com/v3/activity/events/types/#pullrequestevent)
on GitHub. Here we're pulling out the action field, which tells us
something about the state of the pull request. For our bot, we only care
about PRs where the `action` is `closed` and the `merged` key is set to
`true`. When this holds, we'll want to delete the branch the pull
request came from, if it was a topic branch (instead of a PR opened from
a fork). We'll leverage the existing [`@octokit/rest` NPM package](https://www.npmjs.com/package/@octokit/rest) to do this. We can
generate a [personal access token](https://github.com/settings/tokens),
and give it `repo` level scope, so it can act as us and delete branches.
We'll then add this key to our configuration:
$ pulumi config set --secret githubToken [REDACTED]
And we can now update our handler to use this API:
```typescript
import * as pulumi from "@pulumi/pulumi";
import * as serverless from "@pulumi/aws-serverless";
const cfg = new pulumi.Config(pulumi.getProject());
const hookSecret = cfg.require("hookSecret");
function createWebhook(name: string, handler: ((req: serverless.apigateway.Request, body: any) => Promise<void>)) {
/* Same as the previous version */
}
const githubToken = cfg.require("githubToken");
const api = createWebhook("hook", async (req, body) => {
if (body.action === "closed" && body.pull_request.merged &&
body.pull_request.base.user.login === body.pull_request.head.user.login &&
body.pull_request.base.repo.name === body.pull_request.head.repo.name)
{
const octokit = require('@octokit/rest')()
octokit.authenticate({
type: 'token',
token: githubToken
});
console.log(`Deleting reference heads/${body.pull_request.head.ref}`);
await octokit.gitdata.deleteReference({
owner: body.pull_request.head.user.login,
repo: body.pull_request.head.repo.name,
ref: `heads/${body.pull_request.head.ref}`
});
}
});
export const url = api.url;
```
After `pulumi updating`, try merging a pull request! Shortly after you
click the merge button, the source branch should be deleted
automatically. You can also see our log message in the output of
`pulumi logs`:
$ pulumi logs
[matell@matell throwaway]$ pulumi logs
Collecting logs for stack github-hook-blog-dev since 2018-07-09T13:30:26.000-07:00.
2018-07-09T14:28:52.254-07:00[ hook980655da-d0462d4] START RequestId: 1368afec-83bf-11e8-bf0d-cf623578c325 Version: $LATEST
2018-07-09T14:28:52.431-07:00[ hook980655da-d0462d4] 2018-07-09T21:28:52.371Z 1368afec-83bf-11e8-bf0d-cf623578c325 Deleting reference heads/test-branch
2018-07-09T14:28:53.478-07:00[ hook980655da-d0462d4] END RequestId: 1368afec-83bf-11e8-bf0d-cf623578c325
2018-07-09T14:28:53.478-07:00[ hook980655da-d0462d4] REPORT RequestId: 1368afec-83bf-11e8-bf0d-cf623578c325 Duration: 1221.97 ms Billed Duration: 1300 ms Memory Size: 128 MB Max Memory Used: 33 MB
So, in about 75 lines of code, we've defined a small abstraction for
authoring a GitHub Webhook and we've used it to write a little bot that
deletes topic branches after pull requests from them have been merged.
In itself, that's pretty cool, but we can do some more cool stuff with
Pulumi. As is, while we can create the Webhook using Pulumi, we have to
register it using the GitHub console, which is a little tedious. What if
we could manage the registration of the hook with Pulumi itself? We'd
really like to model the hook's association with GitHub as a resource
that can be created, updated and deleted. Is there a way we can do that?
Yes, there is!
## Managing all the things with Pulumi
Pulumi [uses gRPC](https://github.com/pulumi/pulumi/blob/master/proto/pulumi/provider.proto)
to define a contract between resource providers and the rest of Pulumi.
So, we could go implement that contract in a language like go, like we
do in our [Kubernetes](https://github.com/pulumi/pulumi-kubernetes)
provider. Another option would be to take the existing
[Terraform GitHub Provider](https://github.com/terraform-providers/terraform-provider-github)
and use [Pulumi's terraform bridge](https://github.com/pulumi/pulumi-terraform/) wrap it into a
Pulumi resource provider, like we do with our
[pulumi-aws](https://github.com/pulumi/pulumi-aws) provider. There's one
final option, which is to use the "[dynamic provider](https://github.com/pulumi/pulumi/blob/ad3b5e7ee88346bc7e960de9f953957a72f84516/sdk/nodejs/dynamic/index.ts#L109)",
which allows us to implement a resource provider in JavaScript itself.
We wrote this provider to help us with testing, so we could mock out
resources and control their lifecycle, but we can also use it to create
a resource that manages registration of a webhook with GitHub. To do so,
we need to implement `dynamic.ResourceProvider` interface. Once we have
the provider, we can also write a resource that uses it.
```typescript
class GithubWebhookProvider implements dynamic.ResourceProvider {
// Check ensures that all required properties are set. In this case we have three required parameters.
check = async (olds: any, news: any) => {
const failedChecks = [];
for (const prop of ["url", "owner", "repo"]) {
if (news[prop] === undefined) {
failedChecks.push({property: prop, reason: `required property '${prop}' missing`});
}
}
return { inputs: news, failedChecks: failedChecks };
}
// Today the engine does the diff between properties to detect if there is a change but this method does
// tell the engine if the changes between the old and new values require the resource to be "replaced"
// (that is a new one is created and the old one is deleted) vs being edited in place. For us, if the owner
// or repo the hook is installed on changes, we'll trigger a replacement.
diff = async (id: pulumi.ID, olds: any, news: any) => {
const replaces = [];
for (const prop of ["owner", "repo"]) {
if (olds[prop] !== news[prop]) {
replaces.push(prop);
}
}
return { replaces: replaces };
}
// Create actually creates the hook. We use octokit under the hood and return the ID of the hook that was created.
// Pulumi retains this ID and gives it to us when we need to update or delete the hook.
create = async (inputs: any) => {
const octokit = require("@octokit/rest")();
octokit.authenticate({
type: "token",
token: githubToken,
});
const res = await octokit.repos.createHook({
owner: inputs["owner"],
repo: inputs["repo"],
name: "web",
events: ["pull_request"],
config: {
content_type: "json",
url: inputs["url"],
secret: hookSecret,
},
});
if (res.status !== 201) {
throw new Error(`bad response: ${JSON.stringify(res)}`);
}
// The engine expects that the ID property is a string.
return {
id: `${res.data.id}`,
};
}
update = async (id: pulumi.ID, olds: any, news: any) => {
const octokit = require("@octokit/rest")();
octokit.authenticate({
type: "token",
token: githubToken,
});
const res = await octokit.repos.editHook({
hook_id: id,
owner: news["owner"],
repo: news["repo"],
events: ["pull_request"],
config: {
content_type: "json",
url: news["url"],
},
});
if (res.status !== 200) {
throw new Error(`bad response: ${JSON.stringify(res)}`);
}
return {}
}
delete = async (id: pulumi.ID, props: any) => {
const octokit = require("@octokit/rest")();
octokit.authenticate({
type: "token",
token: githubToken,
});
const res = await octokit.repos.deleteHook({
hook_id: id,
owner: props["owner"],
repo: props["repo"],
});
if (res.status !== 204) {
throw new Error(`bad response: ${JSON.stringify(res)}`);
}
}
}
interface GitHubWebhookResourceArgs {
url: pulumi.Input<string>;
owner: pulumi.Input<string>;
repo: pulumi.Input<string>;
}
class GitHubWebhookResource extends dynamic.Resource {
constructor(name: string, args: GitHubWebhookResourceArgs, opts?: pulumi.ResourceOptions) {
super(new GithubWebhookProvider(), name, args, opts);
}
}
```
With this new dynamic resource, registering the hook itself becomes
easy:
```typescript
new GitHubWebhookResource("hook-registration", {
url: api.url,
owner: "ellismg",
repo: "testing",
});
```
Before running `pulumi update` go and manually delete the hook
registration on GitHub. Now, when you `pulumi update` you'll see the new
resource created:
Updating stack 'github-hook-blog-dev'
Performing changes:
Type Name Status Info
* pulumi:pulumi:Stack github-hook-blog-github-hook-blog-dev done
+ └─ pulumi-nodejs:dynamic:Resource hook-registration created
---outputs:---
url: "https://t1vyz1x203.execute-api.us-west-1.amazonaws.com/stage/"
info: 1 change performed:
+ 1 resource created
10 resources unchanged
Update duration: 4.649249758s
And if you look on GitHub, the hook should once again be registered!
Now, if the URL for our handler ends up changing or we destroy our
stack, the hook registration will be updated accordingly. Since our
little program is getting large (~200 lines at this point), let's do a
little more work on our abstraction.
## Building a Component
Pulumi has the concept of a `ComponentResource` which is a resource that
aggregates other resources. Many resources that we interact with day to
day in Pulumi are actually `ComponentResources` (In fact, the API
resource we're using here is itself a component). In another file, we
can create the component resource and move much of our logic into there:
The interesting new code looks like this:
```typescript
export interface GitHubWebhookArgs {
owner: pulumi.Input<string>;
repo: pulumi.Input<string>;
handler: (req: serverless.apigateway.Request, body: any) => Promise<void>;
}
export class GitHubWebhook extends pulumi.ComponentResource {
public readonly url: pulumi.Output<string>;
constructor(name: string, args: GitHubWebhookArgs, opts?: pulumi.ResourceOptions) {
super("github:webhooks:Hook", name, {}, opts);
const api = new serverless.apigateway.API("hook", {
routes: [
{
path: "/",
method: "POST",
handler: async (req, ctx) => {
// Compute the HMAC of the body, using `hookSecret` as the key and `sha1` as the algorithm
// First, grab the body. It may be base64 encoded, depending on if `req.isBase64Encoded` is set. If so, we
// should decode it.
let body = req.body;
if (req.isBase64Encoded) {
body = Buffer.from(body, 'base64').toString();
}
// Now compute the HMAC.
const crypto = await import("crypto");
const hmac = crypto.createHmac("sha1", hookSecret);
hmac.update(body);
const computedSignature = `sha1=${hmac.digest("hex")}`;
// Compare the signature that came in with the request to what we computed.
if (req.headers['X-Hub-Signature'] === undefined ||
!crypto.timingSafeEqual(Buffer.from(req.headers['X-Hub-Signature']), Buffer.from(computedSignature)))
{
console.log(`error: bad signature ${req.headers['X-Hub-Signature']} !== ${computedSignature}`);
return {
statusCode: 400,
body: "bad signature"
}
}
// Call the handler after parsing the body as JSON:
await args.handler(req, JSON.parse(body));
return {
statusCode: 200,
body: ""
}
},
},
],
}, {
parent: this,
});
new GitHubWebhookResource(`${name}-registration-${args.owner}-${args.repo}`, {
owner: args.owner,
repo: args.repo,
url: api.url,
}, {
parent: this,
});
this.url = api.url;
}
}
```
Here, we've transformed our `createWebhook` method into an actual
`GitHubWebhook` component that manages both the API of the hook as the
hook's registration with GitHub. With this abstraction (and all of this
complexity hidden off in a separate `github.ts` file), the code we focus
on when actually writing our bot is quite small:
```typescript
import * as pulumi from "@pulumi/pulumi";
import { GitHubWebhook } from "./github";
const cfg = new pulumi.Config(pulumi.getProject());
const githubToken = cfg.require("githubToken");
const hook = new GitHubWebhook("hook", {
owner: "ellismg",
repo: "testing",
handler: async (req, body) => {
if (body.action === "closed" && body.pull_request.merged &&
body.pull_request.base.user.login === body.pull_request.head.user.login &&
body.pull_request.base.repo.name === body.pull_request.head.repo.name)
{
const octokit = require('@octokit/rest')()
octokit.authenticate({
type: 'token',
token: githubToken
});
console.log(`Deleting reference heads/${body.pull_request.head.ref}`);
await octokit.gitdata.deleteReference({
owner: body.pull_request.head.user.login,
repo: body.pull_request.head.repo.name,
ref: `heads/${body.pull_request.head.ref}`
});
}
},
});
export const url = hook.url;
```
## Going Further
From here, there's a lot of ways we could extend this code. For example,
while we require the configuration value `hookSecret` to be set before
using this abstraction, we could manage this on behalf of the user. To
do so, we could use the dynamic provider to create a special "random"
resource, this resource would generate a new random string when created,
but update calls would not impact the resource. Since Pulumi stores the
state of every resource across invocations in the checkpoint, our random
resource would not need to be backed by any cloud infrastructure. We
could change the shape of the `GitHubWebhook` component to allow
multiple owner/repository pairs to be provided and it would generate a
single AWS API and then multiple `GitHubWebhookResource`'s to register
the same hook across multiple repositories. We could even extend this to
allow registration of both repository and organization level webhooks.
Or instead of hard-coding in the `pull_request` event as the only event
we care about take it as another input property to `GitHubWebhook`.
One of my favorite things about Pulumi is while it is easy to *build*
these abstractions, it's also easy to share them. For example, I've
implemented a few of these ideas to build a framework for our use at
Pulumi. You can find the source on
[GitHub](https://github.com/ellismg/github-webhooks-serverless) and I've
published it as an NPM package at
[`@ellismg/pulumi-github-webhooks`](https://www.npmjs.com/package/@ellismg/pulumi-github-webhooks).