---
layout: guides
page_title: Packer Build Pipelines
sidebar_title: Pipelineing Builds
description: |-
  Here we explore how to break your builds into discrete steps so that your
  builds can be shorter and more reliable.
---

# Why Create a Template Pipeline?

A common issue users face when beginning to create their Packer templates is
that while they may need several specialized images, the early provisioning
steps are all the same. It can feel tedious to copy all of those images' basic
configuraton into each build template. It can feel even more tedious to wait a
long time for similar builds to run duplicate steps.

This is one reason why Packer recommends breaking your builds into small,
discrete steps. Not only does it allow you to create "base" images that you can
build from to create further customizations, but it also allows you to save
time in your build process because the "base" images are likely to change less
than your customizations.

It also makes it so that a failing build takes less time to debug and re-run.

In this example, we will use the Virtualbox builders, but the concepts from
this example can be applied to other builders as well.

## Starting from an ISO

Here is an extremely basic virtualbox-iso template:

<Tabs>
<Tab heading="JSON">

```json
{
  "builders": [
    {
      "type": "virtualbox-iso",
      "vm_name": "vbox-example",
      "boot_command": [
        "<esc><wait>",
        "<esc><wait>",
        "<enter><wait>",
        "/install/vmlinuz<wait>",
        " initrd=/install/initrd.gz",
        " auto-install/enable=true",
        " debconf/priority=critical",
        " preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ubuntu_preseed.cfg<wait>",
        " -- <wait>",
        "<enter><wait>"
      ],
      "http_directory": "./http",

      "disk_size": "40960",
      "guest_os_type": "Ubuntu_64",
      "iso_checksum": "sha256:946a6077af6f5f95a51f82fdc44051c7aa19f9cfc5f737954845a6050543d7c2",
      "iso_url": "http://old-releases.ubuntu.com/releases/14.04.1/ubuntu-14.04-server-amd64.iso",

      "shutdown_command": "echo 'vagrant' | sudo -S shutdown -P now",

      "ssh_port": 22,
      "ssh_username": "vagrant",
      "ssh_password": "vagrant"
    }
  ],
  "provisioners": [
    {
      "type": "shell",
      "inline": ["echo initial provisioning"]
    }
  ],
  "post-processors": [
    {
      "type": "manifest",
      "output": "stage-1-manifest.json"
    }
  ]
}
```

</Tab>
<Tab heading="HCL2">

```hcl
source "virtualbox-iso" "step_1" {
  boot_command     = ["<esc><wait>", "<esc><wait>", "<enter><wait>",
                      "/install/vmlinuz<wait>", " initrd=/install/initrd.gz",
                      " auto-install/enable=true", " debconf/priority=critical",
                      " preseed/url=http://{{ .HTTPIP }}:{{ .HTTPPort }}/ubuntu_preseed.cfg<wait>",
                      " -- <wait>", "<enter><wait>"]
  disk_size        = "40960"
  guest_os_type    = "Ubuntu_64"
  http_directory   = "./http"
  iso_checksum     = "sha256:946a6077af6f5f95a51f82fdc44051c7aa19f9cfc5f737954845a6050543d7c2"
  iso_url          = "http://old-releases.ubuntu.com/releases/14.04.1/ubuntu-14.04-server-amd64.iso"
  shutdown_command = "echo 'vagrant' | sudo -S shutdown -P now"
  ssh_password     = "vagrant"
  ssh_port         = 22
  ssh_username     = "vagrant"
  vm_name          = "vbox-example"
}
build {
  sources = ["source.virtualbox-iso.step_1"]


  provisioner "shell" {
    inline = ["echo initial provisioning"]
  }
  post-processor "manifest" {
    output = "stage-1-manifest.json"
  }
}
```

</Tab>
</Tabs>

In order to build using this template, create a directory named "http" in your
current working directory. Copy the minimal example from our
[preseed guide](https://www.packer.io/guides/automatic-operating-system-installs/preseed_ubuntu#examples)
into a file in your http directory and name it "ubuntu_preseed.cfg". Copy the
above json template into your current working directory and save it as
"example_virtualbox_iso.json"

To run the build, call `packer build example_virtualbox_iso.json`.

This example does not set the output_directory or output_filename, so the file
will be placed in a default name of "output-virtualbox-iso/vbox-example.ovf" --
the builder will print this file name to the UI output, but in this example the
[manifest](https://www.packer.io/docs/post-processors/manifest) post-processor
to will store build information, including the names of the output files, in a
json file named "stage-1-manifest.json". From there, you can programmatically
look up the output file information.

## Customizing the iso using the virtualbox-ovf builder

That output filename generated in the first stage can be used as the
[source_path](https://www.packer.io/docs/builders/virtualbox/ovf#source_path)
for the virtualbox-ovf builder.

<Tabs>
<Tab heading="JSON">

```json
{
  "builders": [
    {
      "type": "virtualbox-ovf",
      "vm_name": "virtualbox-example-ovf",

      "shutdown_command": "echo 'vagrant' | sudo -S shutdown -P now",
      "source_path": "output-virtualbox-iso/vbox-example.ovf",

      "ssh_password": "vagrant",
      "ssh_port": 22,
      "ssh_username": "vagrant"
    }
  ],
  "provisioners": [
    {
      "inline": ["echo secondary provisioning"],
      "type": "shell"
    }
  ]
}
```

</Tab>
<Tab heading="HCL2">

```hcl
source "virtualbox-ovf" "step_2" {
  shutdown_command = "echo 'vagrant' | sudo -S shutdown -P now"
  source_path      = "output-virtualbox-iso/vbox-example.ovf"
  ssh_password     = "vagrant"
  ssh_port         = 22
  ssh_username     = "vagrant"
  vm_name          = "virtualbox-example-ovf"
}

build {
  sources = ["source.virtualbox-ovf.step_2"]

  provisioner "shell" {
    inline = ["echo secondary provisioning"]
  }
}

```

</Tab>
</Tabs>

## More efficiencies

You may find that you want to run time-consuming import post-processors like
the "amazon-import" post-processor independently of the build that produces
the artifacts you want to process.

In this case, you can use a null builder
and manually modify the input to the post-processing chain so that you can get
the behavior you want. The below example shows a "vagrant" post-processor
being used with a null builder, and manually sets the artifact from our
stage-2 ovf build:

<Tabs>
<Tab heading="JSON">

```json
{
  "builders": [
    {
      "type": "null",
      "communicator": "none"
    }
  ],
  "post-processors": [
    [
      {
        "type": "artifice",
        "files": [
          "output-virtualbox-ovf/virtualbox-example-ovf.ovf",
          "output-virtualbox-ovf/virtualbox-example-ovf-disk001.vmdk"
        ]
      },
      {
        "type": "vagrant",
        "provider_override": "virtualbox"
      }
    ]
  ]
}
```

</Tab>
<Tab heading="HCL2">

```hcl
source "null" "step_3" {
  communicator = "none"
}

build {
  sources = ["source.null.step_3"]

  post-processors {
    post-processor "artifice" {
      files = ["output-virtualbox-ovf/virtualbox-example-ovf.ovf", "output-virtualbox-ovf/virtualbox-example-ovf-disk001.vmdk"]
    }
    post-processor "vagrant" {
      provider_override = "virtualbox"
    }
  }
}
```

</Tab>
</Tabs>

By using the null builder instead of just running an ovf builder, we can spare
ourselves all of the time Packer would normally spend launching and destroying
VMs.

## Putting it all together

Packer templates don't come with a custom "glue" to bind them together. We
recommend using your CI system or wrapping scripts to connect the templates
into a chain.

## Chaining together several of the same builders to make "save points"

If you want to use the same builder for several builds in a row, this can feel
tedious to implement in json. We recommend you try using HCL configs so that
you can reuse the same source in several builds:

HCL templates work by allowing you to draw sources and variables from multiple
different files in a single directory, so the following files are assumed to
exist in their own folder:

sources.pkr.hcl
```hcl
// In your sources file, you can create a configuration for a builder that you
// want to reuse between multiple steps in the build. Just leave the source
// and destination images out of this source, and set them specifically in each
// step without having to set all of the other options over and over again.

source "docker" "example" {
  commit = true
  // any other configuration you want for your docker containers
}
```

build.pkr.hcl
```hcl
build {
  // Make sure to name your builds so that you can selectively run them one at
  // a time.
  name = "step1"

  source "source.docker.example" {
    image = "ubuntu"
  }

  provisioner "shell" {
    inline = ["echo example provisioner"]
  }
  provisioner "shell" {
    inline = ["echo another example provisioner"]
  }
  provisioner "shell" {
    inline = ["echo a third example provisioner"]
  }

  // Make sure that the output from your build can be used in the next build.
  // In this example, we're tagging the docker image so that the step-2
  // builder can find it without us having to track it down in a manifest.
  post-processor "docker-tag" {
    repository = "ubuntu"
    tag = ["step-1-output"]
  }
}

build {
    name = "step2"

    source "source.docker.example" {
      // This is the tagged artifact from the stage 1 build. You can retrieve
      // this from a manifest file and setting it as a variable on the command
      // line, or by making sure you define and know the output of the build,
      // if it's something you can define like an output name or directory.
      image = "ubuntu:step-1-output"
      // disable the pull if your image tag only exists locally
      pull = false
    }

    provisioner "shell" {
      inline = ["echo another provision!"]
    }
}
```

pipeline.sh
```sh
#!/bin/bash
packer build -only='step1.docker.example' .
packer build -only='step2.docker.example' .
```

To run the pipeline, call pipeline.sh. You can add error checking to this script to abort if there's an issue with a build. You can create as many build steps as you want. Each can either inhabit one file