diff --git a/website/content/docs/extending/custom-builders.mdx b/website/content/docs/extending/custom-builders.mdx index 7c1412d1c..c8999e9bd 100644 --- a/website/content/docs/extending/custom-builders.mdx +++ b/website/content/docs/extending/custom-builders.mdx @@ -10,23 +10,27 @@ sidebar_title: Custom Builders Packer Builders are the components of Packer responsible for creating a machine, bringing it to a point where it can be provisioned, and then turning -that provisioned machine into some sort of machine image. Several builders are -officially distributed with Packer itself, such as the AMI builder, the VMware -builder, etc. However, it is possible to write custom builders using the Packer -plugin interface, and this page documents how to do that. +that provisioned machine into a machine image. Several builders are +officially maintained and distributed by the HashiCorp Packer team -- these +builders include builders for creating images on Amazon EC2, VMWare, Google +Compute Engine, and many more. It is also possible to write custom builders +using the Packer plugin interface, and this page documents how to do that. -Prior to reading this page, it is assumed you have read the page on [plugin -development basics](/docs/extending/plugins). +Prior to reading this page, you should read the page on [plugin development +basics](/docs/extending/plugins). ~> **Warning!** This is an advanced topic. If you're new to Packer, we -recommend getting a bit more comfortable before you dive into writing plugins. +recommend getting comfortable with using Packer and its officially maintained +plugins before you dive into writing plugins of your own. + + +Custom plugins are written in [golang](https://golang.org/), so this guide +assumes that you have some familiarity with that programming language. ## The Interface -The interface that must be implemented for a builder is the `packer.Builder` -interface. It is reproduced below for reference. The actual interface in the -source code contains some basic documentation as well explaining what each -method should do. +To create your own builder, you must create a struct that implements the +`packer.Builder` interface. It is reproduced below for reference. ```go type Builder interface { @@ -45,72 +49,101 @@ function, check our ### The "Prepare" Method -The `Prepare` method for each builder is called prior to any runs with the -configuration that was given in the template. This is passed in as an array of -`interface{}` types, but is generally `map[string]interface{}`. The prepare -method is responsible for translating this configuration into an internal -structure, validating it, and returning any errors. +The `Prepare` method for each builder will be called by the Packer core +at the beginning of the build. Its purpose is to parse and validate the +configuration template provided to Packer with `packer build +your_packer_template.json`, but not to execute API calls or begin creating any +resources or artifacts. -For multiple parameters, they should be merged together into the final -configuration, with later parameters overwriting any previous configuration. -The exact semantics of the merge are left to the builder author. +The configuration from your Packer template is passed into the Prepare() method +as an array of `interface{}` types, but is generally `map[string]interface{}`. +The Prepare method is responsible for translating this configuration into an +internal structure, validating it, and returning any errors. -For decoding the `interface{}` into a meaningful structure, the -[mapstructure](https://godoc.org/github.com/mitchellh/mapstructure) library is -recommended. Mapstructure will take an `interface{}` and decode it into an -arbitrarily complex struct. If there are any errors, it generates very human -friendly errors that can be returned directly from the prepare method. +If multiple parameters are passed into Prepare(), they should be merged together +into the final configuration, with later parameters overwriting any previous +configuration. The exact semantics of the merge are left to the builder author. -While it is not actively enforced, **no side effects** should occur from -running the `Prepare` method. Specifically, don't create files, don't launch -virtual machines, etc. Prepare's purpose is solely to configure the builder and -validate the configuration. +We recommend that you use the +[mapstructure](https://godoc.org/github.com/mitchellh/mapstructure) library to +decode the interface{} into a meaningful structure. Mapstructure will take an +`interface{}` and decode it into an arbitrarily complex struct. If there are any +errors, it generates very human friendly errors that can be returned directly +from the prepare method. You can find many usage examples of this library within +the Prepare() methods of HashiCorp-maintained Packer plugins. -In addition to normal configuration, Packer will inject a -`map[string]interface{}` with a key of `packer.DebugConfigKey` set to boolean +While Packer does not actively enforce this, **no side effects** should occur +from running the `Prepare` method. Specifically: don't create files, don't +launch virtual machines, etc. Prepare's purpose is solely to load the +configuration from the template into a format usable by your builder, to +validate that configuration, and to apply necessary defaults to that +configuration. + +In addition to the configuration provided in the Packer template, Packer will +also supply a [common.PackerConfig](https://github.com/hashicorp/packer-plugin-sdk/blob/8a28198491f70deca3824ce452adf6f9bd507880/common/packer_config.go#L44) +containing meta-information such as the build name, builder type, core version, +etc, and coded into a `map[string]interface{}`. One important piece of +meta information in this map is the `packer.DebugConfigKey` set to boolean `true` if debug mode is enabled for the build. If this is set to true, then the builder should enable a debug mode which assists builder developers and advanced users to introspect what is going on during a build. During debug builds, parallelism is strictly disabled, so it is safe to request input from stdin and so on. +Prepare() returns an array of strings and an error. The array of strings is a +special list of keys for variables created at runtime which your builder will +make accessible to provisioners using the generatedData mechanism (see below for +more details) An example could be an instance ID for the cloud instance created +by your builder. If you do not plan to make use of the generatedData feature, +just return an empty list. The error should be used if there is something wrong +with the user-provided configuration and the build should not proceed. + ### The "Run" Method -`Run` is where all the interesting stuff happens. Run is executed, often in -parallel for multiple builders, to actually build the machine, provision it, -and create the resulting machine image, which is returned as an implementation -of the `packer.Artifact` interface. +`Run` is executed, often in parallel for multiple builders, to actually build +the machine, provision it, and create the resulting machine image, which is +returned as an implementation of the `packer.Artifact` interface. -The `Run` method takes three parameters. These are all very useful. The -`packer.Ui` object is used to send output to the console. `packer.Hook` is used -to execute hooks, which are covered in more detail in the hook section below. -And `packer.Cache` is used to store files between multiple Packer runs, and is -covered in more detail in the cache section below. +The `Run` method takes three parameters. The context.Context used to cancel the +build. The `packer.Ui` object is used to send output to the console. +`packer.Hook` is used to execute hooks, which are covered in more detail in the +Provisioning section below. Because builder runs are typically a complex set of many steps, the +packer-plugin-sdk contains a [multistep](https://godoc.org/github.com/hashicorp/packer-plugin-sdk/multistep) -package is recommended to bring order to the complexity. Multistep is a library -which allows you to separate your logic into multiple distinct "steps" and -string them together. It fully supports cancellation mid-step and so on. Please -check it out, it is how the built-in builders are all implemented. +module. Multistep allows you to separate your build logic into multiple distinct +"steps" with separate run and cleanup phases, and run them in order. It +supports cancellation mid-step, pausing between steps when debugging, the CLI's +on-error flag, and more. All of the HashiCorp maintained builders make use of +this module, and while it is not required for builder implementation, it will +help you create your builder in a way that matches user and Packer Core +assumptions. The SDK also provides a number of "helper" generic steps that may +prevent you from having to re-implement work that has already been done by the +HashiCorp maintainers. Examples include sending boot commands, connecting to +SSH, and creating virtual CDs to mount on your VM. Take a look at the +[communicator](https://github.com/hashicorp/packer-plugin-sdk/tree/main/communicator) +and +[multistep/commonsteps](https://github.com/hashicorp/packer-plugin-sdk/tree/main/multistep/commonsteps) +modules in the SDK to see what tools are available to you. -Finally, as a result of `Run`, an implementation of `packer.Artifact` should be -returned. More details on creating a `packer.Artifact` are covered in the -artifact section below. If something goes wrong during the build, an error can -be returned, as well. Note that it is perfectly fine to produce no artifact and -no error, although this is rare. +Finally, `Run` should return an implementation of `packer.Artifact`. More +details on creating a `packer.Artifact` are covered in the artifact section +below. If something goes wrong during the build that prevents an artifact from +being correctly created, `Run` should return an error and a nil artifact. Note +that your builder is allowed to produce no artifact and no error, although this +is a rare use case. ### Cancellation -The `Run` method is often run in parallel. - -#### With the "Cancel" Method ( up until packer 1.3 ) +#### With the "Cancel" Method ( for plugins for Packer < v1.3 ) The `Cancel` method can be called at any time and requests cancellation of any builder run in progress. This method should block until the run actually stops. -Not that the Cancel method will no longer be called since packer 1.4.0. +Note that the Cancel method will not be called by Packer versions >= 1.4.0. -#### Context cancellation ( from packer 1.4 ) + +#### Context cancellation ( from Packer v1.4 ) The `<-ctx.Done()` can unblock at any time and signifies request for cancellation of any builder run in progress. @@ -118,32 +151,36 @@ cancellation of any builder run in progress. Cancels are most commonly triggered by external interrupts, such as the user pressing `Ctrl-C`. Packer will only exit once all the builders clean up, so it is important that you architect your builder in a way that it is quick to -respond to these cancellations and clean up after itself. +respond to these cancellations and clean up after itself. If your builder makes +a long-running call, you should consider the possibility that a user may cancel +the build during that call, and make sure that such a cancellation is not +blocked. ## Creating an Artifact The `Run` method is expected to return an implementation of the -`packer.Artifact` interface. Each builder must create their own implementation. -The interface has ample documentation to help you get started. +`packer.Artifact` interface. Each builder must create its own implementation of +this interface. -The only part of an artifact that may be confusing is the `BuilderId` method. -This method must return an absolutely unique ID for the builder. In general, I -follow the practice of making the ID contain my GitHub username and then the -platform it is building for. For example, the builder ID of the VMware builder -is "hashicorp.vmware" or something similar. +Most of the pieces of an artifact should be fairly self-explanatory by reading +the [packer.Artifact interface +documentation](https://godoc.org/github.com/hashicorp/packer-plugin-sdk/packer#Artifact). + +However one part of an artifact that may be confusing is the `BuilderId` method. +This method must return an absolutely unique ID for the builder. In general, a +reasonable ID would be the github username or organization that created the +builder, followed by the platform it is building for. For example, the builder +ID of the VMware builder is "hashicorp.vmware". Post-processors use the builder ID value in order to make some assumptions about the artifact results, so it is important it never changes. -Other than the builder ID, the rest should be self-explanatory by reading the -[packer.Artifact interface -documentation](https://godoc.org/github.com/hashicorp/packer-plugin-sdk/packer#Artifact). - ## Provisioning -Packer has built-in support for provisioning, but the moment when provisioning -runs must be invoked by the builder itself, since only the builder knows when -the machine is running and ready for communication. +Packer has built-in support for provisioning using the Provisioner plugins. But +builders themselves, rather than the Packer core, must determine when to invoke +the provisioners since only the builder knows when the machine is running and +ready for communication. When the machine is ready to be provisioned, run the `packer.HookProvision` hook, making sure the communicator is not nil, since this is required for @@ -156,22 +193,25 @@ hook.Run(context.Context, packer.HookProvision, ui, comm, nil) At this point, Packer will run the provisioners and no additional work is necessary. --> **Note:** Hooks are still undergoing thought around their general design -and will likely change in a future version. They aren't fully "baked" yet, so -they aren't documented here other than to tell you how to hook in provisioners. +If you are using the multistep tooling, the Packer plugin SDK contains a +generic StepProvision which handles execution the provision hook for you and +automatically supplies any custom builder generatedData you would like to +provide to procisioners (see below for more details on generatedData.) ## Template Engine ### Build variables -Packer makes it possible to provide custom template engine variables to be shared with -provisioners and post-processors using the `build` function. +Packer makes it possible to provide custom template engine variables to be +shared with provisioners and post-processors using the `build` function. +Json template `build` docs are [here](https://www.packer.io/docs/templates/engine#build) +and HCL template build docs are [here](https://www.packer.io/docs/from-1.5/contextual-variables#build-variables). -Part of the builder interface changes made in 1.5.0 was to make builder Prepare() methods -return a list of custom variables which we call `generated data`. -We use that list of variables to generate a custom placeholder map per builder that -combines custom variables with the placeholder map of default build variables created by Packer. -Here's an example snippet telling packer what will be made available by the builder: +As of Packer v1.5.0, builder Prepare() methods return a list of custom variables +which we call `generated data`. We use that list of variables to generate a +custom placeholder map per builder that combines custom variables with the +placeholder map of default build variables created by Packer. Here's an example +snippet telling packer what will be made available by the builder: ```go func (b *Builder) Prepare(raws ...interface{}) ([]string, []string, error) { @@ -182,11 +222,13 @@ func (b *Builder) Prepare(raws ...interface{}) ([]string, []string, error) { } ``` -Returning the custom variable name(s) within the `generated_data` placeholder is necessary -for the template containing the build variable(s) to validate. +When a user provides a Packer template that uses a `build` function, Packer +validates that the key in the `build` function exists for a given builder using +this generatedData array. If it does not exist, then Packer validation will +fail. -Once the placeholder is set, it's necessary to pass the variables' real values when calling -the provisioner. This can be done as the example below: +Once the placeholder is set, it's necessary to pass the variables' real values +when calling the provisioner. This can be done as the example below: ```go func (b *Builder) Run(ctx context.Context, ui packer.Ui, hook packer.Hook) (packer.Artifact, error) { @@ -201,10 +243,11 @@ func (b *Builder) Run(ctx context.Context, ui packer.Ui, hook packer.Hook) (pack } ``` -In order to make these same variables and the Packer default ones also available to post-processor, -it is necessary to add them to the Artifact returned by the builder. This can be done by adding an attribute of type -`map[string]interface{}` to the Artifact and putting the generated data in it. The post-processor -will access this data later via the Artifact's `State` method. +In order to make these same variables and the Packer default ones also available +to post-processors, your builder will need to add them to its Artifact. +This can be done by adding an attribute of type `map[string]interface{}` to the +Artifact and putting the generated data in it. The post-processor will access +this data later via the Artifact's `State` method. The Artifact code should be implemented similar to the below: @@ -226,8 +269,8 @@ func (a *Artifact) State(name string) interface{} { // ... ``` -The builder should return the above Artifact containing the generated data and the code should be similar -to the example snippet below: +The builder should return the above Artifact containing the generated data and +the code should be similar to the example snippet below: ```go func (b *Builder) Run(ctx context.Context, ui packer.Ui, hook packer.Hook) (packer.Artifact, error) { @@ -240,7 +283,8 @@ func (b *Builder) Run(ctx context.Context, ui packer.Ui, hook packer.Hook) (pack } ``` -The code above assigns the `generated_data` state to the `StateData` map with the key `generated_data`. +The code above assigns the `generated_data` state to the `StateData` map with +the key `generated_data`. Here some example of how this data will be used by post-processors: @@ -254,4 +298,13 @@ func (p *PostProcessor) PostProcess(ctx context.Context, ui packer.Ui, source pa } ``` -To know more about the template engine build function, please refer to the [template engine docs](/docs/templates/engine). +## Putting it all together + +This page has focused up until now on the implementation details for the Builder +interface. You will need to create a server and store the builder in a binary in +order to make it available to the Packer core as a plugin. We have created a +[scaffolding](https://github.com/hashicorp/packer-plugin-scaffolding/blob/main/builder/scaffolding/builder.go) +repo to give you an idea of the relationship between the builder +implementation and the server implementation within a repository, and then read +[basics of how Plugins work](/docs/extending/plugins), which breaks down all the +server details.