diff --git a/mlc_config.json b/mlc_config.json index 1f3956ef1..5bd08fbb7 100644 --- a/mlc_config.json +++ b/mlc_config.json @@ -21,6 +21,12 @@ { "pattern": "^/docs/templates/hcl_templates/" }, + { + "pattern": "^/docs/datasources" + }, + { + "pattern": "^/docs/extending/custom-datasources" + }, { "pattern": "^/docs/templates/legacy_json_templates" }, diff --git a/website/content/docs/extending/custom-datasources.mdx b/website/content/docs/extending/custom-datasources.mdx new file mode 100644 index 000000000..8ef063ae5 --- /dev/null +++ b/website/content/docs/extending/custom-datasources.mdx @@ -0,0 +1,96 @@ +--- +description: > + Packer Data Sources are the components of Packer that allow data to be fetched for use within the Packer configuration. + Use of data sources allows a build to use information defined outside of Packer. +page_title: Custom Data Sources - Extending +sidebar_title: Custom Data Sources +--- + +# Custom Data Sources + +Packer Data Sources are the components of Packer that allow data to be fetched for use within the configuration. +Use of data sources allows a build to use information defined outside of Packer. An example of +data source is the [amazon-ami data source](/docs/datasources/amazon-ami), which outputs the data of a fetched Amazon AMI. + +Prior to reading this page, it is assumed you have read the page on [plugin +development basics](/docs/extending/plugins). + +Data Source plugins implement the `packersdk.Datasource` interface and are registered within a plugin Set +with `set.RegisterDatasource(...)` function and served using the `set.Run()`. + +~> **Warning!** This is an advanced topic. If you're new to Packer, we +recommend getting a bit more comfortable before you dive into writing plugins. + +## The Interface + +The interface that must be implemented for a datasource is the +`packersdk.Datasource` interface. It is reproduced below for reference. The +actual interface in the source code contains some basic documentation as well +explaining what each method should do. + +```go +type Datasource interface { + ConfigSpec() hcldec.ObjectSpec + OutputSpec() hcldec.ObjectSpec + Configure(...interface{}) error + Execute() (cty.Value, error) +} +``` + +### The "ConfigSpec" Method + +This method returns a hcldec.ObjectSpec, which is a spec necessary for using +HCL2 templates with Packer. For information on how to use and implement this +function, check our +[object spec docs](/guides/hcl/component-object-spec) + +### The "OutputSpec" Method + +This method returns a [hcldec.ObjectSpec](/guides/hcl/component-object-spec) of the data source output. +The object spec can be generated using the command [`mapstructure-to-hcl2`](https://github.com/hashicorp/packer/tree/master/cmd/mapstructure-to-hcl2) +just like the configuration spec for the `ConfigSpec` method. + +This method is used in `packer validate` command. Packer will use the spec to assign +unknown values to the data source, instead of executing it and fetching real values. + +### The "Configure" Method + +The `Configure` method is called prior to any runs with the configuration that was given in the template. +This is passed in as an array of `interface{}` types, but is generally `map[string]interface{}`. The Configure +method is responsible for translating this configuration into an internal structure, validating it, +and returning any errors. + +For multiple parameters, they should be merged together into the final +configuration, with later parameters overwriting any previous configuration. +The exact semantics of the merge are left to the builder author. + +For decoding the `interface{}` into a meaningful structure, the +[mapstructure](https://github.com/mitchellh/mapstructure) library is +recommended. Mapstructure will take an `interface{}` and decode it into an +arbitrarily complex struct. If there are any errors, it generates very human +friendly errors that can be returned directly from the Configure method. + +While it is not actively enforced, **no side effects** should occur from +running the `Configure` method. Specifically, don't create files, don't launch +virtual machines, etc. Configure's purpose is solely to configure the data source and +validate the configuration. + +The `Configure` method is called very early in the build process so that errors +may be displayed to the user before anything actually happens. + +### The "Execute" Method + +This method returns an HCL cty.Value and an error. Packer will run the Execute method on `packer build` command +so that the data source output will be available prior to evaluating build and locals blocks. Is expected that the +Execute command will fetch the data and return it as a cty.Value. + +To get the equivalent cty.Value from an output config, we suggest using our +[packer-plugin-sdk hcl2helper functions](https://github.com/hashicorp/packer-plugin-sdk/blob/v0.0.7/hcl2helper/values.go). + + +## Scaffolding template + +To make your experience easier when developing your new data source plugin, we provide you a +[Packer plugin scaffolding](https://github.com/hashicorp/packer-plugin-scaffolding) +to use as a template to your plugin repository. The template is structure with the basics and contain the necessary +configuration to start creating your own plugin. diff --git a/website/content/docs/extending/index.mdx b/website/content/docs/extending/index.mdx index 363c350cd..48f7316c7 100644 --- a/website/content/docs/extending/index.mdx +++ b/website/content/docs/extending/index.mdx @@ -10,6 +10,6 @@ sidebar_title: Extending Packer # Extending Packer Packer is designed to be extensible. Because the surface area for workloads is -infinite, Packer supports plugins for builders, provisioners, and -post-processors. To learn more about the different customizations, please +infinite, Packer supports plugins for builders, provisioners, +post-processors, and data sources. To learn more about the different customizations, please choose a link from the sidebar. diff --git a/website/content/docs/templates/hcl_templates/blocks/data.mdx b/website/content/docs/templates/hcl_templates/blocks/data.mdx new file mode 100644 index 000000000..1740e8dba --- /dev/null +++ b/website/content/docs/templates/hcl_templates/blocks/data.mdx @@ -0,0 +1,17 @@ +--- +description: > + The data block defines data sources within your Packer configuration. +page_title: data - Blocks +sidebar_title: data +--- + +# The `data` block + +The `data` block defines data sources within your Packer configuration. + +`@include 'from-1.5/datasources/example-block.mdx'` + +# More on data sources + +- Read the [full Data Sources](/docs/templates/hcl_templates/datasources) description for a more + thorough read. diff --git a/website/content/docs/templates/hcl_templates/blocks/index.mdx b/website/content/docs/templates/hcl_templates/blocks/index.mdx index ddd9bc245..ca5df5cfb 100644 --- a/website/content/docs/templates/hcl_templates/blocks/index.mdx +++ b/website/content/docs/templates/hcl_templates/blocks/index.mdx @@ -57,3 +57,7 @@ list of all of the available built-in HCL2 blocks. `@include 'from-1.5/builds/example-block.mdx'` - [build block documentation](/docs/templates/hcl_templates/blocks/build). + +`@include 'from-1.5/datasources/example-block.mdx'` + +- [data block documentation](/docs/templates/hcl_templates/blocks/data). diff --git a/website/content/docs/templates/hcl_templates/datasources.mdx b/website/content/docs/templates/hcl_templates/datasources.mdx new file mode 100644 index 000000000..f0a6359bb --- /dev/null +++ b/website/content/docs/templates/hcl_templates/datasources.mdx @@ -0,0 +1,82 @@ +--- +page_title: Data Sources +sidebar_title: Data Sources +description: >- + Data sources allow data to be fetched or computed for use elsewhere in local variables and + build sources configuration. Use of data sources + allows a Builder to make use of information defined outside of Packer. +--- + +# Data Sources + +-> **Note:** Data Sources is a feature included in Packer 1.7 and later. + +Data sources allow data to be fetched or computed for use elsewhere in [locals](/docs/templates/hcl_templates/blocks/locals) and +[sources](/docs/templates/hcl_templates/blocks/source) configuration. +Use of data sources allows a Builder to make use of information defined outside of Packer. + +# Using Data Sources + +A data source is declared using a data block, and the configuration looks like the following: + +```hcl +data "amazon-ami" "example" { + filters = { + virtualization-type = "hvm" + name = "ubuntu/images/*ubuntu-xenial-16.04-amd64-server-*" + root-device-type = "ebs" + } + owners = ["099720109477"] + most_recent = true +} +``` + +A data block requests that Packer read from a given data source ("amazon-ami") and export the result under the given +local name ("example"). The name is used to refer to this data source from elsewhere in the same Packer configuration. + +The data block creates a data instance of the given _type_ (first block label) and _name_ (second block label). +The combination of the type and name must be unique within a configuration. + +Within the block (the `{ }`) is the configuration for the data instance. The configuration is dependent on the type, +and is documented for each data source in the [data sources](/docs/datasources) section. + +A data source can output one or more attributes, which can be used by adding their key name to the data source unique +identifier, like `data...`. + +The output from the `amazon-ami.example` above can be accessed as follows: + +Output data: +``` +"data.amazon-ami.example" { + id = "ami12345" + name = "MyAMI" + creation_date = "01/01/2021" + owner = "123456789" + owner_name = "Some Name" + tags = {"tag1": "value"} +} +``` + +Usage: + +```hcl +// in a local +locals { + source_ami_id = data.amazon-ami.example.id + source_ami_name = data.amazon-ami.example.name + } +``` + +```hcl +// in a source +source "amazon-ebs" "basic-example" { + source_ami = locals.source_ami + // ... +} +``` + +## Related + +- The list of available data sources can be found in the [data sources](/docs/datasources) + section. +- Create your own [custom data source](/docs/extending/custom-datasources) ! diff --git a/website/content/partials/from-1.5/datasources/example-block.mdx b/website/content/partials/from-1.5/datasources/example-block.mdx new file mode 100644 index 000000000..b77bae8e8 --- /dev/null +++ b/website/content/partials/from-1.5/datasources/example-block.mdx @@ -0,0 +1,6 @@ +```hcl +# datasource.pkr.hcl +data "amazon-ami" "basic-example" { + // ... +} +``` diff --git a/website/data/docs-navigation.js b/website/data/docs-navigation.js index 7e9b6019a..b0b2cf52d 100644 --- a/website/data/docs-navigation.js +++ b/website/data/docs-navigation.js @@ -45,6 +45,7 @@ export default [ 'source', 'variable', 'packer', + 'data' ], }, { @@ -179,6 +180,7 @@ export default [ 'variables', 'locals', 'contextual-variables', + 'datasources', 'path-variables', 'syntax', 'onlyexcept', @@ -306,6 +308,7 @@ export default [ 'custom-builders', 'custom-post-processors', 'custom-provisioners', + 'custom-datasources', ], }, '---------',