Merge pull request #10489 from hashicorp/datasource_documentation

(1) Add Data Sources docs

mlc_config.json

@@ -21,6 +21,12 @@
     {
       "pattern": "^/docs/templates/hcl_templates/"
     },
+    {
+      "pattern": "^/docs/datasources"
+    },
+    {
+      "pattern": "^/docs/extending/custom-datasources"
+    },
     {
       "pattern": "^/docs/templates/legacy_json_templates"
     },

website/content/docs/extending/custom-datasources.mdx

@@ -0,0 +1,96 @@
+---
+description: >
+  Packer Data Sources are the components of Packer that allow data to be fetched for use within the Packer configuration.
+  Use of data sources allows a build to use information defined outside of Packer.
+page_title: Custom Data Sources - Extending
+sidebar_title: Custom Data Sources
+---
+
+# Custom Data Sources
+
+Packer Data Sources are the components of Packer that allow data to be fetched for use within the configuration.
+Use of data sources allows a build to use information defined outside of Packer. An example of
+data source is the [amazon-ami data source](/docs/datasources/amazon-ami), which outputs the data of a fetched Amazon AMI.
+
+Prior to reading this page, it is assumed you have read the page on [plugin
+development basics](/docs/extending/plugins).
+
+Data Source plugins implement the `packersdk.Datasource` interface and are registered within a plugin Set
+with `set.RegisterDatasource(...)` function and served using the `set.Run()`.
+
+~> **Warning!** This is an advanced topic. If you're new to Packer, we
+recommend getting a bit more comfortable before you dive into writing plugins.
+
+## The Interface
+
+The interface that must be implemented for a datasource is the
+`packersdk.Datasource` interface. It is reproduced below for reference. The
+actual interface in the source code contains some basic documentation as well
+explaining what each method should do.
+
+```go
+type Datasource interface {
+    ConfigSpec() hcldec.ObjectSpec
+    OutputSpec() hcldec.ObjectSpec
+    Configure(...interface{}) error
+    Execute() (cty.Value, error)
+}
+```
+
+### The "ConfigSpec" Method
+
+This method returns a hcldec.ObjectSpec, which is a spec necessary for using
+HCL2 templates with Packer. For information on how to use and implement this
+function, check our
+[object spec docs](/guides/hcl/component-object-spec)
+
+### The "OutputSpec" Method
+
+This method returns a [hcldec.ObjectSpec](/guides/hcl/component-object-spec) of the data source output.
+The object spec can be generated using the command [`mapstructure-to-hcl2`](https://github.com/hashicorp/packer/tree/master/cmd/mapstructure-to-hcl2)
+just like the configuration spec for the `ConfigSpec` method.
+
+This method is used in `packer validate` command. Packer will use the spec to assign
+unknown values to the data source, instead of executing it and fetching real values.
+
+### The "Configure" Method
+
+The `Configure` method is called prior to any runs with the configuration that was given in the template.
+This is passed in as an array of `interface{}` types, but is generally `map[string]interface{}`. The Configure
+method is responsible for translating this configuration into an internal structure, validating it,
+and returning any errors.
+
+For multiple parameters, they should be merged together into the final
+configuration, with later parameters overwriting any previous configuration.
+The exact semantics of the merge are left to the builder author.
+
+For decoding the `interface{}` into a meaningful structure, the
+[mapstructure](https://github.com/mitchellh/mapstructure) library is
+recommended. Mapstructure will take an `interface{}` and decode it into an
+arbitrarily complex struct. If there are any errors, it generates very human
+friendly errors that can be returned directly from the Configure method.
+
+While it is not actively enforced, **no side effects** should occur from
+running the `Configure` method. Specifically, don't create files, don't launch
+virtual machines, etc. Configure's purpose is solely to configure the data source and
+validate the configuration.
+
+The `Configure` method is called very early in the build process so that errors
+may be displayed to the user before anything actually happens.
+
+### The "Execute" Method
+
+This method returns an HCL cty.Value and an error. Packer will run the Execute method on `packer build` command
+so that the data source output will be available prior to evaluating build and locals blocks. Is expected that the
+Execute command will fetch the data and return it as a cty.Value.
+
+To get the equivalent cty.Value from an output config, we suggest using our
+[packer-plugin-sdk hcl2helper functions](https://github.com/hashicorp/packer-plugin-sdk/blob/v0.0.7/hcl2helper/values.go).
+
+
+## Scaffolding template
+
+To make your experience easier when developing your new data source plugin, we provide you a
+[Packer plugin scaffolding](https://github.com/hashicorp/packer-plugin-scaffolding)
+to use as a template to your plugin repository. The template is structure with the basics and contain the necessary
+configuration to start creating your own plugin.

website/content/docs/extending/index.mdx

@@ -10,6 +10,6 @@ sidebar_title: Extending Packer
 # Extending Packer
 
 Packer is designed to be extensible. Because the surface area for workloads is
-infinite, Packer supports plugins for builders, provisioners, and
-post-processors. To learn more about the different customizations, please
+infinite, Packer supports plugins for builders, provisioners,
+post-processors, and data sources. To learn more about the different customizations, please
 choose a link from the sidebar.

website/content/docs/templates/hcl_templates/blocks/data.mdx

@@ -0,0 +1,17 @@
+---
+description: >
+  The data block defines data sources within your Packer configuration.
+page_title: data - Blocks
+sidebar_title: <tt>data</tt>
+---
+
+# The `data` block
+
+The `data` block defines data sources within your Packer configuration.
+
+`@include 'from-1.5/datasources/example-block.mdx'`
+
+# More on data sources
+
+- Read the [full Data Sources](/docs/templates/hcl_templates/datasources) description for a more
+  thorough read.

website/content/docs/templates/hcl_templates/blocks/index.mdx

@@ -57,3 +57,7 @@ list of all of the available built-in HCL2 blocks.
 `@include 'from-1.5/builds/example-block.mdx'`
 
 - [build block documentation](/docs/templates/hcl_templates/blocks/build).
+
+`@include 'from-1.5/datasources/example-block.mdx'`
+
+- [data block documentation](/docs/templates/hcl_templates/blocks/data).

website/content/docs/templates/hcl_templates/datasources.mdx

@@ -0,0 +1,82 @@
+---
+page_title: Data Sources
+sidebar_title: Data Sources
+description: >-
+  Data sources allow data to be fetched or computed for use elsewhere in local variables and
+  build sources configuration. Use of data sources
+  allows a Builder to make use of information defined outside of Packer.
+---
+
+# Data Sources
+
+-> **Note:** Data Sources is a feature included in Packer 1.7 and later.
+
+Data sources allow data to be fetched or computed for use elsewhere in [locals](/docs/templates/hcl_templates/blocks/locals) and
+[sources](/docs/templates/hcl_templates/blocks/source) configuration.
+Use of data sources allows a Builder to make use of information defined outside of Packer.
+
+# Using Data Sources
+
+A data source is declared using a data block, and the configuration looks like the following:
+
+```hcl
+data "amazon-ami" "example" {
+    filters = {
+        virtualization-type = "hvm"
+        name = "ubuntu/images/*ubuntu-xenial-16.04-amd64-server-*"
+        root-device-type = "ebs"
+    }
+    owners = ["099720109477"]
+    most_recent = true
+}
+```
+
+A data block requests that Packer read from a given data source ("amazon-ami") and export the result under the given
+local name ("example"). The name is used to refer to this data source from elsewhere in the same Packer configuration.
+
+The data block creates a data instance of the given _type_ (first block label) and _name_ (second block label).
+The combination of the type and name must be unique within a configuration.
+
+Within the block (the `{ }`) is the configuration for the data instance. The configuration is dependent on the type,
+and is documented for each data source in the [data sources](/docs/datasources) section.
+
+A data source can output one or more attributes, which can be used by adding their key name to the data source unique
+identifier, like `data.<TYPE>.<NAME>.<ATTRIBUTE>`.
+
+The output from the `amazon-ami.example` above can be accessed as follows:
+
+Output data:
+```
+"data.amazon-ami.example" {
+  id = "ami12345"
+  name = "MyAMI"
+  creation_date = "01/01/2021"
+  owner = "123456789"
+  owner_name = "Some Name"
+  tags = {"tag1": "value"}
+}
+```
+
+Usage:
+
+```hcl
+// in a local
+locals {
+  source_ami_id = data.amazon-ami.example.id
+  source_ami_name = data.amazon-ami.example.name
+ }
+```
+
+```hcl
+// in a source
+source "amazon-ebs" "basic-example" {
+  source_ami = locals.source_ami
+  // ...
+}
+```
+
+## Related
+
+- The list of available data sources can be found in the [data sources](/docs/datasources)
+  section.
+- Create your own [custom data source](/docs/extending/custom-datasources) !

website/content/partials/from-1.5/datasources/example-block.mdx

@@ -0,0 +1,6 @@
+```hcl
+# datasource.pkr.hcl
+data "amazon-ami" "basic-example" {
+  // ...
+}
+```

website/data/docs-navigation.js

@@ -45,6 +45,7 @@ export default [
               'source',
               'variable',
               'packer',
+              'data'
             ],
           },
           {
@@ -179,6 +180,7 @@ export default [
           'variables',
           'locals',
           'contextual-variables',
+          'datasources',
           'path-variables',
           'syntax',
           'onlyexcept',
@@ -306,6 +308,7 @@ export default [
       'custom-builders',
       'custom-post-processors',
       'custom-provisioners',
+      'custom-datasources',
     ],
   },
   '---------',