Commit Graph

147 Commits

Author SHA1 Message Date
Megan Marsh 2d90ffe7a4 move GeneratedData struct and PlaceholderMsg to same package under common 2020-11-06 10:45:38 -08:00
Wilken Rivera bbb6754367 Fix issue with AWS secrets manager override default stage
Before change
```
⇶  packer build amazon-ebs_secretsmanager_shell-local.json
Error:
template: root:1:3: executing "root" at <aws_secretsmanager `packer/test/keys`

`shell`>: error calling aws_secretsmanager: ResourceNotFoundException: Secrets
Manager can't find the specified secret value for staging label: shell

2020/10/30 12:53:40 [INFO] (telemetry) Finalizing.
template: root:1:3: executing "root" at <aws_secretsmanager `packer/test/keys`
`shell`>: error calling aws_secretsmanager: ResourceNotFoundException: Secrets
Manager can't find the specified secret value for staging label: shell

⇶  packer build amazon-ebs_secretsmanager_shell-local.json.pkr.hcl
Error: Error in function call

  on amazon-ebs_secretsmanager_shell-local.json.pkr.hcl line 28:
  (source code not available)

Call to function "aws_secretsmanager" failed: ResourceNotFoundException: Secrets
Manager can't find the specified secret value for staging label: home.

```

After change
```
⇶  packer.test build amazon-ebs_secretsmanager_shell-local.json
null: output will be in this color.

==> null: Running local shell script: /tmp/packer-shell463393820
    null: boo
    null: keys:powershell
Build 'null' finished after 8 milliseconds 225 microseconds.

==> Wait completed after 8 milliseconds 319 microseconds

==> Builds finished. The artifacts of successful builds are:
--> null: Did not export anything. This is the null builder

⇶  packer.test build amazon-ebs_secretsmanager_shell-local.json.pkr.hcl
null.autogenerated_1: output will be in this color.

==> null.autogenerated_1: Running local shell script: /tmp/packer-shell834410761
    null.autogenerated_1: boo
    null.autogenerated_1: keys:powershell
Build 'null.autogenerated_1' finished after 18 milliseconds 834 microseconds.

==> Wait completed after 18 milliseconds 954 microseconds

==> Builds finished. The artifacts of successful builds are:
--> null.autogenerated_1: Did not export anything. This is the null builder

```
2020-10-30 13:17:38 -04:00
Adrien Delorme 6d4fae0f2d
Add HCL2 aws_secretsmanager function (#10124)
* refactor aws get secrets function out to reuse it else where
* add aws_secretsmanager func and docs for HCL2
* fix GetSecret: allow to pick secret version
2020-10-20 16:21:40 +02:00
Megan Marsh cc7dbf6092
Add consul_key function to integrate concul with hcl2 tempaltes. (#10119)
* Add consul_key function to integrate concul with hcl2 tempaltes.
* sidebar nav
2020-10-19 14:07:13 +02:00
Wilken Rivera 20eeffee0d
integration/secretsmanager: Add support for AWS SharedConfig file (#9781)
This changes updates the AWS Secrets manager session authentication
logic to support reading the AWS configuration file for default
credentials and region settings, if they are not provided via
environment variables.

* Modify error output a little to remove stutter "error ... : error ...`

Results before change
```
unset AWS_REGION
⇶  ~/pkg/packer build amazon-ebs_secretsmanager_shell-local.json
template: root:1:3: executing "root" at <aws_secretsmanager `packer/test/keys`
`shell`>: error calling aws_secretsmanager: Error getting secret: MissingRegion:
could not find region configuration
```

Results after change
```
unset AWS_REGION
⇶  ~/pkg/packer build amazon-ebs_secretsmanager_shell-local.json
null: output will be in this color.

==> null: Running local shell script: /tmp/packer-shell721444992
    null: powershell
Build 'null' finished after 4 milliseconds 121 microseconds.

==> Wait completed after 4 milliseconds 192 microseconds

==> Builds finished. The artifacts of successful builds are:
```
2020-08-18 10:51:48 -04:00
Wilken Rivera 56f6a976de
integrations/secretsmanager: Add support for plaintext, non key/pair, secrets (#9773) 2020-08-17 13:41:21 +02:00
jsmcnair 78c280b1b7 Use common funcs / fix panic / clarify docs 2020-08-12 22:34:13 +01:00
Sylvia Moss 9181a435aa
Make max_retries a string to allow variable interpolation (#9673) 2020-08-05 17:41:20 +02:00
Adrien Delorme 40947c2bf6
HCL2: generate hcl tags with go-cty tags too (#9306)
This will allow to generate the config files:

```go
package main

import (
	"fmt"

	"github.com/hashicorp/hcl/v2/gohcl"
	"github.com/hashicorp/hcl/v2/hclwrite"
	"github.com/hashicorp/packer/builder/alicloud/ecs"
)

func main() {
	name := "name"
	app := ecs.FlatConfig{
		AlicloudImageName: &name,
		ECSSystemDiskMapping: &ecs.FlatAlicloudDiskDevice{
			DiskName: &name,
		},
	}

	f := hclwrite.NewEmptyFile()
	block := gohcl.EncodeAsBlock(&app, `source "something" "something"`)
	f.Body().AppendBlock(block)
	fmt.Printf("%s", f.Bytes())
}

```

Will output:

```
source "something" "something" {
  packer_user_variables      = null
  packer_sensitive_variables = null
  image_name                 = "name"
  image_share_account        = null
  image_unshare_account      = null
  image_copy_regions         = null
  image_copy_names           = null
  tags                       = null
  tag                        = null
  system_disk_mapping        = { disk_category = null, disk_delete_with_instance = null, disk_descri
ption = null, disk_device = null, disk_encrypted = null, disk_name = "name", disk_size = null, disk_
snapshot_id = null }
  image_disk_mappings        = null
  ssh_remote_tunnels         = null
  ssh_local_tunnels          = null
  ssh_public_key             = null
  ssh_private_key            = null
}
```

This is a good first step for #9015 and #9282 

fix #9304
2020-05-28 11:19:00 +02:00
Megan Marsh 4a058dfd19 un-bypass secretsmanager tests 2020-05-26 11:50:46 -04:00
Juan Mesa cc34da365a Add error check
Thrown an error if no key is given when secret has mutliple values
2020-05-23 15:15:06 +02:00
Megan Marsh 8f73a44410 stop failing flaky test until I can fix it 2020-05-21 10:24:42 -07:00
Juan Mesa b73602eadd Fix corrections
* Fix typos in comments.
* Add documentation of how to use multiple keys with `aws_secretsmanager` function.
* Move `Client` struct to `secretsmanager.go` file.
2020-05-19 13:16:28 +02:00
Juan Mesa 55fa3e1b0b Add support for specifying key to fetch from AWS Secrets Manager 2020-05-15 13:50:33 +02:00
Adrien Delorme 4ebcbad332
update consul and vault dependencies (#9205)
* update consul and vault dependencies

* update triton sign call accorting to https://github.com/joyent/triton-go/pull/135

* Delete readme.md

* put back github.com/mitchellh/reflectwalk to v1.0.0 and explain why

* fix/fixer_pp_docker_tag_tags.go: simplify deduplication loop and keep seen/stable order
2020-05-11 15:26:01 -04:00
Juan Manuel Mesa 75efe3fcd1
Clean linting issues for packages template and interpolate (#9120) 2020-04-27 10:35:47 +02:00
Juan Mesa e6dbbe454e Fix linting issues 2020-04-21 19:20:45 +02:00
Juan Mesa 0fca4a8ba8 Add template function aws_secretsmanager
Allow users to read secrets from AWS Secrets Manager via
packer template functions
2020-04-21 18:37:29 +02:00
Sylvia Moss 553b1fb9f8
Add RetriedProvisioner to allow retry provisioners (#9061) 2020-04-16 11:58:54 +02:00
Adrien Delorme 840e67f775
HCL2: when we see a map generate an attribute spec instead of a block spec (#9035)
* mapstructure-to-hcl2: when we see a map generate an attribute spec and not a block spec

this will alow to do

tags = {
 key = "value"
}

instead of

tags {
  key = "value"
}

This will also enable using variables directly for those tags

* generate code

* update tests
2020-04-14 16:05:13 +02:00
Jeff Escalante 9165be87da
packer.io prefix removal, html extension remove for in-code errors 2020-04-07 17:53:22 -04:00
Megan Marsh 0cfcbd66ca
execute_command in powershell wasn't interpolating build vars properly (#8771) 2020-03-02 04:20:32 -05:00
Wilken Rivera 9ec8b67392
Add golangci-lint to project (#8686)
* Add golangci-lint as linting tool

* Disable failing staticchecks to start; GitHub issue to handle coming soon

* Run `goimports -w` to repair all source files that have improperly
formatted imports

* makefile: Add ci-lint target to run on travis

This change adds a new make target for running golangci-lint on newly
added Go files only. This target is expected to run during Packer ci builds.

* .github/contributing: Add code linting instructions

* travis: Update job configuration to run parallel builds
2020-02-14 11:42:29 -05:00
Megan Marsh 391cea13ed
Merge pull request #8725 from hashicorp/fix_8655
Check if JSON template doesn't have duplicate configuration
2020-02-13 12:04:26 -08:00
Sylvia Moss 23fa3107a3
Render variables one time on prepare method (#8727) 2020-02-13 17:35:23 +01:00
Moss dc81720dc9 Fix format 2020-02-12 14:44:28 +01:00
Moss d654898ebf Add tests for check of json duplicate fields 2020-02-12 14:34:20 +01:00
Moss a6d90babbf Add check for json duplicate fields 2020-02-12 11:56:18 +01:00
Lars Lehtonen c6e6220fef
template: replace deprecated os.SEEK_SET with io.SeekStart 2020-02-05 08:56:59 -08:00
Sylvia Moss 328eb8ee96
Interpolate boot_command when defined by user variable (#8640) 2020-01-27 19:10:16 +01:00
Megan Marsh 8490bbc45c add tests for info sharing 2019-12-17 13:41:48 -08:00
Megan Marsh cd7abf1f9e clean up a few final issues with info sharing 2019-12-17 11:20:57 -08:00
Adrien Delorme 4b7132c87c Merge remote-tracking branch 'origin/master' into sharing_info 2019-12-17 11:57:09 +01:00
Adrien Delorme 0785c2f6fc
build using HCL2 (#8423)
This follows #8232 which added the code to generate the code required to parse
HCL files for each packer component.

All old config files of packer will keep on working the same. Packer takes one
argument. When a directory is passed, all files in the folder with a name
ending with  “.pkr.hcl” or “.pkr.json” will be parsed using the HCL2 format.
When a file ending with “.pkr.hcl” or “.pkr.json” is passed it will be parsed
using the HCL2 format. For every other case; the old packer style will be used.

## 1. the hcl2template pkg can create a packer.Build from a set of HCL (v2) files

I had to make the packer.coreBuild (which is our one and only packer.Build ) a public struct with public fields

## 2. Components interfaces get a new ConfigSpec Method to read a file from an HCL file.

  This is a breaking change for packer plugins.

a packer component can be a: builder/provisioner/post-processor

each component interface now gets a `ConfigSpec() hcldec.ObjectSpec`
which allows packer to tell what is the layout of the hcl2 config meant
to configure that specific component.

This ObjectSpec is sent through the wire (RPC) and a cty.Value is now
sent through the already existing configuration entrypoints:

 Provisioner.Prepare(raws ...interface{}) error
 Builder.Prepare(raws ...interface{}) ([]string, error)
 PostProcessor.Configure(raws ...interface{}) error

close #1768


Example hcl files:

```hcl
// file amazon-ebs-kms-key/run.pkr.hcl
build {
    sources = [
        "source.amazon-ebs.first",
    ]

    provisioner "shell" {
        inline = [
            "sleep 5"
        ]
    }

    post-processor "shell-local" {
        inline = [
            "sleep 5"
        ]
    }
}

// amazon-ebs-kms-key/source.pkr.hcl

source "amazon-ebs" "first" {

    ami_name = "hcl2-test"
    region = "us-east-1"
    instance_type = "t2.micro"

    kms_key_id = "c729958f-c6ba-44cd-ab39-35ab68ce0a6c"
    encrypt_boot = true
    source_ami_filter {
        filters {
          virtualization-type = "hvm"
          name =  "amzn-ami-hvm-????.??.?.????????-x86_64-gp2"
          root-device-type = "ebs"
        }
        most_recent = true
        owners = ["amazon"]
    }
    launch_block_device_mappings {
        device_name = "/dev/xvda"
        volume_size = 20
        volume_type = "gp2"
        delete_on_termination = "true"
    }
    launch_block_device_mappings {
        device_name = "/dev/xvdf"
        volume_size = 500
        volume_type = "gp2"
        delete_on_termination = true
        encrypted = true
    }

    ami_regions = ["eu-central-1"]
    run_tags {
        Name = "packer-solr-something"
        stack-name = "DevOps Tools"
    }
    
    communicator = "ssh"
    ssh_pty = true
    ssh_username = "ec2-user"
    associate_public_ip_address = true
}
```
2019-12-17 11:25:56 +01:00
Megan Marsh 6bc5a95c3a tweak docs; change to 2019-12-16 21:35:42 -08:00
Megan Marsh 0f6d1beccf add an extra string array to the Prepare() return values in the builder interfaces; this sets up the ability for builders to give the provisioners custom user-accessible build-time variables. 2019-12-16 21:23:05 -08:00
Megan Marsh d47f151bf9 punctuation 2019-12-14 03:38:27 -08:00
Megan Marsh 82367a88f8 reorganize placeholder data call to live with provisioner implementation; force users to use the generated function, therefore forcing validation, for all variables except winrmpassword, by doing a simple string check against the placeholder data. 2019-12-14 03:32:38 -08:00
Megan Marsh 39fd462b56 change all provision func signatures to use map[string]interface{} 2019-12-12 15:38:32 -08:00
Megan Marsh 0ca7c9f397 fix tests 2019-12-11 16:29:35 -08:00
Megan Marsh 5d266b20d0 generated func now validates based on data provided in context, which needs to include the future values if it is to be interpolated properly. 2019-12-11 15:43:38 -08:00
Megan Marsh 528b853178 stub out generated func for validation 2019-12-11 15:43:38 -08:00
Megan Marsh 14baa769ab improve error message 2019-10-23 12:02:31 -07:00
Adrien Delorme 078ba7c8c3 commit old code generation tool
for history
2019-10-15 12:56:42 +02:00
psinghal20 6492995a20
feat: add strftime function in templates
This commit adds strfttime function in templates to provide conventional format for ISO time based on C standard.

Signed-off-by: psinghal20 <psinghal20@gmail.com>
2019-10-11 14:35:39 +05:30
Megan Marsh 4c29f88a0a reuse code 2019-09-27 13:41:22 -07:00
Megan Marsh a2824a942d remove extraneous import 2019-09-25 14:43:32 -07:00
Megan Marsh c57e0a9837 remove logline 2019-09-25 13:55:16 -07:00
Megan Marsh 59efa0faee rename option to error-cleanup-privisoner, which I think is clearer 2019-09-25 13:43:29 -07:00
Megan Marsh 0683bc409b add test for error-cleanup provisioner, and fix tests by fixing null builder to use an actual none communicator instead of skipping communicator generation altogether 2019-09-25 13:39:09 -07:00