Commit Graph

904 Commits

Author SHA1 Message Date
Megan Marsh d66b358100
Merge pull request #9991 from hashicorp/b-amazon-ssm-infinite-loop
amazon/ssm_driver: Update log polling logic
2020-09-24 09:55:39 -07:00
Wilken Rivera 2f2b5683a2 amazon/ssm_driver: Update log polling logic
This change checks for closed iochans related to the log polling
function and will break out of the loop when both channels are closed.
Which is an indicator that the SSM session has been terminated by some
external process. This fixes an issue where Packer hangs, due to the
infinite loop, when an SSM session is killed outside of Packer.

Related to #9442
2020-09-24 10:19:37 -04:00
Megan Marsh 429d431365 Implement assume_role for Packer. This functionality is copied shamelessly
from the Terraform AWS provider
2020-09-23 13:18:26 -07:00
Sylvia Moss d347bbec4c
Retry fleet creating only on invalid IAM instance profile (#9946) 2020-09-18 17:05:36 +02:00
Adrien Delorme 0d0bd9ce75
name fields of PolicyDocument correctly in HCL (#9812)
withouth this fix we would have had to do

```hcl
  temporary_iam_instance_profile_policy_document {
    statement {
      action   = ["*"]
      effect   = "Allow"
      resource = ["*"]
    }
    version = "2012-10-17"
  }
```

instead of the same document but with capitalised fields
2020-08-25 10:53:56 +02:00
Adrien Delorme a0c09e85df
retry spot instance creation when an "Invalid IAM Instance Profile name" error pops up (#9810)
PutRolePolicy & AddRoleToInstanceProfile are eventually consistent but it is not possible to wait for them to be done here: 0785c2f6fc/builder/amazon/common/step_iam_instance_profile.go (L117-L134) which was causing the `CreateFleet` to fail (100% for me). So for now we retry a bit later. Waiting 5 seconds after the previously linked code also fixed this.

Test file:

```json
{
	"builders": [
		{
			"type": "amazon-ebs",
			"region": "eu-west-1",
			"ami_name": "ubuntu-16.04 test {{timestamp}}",
			"ami_description": "Ubuntu 16.04 LTS - expand root partition",
			"source_ami_filter": {
				"filters": {
					"virtualization-type": "hvm",
					"name": "ubuntu/images/*/ubuntu-xenial-16.04-amd64-server-*",
					"root-device-type": "ebs"
				},
				"owners": [
					"099720109477"
				],
				"most_recent": true
			},
			"spot_price": "0.03",
			"spot_instance_types": [
				"t2.small"
			],
			"encrypt_boot": true,
			"ssh_username": "ubuntu",
			"ssh_interface": "session_manager",
			"temporary_iam_instance_profile_policy_document": {
				"Version": "2012-10-17",
				"Statement": [
					{
						"Effect": "Allow",
						"Action": [
							"*"
						],
						"Resource": "*"
					}
				]
			},
			"communicator": "ssh"
		}
]}
```
2020-08-25 10:10:32 +02:00
sylviamoss 1a0d5a93ee fix max_attempts typo 2020-08-17 17:58:22 +02:00
sylviamoss 788dc32598 add aws_polling config option to override env variables 2020-08-17 17:15:32 +02:00
Megan Marsh 8b4993e44c
fix docs for kms key ids (#9766) 2020-08-14 11:35:35 +02:00
Megan Marsh 54469c4728
Add all of the custom AWS template engines to GeneratedData for use b… (#9751) 2020-08-12 10:43:15 +02:00
Megan Marsh f9df185ef1
Merge pull request #9579 from hashicorp/fix_9562
wrap CreateImage call in a retry to account for eventual consistencey…
2020-07-22 08:43:10 -07:00
Megan Marsh a56942d3c7 change isAwsErr to an exported func so I can use it in other aws modules. 2020-07-21 15:30:45 -07:00
Adrien Delorme 810f161d22 builder/amazon/common/block_device.go: replace tabs with words 2020-07-20 16:34:13 +02:00
Adrien Delorme 966b17d2b5 builder/amazon/common/access_config.go: replace tabs with words 2020-07-20 16:31:23 +02:00
Adrien Delorme 5c4901c2c2 builder/amazon/common/run_config.go: replace tabs with words 2020-07-20 16:28:14 +02:00
Adrien Delorme 1455a29f2b remove tabs for security_group_filter too 2020-07-20 16:24:47 +02:00
Adrien Delorme 191094edcc better `disable_stop_instance` docs 2020-07-20 16:21:43 +02:00
Megan Marsh 530f5e3f63 more additions to partials 2020-07-17 15:52:11 -07:00
Megan Marsh a1651977b3 add tabs to generated docs from access_config and run_config structs 2020-07-17 15:10:55 -07:00
Megan Marsh 4d3b2cea96
Merge pull request #9480 from adongy/master
aws: retry fetching blockdevicemapping if empty
2020-07-08 12:32:12 -07:00
Megan Marsh a885e620ee don't error for no mappings if no mappings were set. 2020-07-08 11:55:56 -07:00
Megan Marsh 122e80f591 generated code 2020-06-30 10:12:50 -07:00
Megan Marsh 0122530a07 make statement an array of strings 2020-06-30 10:01:49 -07:00
Anthony Dong 2322d3ae6b aws: retry fetching blockdevicemapping if empty 2020-06-25 10:48:24 +02:00
Wilken Rivera a8dfb66d6c docs/builder/amazon: Add WinRM unsupported note to Session Manager docs
Closes #9451

Co-authored-by: Adrien Delorme <azr@users.noreply.github.com>
2020-06-23 07:41:21 -04:00
Adrien Delorme 40947c2bf6
HCL2: generate hcl tags with go-cty tags too (#9306)
This will allow to generate the config files:

```go
package main

import (
	"fmt"

	"github.com/hashicorp/hcl/v2/gohcl"
	"github.com/hashicorp/hcl/v2/hclwrite"
	"github.com/hashicorp/packer/builder/alicloud/ecs"
)

func main() {
	name := "name"
	app := ecs.FlatConfig{
		AlicloudImageName: &name,
		ECSSystemDiskMapping: &ecs.FlatAlicloudDiskDevice{
			DiskName: &name,
		},
	}

	f := hclwrite.NewEmptyFile()
	block := gohcl.EncodeAsBlock(&app, `source "something" "something"`)
	f.Body().AppendBlock(block)
	fmt.Printf("%s", f.Bytes())
}

```

Will output:

```
source "something" "something" {
  packer_user_variables      = null
  packer_sensitive_variables = null
  image_name                 = "name"
  image_share_account        = null
  image_unshare_account      = null
  image_copy_regions         = null
  image_copy_names           = null
  tags                       = null
  tag                        = null
  system_disk_mapping        = { disk_category = null, disk_delete_with_instance = null, disk_descri
ption = null, disk_device = null, disk_encrypted = null, disk_name = "name", disk_size = null, disk_
snapshot_id = null }
  image_disk_mappings        = null
  ssh_remote_tunnels         = null
  ssh_local_tunnels          = null
  ssh_public_key             = null
  ssh_private_key            = null
}
```

This is a good first step for #9015 and #9282 

fix #9304
2020-05-28 11:19:00 +02:00
nitrocode 6d29ec956b Added SourceAMICreationDate 2020-05-26 15:04:03 -04:00
Wilken Rivera b5639d4697
tests/builder/amazon: Enhance the test coverage for SSM Tunnel related steps (#9213)
* test/builder/amazon: Update mocks to properly test SSM tunnel and driver code base

* Update mod files
2020-05-13 10:10:55 -04:00
Wilken Rivera 6f5f6bf999 builder/amazon Fix invalid pointer issue for non SSMAgengtEnabled builds
Tests before change
```
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x0 pc=0x1392ca2]

goroutine 299 [running]:
github.com/hashicorp/packer/builder/amazon/common.(*StepCreateSSMTunnel).Cleanup(0xc0003dc460, 0x4d1a4c0, 0xc0006e9800)
        /home/wilken/Development/packer/builder/amazon/common/step_create_ssm_tunnel.go:95 +0xf2
github.com/hashicorp/packer/helper/multistep.(*BasicRunner).Run(0xc0006e98f0, 0x4d408c0, 0xc00065fcc0, 0x4d1a4c0, 0xc0006e9800)
        /home/wilken/Development/packer/helper/multistep/basic_runner.go:79 +0x2c6
github.com/hashicorp/packer/builder/amazon/ebs.(*Builder).Run(0xc000726800, 0x4d408c0, 0xc00065fcc0, 0x4d5e300, 0xc0006e8d80, 0x4cc7220, 0xc000434120, 0x0, 0x0, 0x0, ...)
        /home/wilken/Development/packer/builder/amazon/ebs/builder.go:330 +0x17e2
github.com/hashicorp/packer/packer.(*CoreBuild).Run(0xc000720500, 0x4d408c0, 0xc00065fcc0, 0x4d5e180, 0xc0006fe510, 0x0, 0x0, 0x0, 0x0, 0x0)
        /home/wilken/Development/packer/packer/build.go:287 +0x7ef
github.com/hashicorp/packer/command.(*BuildCommand).RunContext.func1(0xc0004d14d0, 0xc0003dc3c0, 0xc000441500, 0xa, 0x4d5e1e0, 0xc000720500, 0x4d408c0, 0xc00065fcc0, 0x4d5e180, 0xc0006fe510, ...)
        /home/wilken/Development/packer/command/build.go:290 +0x189
created by github.com/hashicorp/packer/command.(*BuildCommand).RunContext
        /home/wilken/Development/packer/command/build.go:284 +0xd5a
FAIL    github.com/hashicorp/packer/provisioner/shell   188.335s
FAIL
```

Test After change
```
--- PASS: TestShellProvisioner (212.39s)
    --- PASS: TestShellProvisioner/testing_amazon-ebs_builder_against_shell_provisioner (212.39s)
PASS

```
2020-05-12 14:49:57 -04:00
Wilken Rivera 89fb7bb080 Apply suggestions from code review
Co-authored-by: Megan Marsh <megan@hashicorp.com>
2020-05-08 09:36:43 -04:00
Wilken Rivera 056f1f6e76 docs/amazon/builder: Add not about using a temporary_iam_instance_profile_policy_document 2020-05-08 09:32:54 -04:00
Wilken Rivera bf1da57a05 Update address to localhost to be consistent with other proxies 2020-05-08 09:32:54 -04:00
Wilken Rivera 7cea7466ba buider/amazon: Add `session_manager_port` option for setting a custom local port 2020-05-08 09:32:54 -04:00
Wilken Rivera 0d13c634b7 Update logging constructs for Session Manger driver 2020-05-08 09:32:49 -04:00
Wilken Rivera 81c40b8d08 Add section to docs about the session-manager-plugin tunnel 2020-05-08 09:32:49 -04:00
Wilken Rivera 5f68303504 Remove no longer needed port check
The communicator code base will default to the correct port settings for
valid communicators and fail the build for non valid communicators.
2020-05-08 09:32:49 -04:00
Wilken Rivera 3ae7ab994b step_create_ssm_tunnel: Add port availability check for LocalPortNumber 2020-05-08 09:32:49 -04:00
Wilken Rivera 08dc2cb847 Remove call to unstubbed session start 2020-05-08 09:32:49 -04:00
Wilken Rivera f498866c3a Update validation for session_manager connectivity
Validation will now check that either a IamInstanceProfile or a TemporaryIamInstanceProfilePolicyDocument is set
2020-05-08 09:32:49 -04:00
Wilken Rivera 2f09b74587 amazon/builder: Move port validation for security group into RunConfig.Prepare 2020-05-08 09:32:48 -04:00
Wilken Rivera 743df19af2 Add tests for driver and ssm tunnel step 2020-05-08 09:32:48 -04:00
Wilken Rivera e53d6aea66 amazon/ssm_driver: Update to support command context
* Defer the release of the found port to prevent possible Packer build collisions.
2020-05-08 09:32:48 -04:00
Wilken Rivera 5e14089851 tests/builder/amazon: Add basic testing for session_manager
* Add a test to ensure the proper host is returned when using session_manager
* Add a basic driver test for the validating that the SSMDriver will fail if the plugin is not in the path.
2020-05-08 09:32:48 -04:00
Wilken Rivera 60e0ba7e68 Update iam instance profile name 2020-05-08 09:32:48 -04:00
Wilken Rivera e515aefee3 Add validation checks for session_manager
* Session manager connectivity only supported via SSH
* Session manager requires the use of an iam instance profile with system manager permissions
2020-05-08 09:32:48 -04:00
Wilken Rivera c1d2477d18 builder/amzon: Update port configuration for WinRM
Connectivity for WinRM works for the initial connection, but fails to upload any provisioning scripts - need to dive in deeper.
For now connectivity over SSH works as expected.
2020-05-08 09:32:48 -04:00
Wilken Rivera fbffbd1899 docs/builder/amazon: Add short blurb for connecting via the AWS session_manager interface 2020-05-08 09:32:48 -04:00
Wilken Rivera 520061dee6 builder/amazon: Update Session Manger connectivity
* Update security group creation step skip ingress rules if using session manager
* Update create ssm tunnel step to dynamically set a session port
* Add SSHPort function to common to return session-manager tunnel port
* Update SSHHost to return proper host for session-manager
2020-05-08 09:32:48 -04:00
Wilken Rivera 3dd46eb5f4 Add PoC of connectivity using session-manager-plugin
* Add a bunch of hard coded values for testing on port 8081; ssh configs
are set to localhost and 8081
* Add a base drive for communicating with the session manager plugin
* Update step for creating tunnel to actually create SSM session tunnel via driver
2020-05-08 09:32:48 -04:00
Wilken Rivera 16604373ac initial ssm tunnel changes - not working 2020-05-08 09:32:48 -04:00
Megan Marsh f0d2680426
remove spot_price_auto_product from docs (#9173) 2020-05-05 20:48:27 -04:00
Adrien Delorme 73242e8de4 make sure keyvalue and namevalue filters are using the same naming as the service being used 2020-04-16 17:22:13 +02:00
Adrien Delorme 840e67f775
HCL2: when we see a map generate an attribute spec instead of a block spec (#9035)
* mapstructure-to-hcl2: when we see a map generate an attribute spec and not a block spec

this will alow to do

tags = {
 key = "value"
}

instead of

tags {
  key = "value"
}

This will also enable using variables directly for those tags

* generate code

* update tests
2020-04-14 16:05:13 +02:00
Jeff Escalante 9165be87da
packer.io prefix removal, html extension remove for in-code errors 2020-04-07 17:53:22 -04:00
Jeff Escalante 4afd107eeb
patch source files for extensions changes 2020-04-07 17:50:51 -04:00
Megan Marsh 8ae6256c03 add test of NoEphemeral 2020-03-24 09:10:31 -07:00
Megan Marsh fc36a37db8 only set NoDevice if NoEphemeral is set; otherwise, legit block device mappings get destroyed 2020-03-24 08:51:32 -07:00
Adrien Delorme fab344a1fc Update ami_config.go
quote string  in err
2020-03-17 15:41:58 +01:00
Adrien Delorme b5e8750f5f Change key/value to name/value to remain consistent with Terraform 2020-03-17 11:23:11 +01:00
Adrien Delorme 3cae1a81ef up tests 2020-03-16 17:54:52 +01:00
Adrien Delorme d532bb376c wip 2020-03-16 17:47:44 +01:00
Adrien Delorme c845436e32 KeyValues.CopyOn: make sure a receiving nil map is set too 2020-03-16 15:46:08 +01:00
Adrien Delorme e1c84806cc rewrap 2020-03-16 14:55:03 +01:00
Adrien Delorme fc837e3698 Update run_config.go 2020-03-16 12:36:20 +01:00
Adrien Delorme ce8d3d3747 wip 2020-03-16 12:35:25 +01:00
Adrien Delorme dab689ce2e WIP 2020-03-16 12:19:34 +01:00
Adrien Delorme dbb4981d66 WIP 2020-03-16 12:13:05 +01:00
Adrien Delorme 6e6490d688 WIP 2020-03-13 18:04:48 +01:00
Adrien Delorme 260dc59500 use the hcl2template.KVFilter where we can 2020-03-13 17:17:46 +01:00
Wilken Rivera 7b705545d9
docs: Various doc clean-ups to fix code block formatting and syntax highlighting (#8868) 2020-03-12 15:05:08 +01:00
Megan Marsh 4133acbdf3 update spot price docs 2020-03-09 08:46:30 -07:00
Wilken Rivera e1a46ec293
Fix gosimple S1025 linting errors (#8838)
Remove unneeded use of fmt.Sprintf for variables that are already strings.
2020-03-04 15:31:30 -05:00
Megan Marsh dde74232f2
Clearly document all template-engine-enabled options (#8770) 2020-02-20 11:05:27 +01:00
Wilken Rivera 9ec8b67392
Add golangci-lint to project (#8686)
* Add golangci-lint as linting tool

* Disable failing staticchecks to start; GitHub issue to handle coming soon

* Run `goimports -w` to repair all source files that have improperly
formatted imports

* makefile: Add ci-lint target to run on travis

This change adds a new make target for running golangci-lint on newly
added Go files only. This target is expected to run during Packer ci builds.

* .github/contributing: Add code linting instructions

* travis: Update job configuration to run parallel builds
2020-02-14 11:42:29 -05:00
Adrien Delorme 7d5f0c11af add max_retries option to aws builders 2020-02-07 16:36:14 +01:00
Wilken Rivera a1d9ba0e32
Merge pull request #8699 from fly1028/fix-instance-running-timeout
Instance Running waiter not aware of global waiter settings
2020-02-06 17:18:59 -05:00
Patrick Ng e5e5fd1198 Instance Running waiter not aware of global waiter settings 2020-02-05 15:09:09 -08:00
Megan Marsh d252a24b82 allow user to set winrm_host override like we do ssh_host override 2020-01-30 15:22:22 -08:00
Sylvia Moss dc31bad539
Sharing info with post-processors via artifact (#8632) 2020-01-30 11:27:58 +01:00
nywilken d2a9e6b32e Format code with `gofmt -s -w` 2020-01-28 16:52:50 -05:00
Wilken Rivera 111bab86d9
Merge pull request #8622 from datalot/pre-validate-with-filters
Allow aws builder pre validation to pass when subnet filters are present
2020-01-22 15:00:44 -05:00
Sylvia Moss 0b7251a4bb
Share .Device and .MountPath between builders, provisioners and post-processors (#8621) 2020-01-20 16:29:38 +01:00
Rich Cavanaugh 4fb0de5256 allow validation to pass when subnet filters are present 2020-01-17 11:57:42 -05:00
Sylvia Moss 39c25b2c66
Add retry mechanism to amazon DeleteSnapshot (#8614) 2020-01-16 14:38:46 +01:00
Sylvia Moss 0677b02e18
Share SourceImageName with provisioners and manifest post-processor (#8603) 2020-01-16 12:04:03 +01:00
Sylvia Moss 0e177915f8
Add retry mechanism to DeregisterImage from awsec2 (#8598) 2020-01-15 10:28:14 +01:00
Megan Marsh b7ee807cf1
Merge pull request #8571 from hashicorp/fix_8565
fix ssh interface docs for amazon builders
2020-01-06 14:34:40 -08:00
Megan Marsh 6655f93b59 fix typo and regenerate code 2020-01-06 14:12:29 -08:00
Megan Marsh 76251e5fd2 fix ssh interface docs for amazon builders 2020-01-06 12:01:30 -08:00
Trevor Pounds 47e48ebc24 Add source AMI owner ID/name to build template. 2019-12-30 14:04:49 -05:00
Adrien Delorme 4b7132c87c Merge remote-tracking branch 'origin/master' into sharing_info 2019-12-17 11:57:09 +01:00
Adrien Delorme 0785c2f6fc
build using HCL2 (#8423)
This follows #8232 which added the code to generate the code required to parse
HCL files for each packer component.

All old config files of packer will keep on working the same. Packer takes one
argument. When a directory is passed, all files in the folder with a name
ending with  “.pkr.hcl” or “.pkr.json” will be parsed using the HCL2 format.
When a file ending with “.pkr.hcl” or “.pkr.json” is passed it will be parsed
using the HCL2 format. For every other case; the old packer style will be used.

## 1. the hcl2template pkg can create a packer.Build from a set of HCL (v2) files

I had to make the packer.coreBuild (which is our one and only packer.Build ) a public struct with public fields

## 2. Components interfaces get a new ConfigSpec Method to read a file from an HCL file.

  This is a breaking change for packer plugins.

a packer component can be a: builder/provisioner/post-processor

each component interface now gets a `ConfigSpec() hcldec.ObjectSpec`
which allows packer to tell what is the layout of the hcl2 config meant
to configure that specific component.

This ObjectSpec is sent through the wire (RPC) and a cty.Value is now
sent through the already existing configuration entrypoints:

 Provisioner.Prepare(raws ...interface{}) error
 Builder.Prepare(raws ...interface{}) ([]string, error)
 PostProcessor.Configure(raws ...interface{}) error

close #1768


Example hcl files:

```hcl
// file amazon-ebs-kms-key/run.pkr.hcl
build {
    sources = [
        "source.amazon-ebs.first",
    ]

    provisioner "shell" {
        inline = [
            "sleep 5"
        ]
    }

    post-processor "shell-local" {
        inline = [
            "sleep 5"
        ]
    }
}

// amazon-ebs-kms-key/source.pkr.hcl

source "amazon-ebs" "first" {

    ami_name = "hcl2-test"
    region = "us-east-1"
    instance_type = "t2.micro"

    kms_key_id = "c729958f-c6ba-44cd-ab39-35ab68ce0a6c"
    encrypt_boot = true
    source_ami_filter {
        filters {
          virtualization-type = "hvm"
          name =  "amzn-ami-hvm-????.??.?.????????-x86_64-gp2"
          root-device-type = "ebs"
        }
        most_recent = true
        owners = ["amazon"]
    }
    launch_block_device_mappings {
        device_name = "/dev/xvda"
        volume_size = 20
        volume_type = "gp2"
        delete_on_termination = "true"
    }
    launch_block_device_mappings {
        device_name = "/dev/xvdf"
        volume_size = 500
        volume_type = "gp2"
        delete_on_termination = true
        encrypted = true
    }

    ami_regions = ["eu-central-1"]
    run_tags {
        Name = "packer-solr-something"
        stack-name = "DevOps Tools"
    }
    
    communicator = "ssh"
    ssh_pty = true
    ssh_username = "ec2-user"
    associate_public_ip_address = true
}
```
2019-12-17 11:25:56 +01:00
Megan Marsh 84063d2132 implement ID and Type values in the provisioner template info sharing 2019-12-13 11:57:01 -08:00
Megan Marsh 6f418d0e54 get data sharing to a working state with the powershell provisioner 2019-12-11 15:43:38 -08:00
Megan Marsh 601007e3e2 pas data into provisioners well 2019-12-11 15:43:38 -08:00
Oded Priva 5e81c6f44e added the option to skip IAM instance profile validation (#8457) 2019-12-10 09:17:38 +01:00
Moss 28c720bd3a Organize imports 2019-12-02 10:17:36 +01:00
Moss 250643d54b Remove 5 sec sleep after adding iam instance profile 2019-12-02 10:13:26 +01:00
Moss 4516978555 Fix request retry mechanism to run aws instance 2019-11-29 18:11:10 +01:00
Jon Allie c31bbcf4cf builder/amazon: AmiFilterOptions.GetOwners: avoid taking the address of a loop iterator var (#8417)
GetOwners() always returned a slice of pointers to the last value. Because slice ranging reuses the same local variable.
2019-11-27 12:28:07 +01:00