Commit Graph

264 Commits

Author SHA1 Message Date
nywilken f9f4726eff builder/amazon/step_pre_validate: Add check for non-default VPCs
Subnet information is only really needed when the specified `vpc_id` is
not the default VPC for the region where the builder is being executed.
This change uses the AWS API to determine if the VPC provided is a
non-default VPC and only validates the existence of a `subnet_id` if a
user has provided a non-default `vpc_id`.

Tests after change
```
> make test TEST=./builder/amazon/... TESTARGS='-count=1 -v -run=TestStepPreValidate_checkVpc'
...

=== RUN   TestStepPreValidate_checkVpc
=== RUN   TestStepPreValidate_checkVpc/DefaultVpc
=== RUN   TestStepPreValidate_checkVpc/NonDefaultVpcNoSubnet
=== RUN   TestStepPreValidate_checkVpc/NonDefaultVpcWithSubnet
=== RUN   TestStepPreValidate_checkVpc/SubnetWithNoVpc
=== RUN   TestStepPreValidate_checkVpc/NoVpcInformation
--- PASS: TestStepPreValidate_checkVpc (0.00s)
    --- PASS: TestStepPreValidate_checkVpc/DefaultVpc (0.00s)
    --- PASS: TestStepPreValidate_checkVpc/NonDefaultVpcNoSubnet (0.00s)
    --- PASS: TestStepPreValidate_checkVpc/NonDefaultVpcWithSubnet (0.00s)
    --- PASS: TestStepPreValidate_checkVpc/SubnetWithNoVpc (0.00s)
    --- PASS: TestStepPreValidate_checkVpc/NoVpcInformation (0.00s)
PASS

...
```
2019-11-18 16:44:09 -05:00
Megan Marsh d15f43e4eb
Merge pull request #8232 from hashicorp/artifact-tree
HCL2 configs - part one
2019-10-22 11:54:34 -07:00
Bernard Baltrusaitis 8b83cca064 adding temporary instance profile document policy 2019-10-19 20:56:18 +11:00
Adrien Delorme 078ba7c8c3 commit old code generation tool
for history
2019-10-15 12:56:42 +02:00
Megan Marsh 0cbd3ff0f3 make sure amazon builders respect ssh_host option 2019-09-26 12:30:04 -07:00
Adrien Delorme 3fe9d52e3f Merge remote-tracking branch 'origin/master' into scrape_doc_to_builder_struct_config 2019-09-20 11:15:44 +02:00
Megan Marsh 1a6adadb89 convert EnableAMIENASupport to trilean in code 2019-08-22 13:18:22 -07:00
Adrien Delorme 4cb7c30987 Merge remote-tracking branch 'origin/master' into scrape_doc_to_builder_struct_config 2019-08-21 16:44:23 +02:00
Megan Marsh 6ad975f296 Revert "move SSHInterface/SSHIPVersion fields to communitator.Config struct"
This reverts commit b0c09087a2.
2019-07-02 10:46:10 -07:00
Megan Marsh c722afe707 stop calculating spot bids; amazon changed the way spot instances are priced to be stable rather than bid-based, so when user sets auto, we can just default to the ondemand price and know that they'll get the same price as everyone else bidding at that time. 2019-07-01 13:01:41 -07:00
Adrien Delorme 9f4c8fb321 document ebs builder using partials 2019-06-18 17:41:50 +02:00
Adrien Delorme ff2e0298ba simplify block devices utilisation by removing intermediary types 2019-06-18 12:37:47 +02:00
Megan Marsh 0d55bc46ee add skip_save_build_region flag to fix naming conflicts when building for multiple regions 2019-06-17 14:39:11 -07:00
Adrien Delorme 8e857d64f1 aws: rewrap struct comments for documentation generation 2019-06-06 17:34:17 +02:00
Megan Marsh 708165eaa8 refactoring step_run_spot_instance and swapping out the calls to create a spot instance with a call to create a spot fleet of one, so that we can be more flexible with the instance type when desired 2019-05-24 09:32:54 -07:00
Megan Marsh 2c352cc681 add tests for step_ami_region_copy 2019-05-03 14:47:09 -07:00
Megan Marsh 201c8c1332 make sure that you encrypt using the provided kms key id and not the default 2019-04-15 16:44:56 -07:00
Adrien Delorme a4bf94dd3c change Builder to be passed a context for cancellation
we have to to give it to our hook
2019-04-03 15:55:55 +02:00
Akshat Mahajan f07e4214cc Addresses issues #5384, #5494: Rename and change `temporary_security_group_source_cidr`
to accept a list of strings (for Amazon builders).

Per this change, `temporary_security_group_source_cidr` in the configuration:

1. Will be renamed to `temporary_security_group_source_cidrs`.
2. Will accept a list of CIDRs.
3. Will have its documentation updated to reflect this change.
4. Will have a fixer attached for newer templates to avail of.
2019-03-30 15:47:03 -07:00
Adrien Delorme 9f82b75e57 Use the hashicorp/go-getter to download files
* removed packer.Cache and references since packer.Cache is never used except in the download step. The download step now uses the new func packer.CachePath(targetPath) for this, the behavior is the same.
* removed download code from packer that was reimplemented into the go-getter library: progress bar, http download restart, checksuming from file, skip already downloaded files, symlinking, make a download cancellable by context.
* on windows if packer is running without symlinking rights and we are getting a local file, the file will be copied instead to avoid errors.
* added unit tests for step_download that are now CI tested on windows, mac & linux.
* files are now downloaded under cache dir `sha1(filename + "?checksum=" + checksum) + file_extension`
* since the output dir is based on the source url and the checksum, when the checksum fails, the file is auto deleted.
* a download file is protected and locked by a file lock,
* updated docs
* updated go modules and vendors
2019-03-13 12:11:58 +01:00
Megan Marsh 3ed9fe6dee remove unused imports
remove HttpClientWithEnvironmentProxy from amazon builders; let access config handle it
2019-03-04 16:42:33 -08:00
Megan Marsh 0ed57a37a3 dont overwrite session config in ebs builder 2019-03-04 16:41:31 -08:00
Megan Marsh ec3b0a5138 fix conflict from mergeing too many things to master at once 2019-03-01 11:13:32 -08:00
Megan Marsh 3b726e66c1 swap region copy and encryption steps 2019-02-25 11:03:03 -08:00
Megan Marsh 3704a053d0 move region validation and credential wait into step pre validate 2019-02-05 14:07:04 -08:00
Megan Marsh 035c3506ba manually set proxyfromenvironment in default http client for ec2 sessions 2019-01-25 12:33:25 -08:00
Megan Marsh fbb3d526e3 move region validation to run so that we don't break validation when we don't have aws creds set 2018-11-20 15:01:28 -08:00
Megan Marsh 9573013d3a
Merge pull request #6872 from aspectcapital/disable-ena-support
Add support to explicitly disable ENA support
2018-10-19 10:54:38 -07:00
Matt Dainty feb8067c7d Convert `ena_support` to a pointer
This means it now has three states, `true`, `false`, & `nil`. The
default state is now `nil` which does nothing instead of `false` which
now will explicitly disable ENA support instead of just not enabling it.
2018-10-19 12:40:43 +01:00
Megan Marsh 57f87f58b9
Merge pull request #6374 from rickard-von-essen/aws-filters
builder/amazon: Add suppport for vpc_filter, subnet_filter, and security_group_filter
2018-10-18 14:34:02 -07:00
Adrien Delorme 22d3e9b5c4 builder/amazon/ebs: pass config as a pointer so values could be populated by a run 2018-09-27 16:27:34 +02:00
Matthew Hooker 9181570bc3
remove stray struct dumps in logs 2018-09-27 00:14:05 -07:00
Megan Marsh 38cc525ec7 new option allowing user to clean up the ephemeral ssh key from the authorized_keys file 2018-09-14 11:06:38 -07:00
Adrien Delorme 2e73ed8d46 pass Comm instance to fix Spot instance creation 2018-09-11 18:20:14 -07:00
Adrien Delorme e02d0dacc4
Merge branch 'master' into aws-filters 2018-09-07 16:14:00 +02:00
Megan Marsh b94e6e913e
Merge pull request #6670 from sargun/fix-ena-on-pv
Allow users to build hvm images from pv images to have ena_support
2018-09-06 12:50:46 -07:00
Sargun Dhillon 902497d6ce Allow users to build hvm images from pv images to have ena_support 2018-09-04 18:16:52 -07:00
RenaudS 26aab49aaf aws spot instances: add block_duration_minutes option (#6638)
* Add block_duration_minutes option

* int64 cannot be nil

* Update doc

* Fix formating
2018-09-03 08:33:58 +02:00
Adrien Delorme df9e756865 pass the Comm instance to the amazon ebs builders steps 2018-08-30 12:50:29 +02:00
Adrien Delorme b0c09087a2 move SSHInterface/SSHIPVersion fields to communitator.Config struct 2018-08-29 14:51:28 +02:00
Adrien Delorme ef4ca9c48e builder.amazon: use c.Comm for ssh 2018-08-29 14:40:33 +02:00
Adrien Delorme 293c485372 builder.aws: use SSHAgentAuth from Comm 2018-08-29 14:40:33 +02:00
Adrien Delorme ef08c441c9 builder.aws: use Comm field for StepKeyPair & StepGetPassword 2018-08-29 14:40:33 +02:00
Adrien Delorme 51d2aac9f6 SSHPrivateKey => SSHPrivateKeyFile 2018-08-29 14:40:32 +02:00
Matthew Hooker e41e99954d
go 1.11 format rules 2018-08-24 15:56:44 -07:00
Rickard von Essen 533967cb66 builder/amazon: Add suppport for security_group_filter
Allow selecting SG's using a filter. Also some cleanup of vpc_filter and
subnet_filter.
2018-08-24 12:42:38 +02:00
Megan Marsh 340363a649
Merge pull request #6613 from hashicorp/merge_ssh_config_funcs
Merge ssh config funcs from builders
2018-08-23 16:03:36 -07:00
Rickard von Essen 9840862757 builder/amazon: Add suppport for vpc_filter and subnet_filter
First step of adding support for discovering VPC's and Subnets using
filters.
2018-08-23 12:52:03 +02:00
Adrien Delorme 41f6e0334d refactor all copy pasted sshConfig with into communicator.Config.SSHConfigFunc
* still need to append the auth methods into an array for gracefullness
2018-08-22 17:02:23 +02:00
Megan Marsh ff6a039d5b replace scrubconfig with packer.LogSecretFilter.Set
filter winrm password from logs
Add new root-level packer template option, sensitive-variables, to tell us what user variables to mark sensitive.
2018-08-20 15:35:55 -07:00
Matthew Hooker 31fcfe4bc2 PoC for filtering logs 2018-08-20 15:20:21 -07:00
Anshul Sharma c1edcd3774
amazon-ebssurrogate clean up volumes 2018-07-26 09:38:59 +03:00
Mark Meyer 3dbf1cb371 Enable tagging of spot requests
This adds a new parameter to the EBS builders named `spot_tags'. This
parameter accepts a map of tags, much like `tags'. These tags will be
applied to a spot request that is created.

Improve visibility.
2018-07-07 16:06:51 +02:00
DanHam d5304a25e9
Pass T2 Unlimited settings to run instance step for appropriate EC2 builders 2018-05-15 12:25:06 +01:00
Megan Marsh 3afb243f11 use build name to ensure that winrm password and other shared state is not overwritten if two builders need the password in the same packer run. 2018-04-16 13:41:17 -07:00
Matthew Hooker 21812fa17f
Add volume and run tags if in us-gov/china
We can't tag on instance creation when we're in "restricted" regions,
so let's add the tags after the resources have been created.

Adds methods to AccessConfig to detect if we're in China or US Gov
regions (i.e. "restricted").

Also turns tag:tag maps into a type, and moves methods around validating
and converting them to ec2Tags to methods of the type.
2018-02-05 16:39:20 -08:00
Matthew Hooker 366dc3da0a
move multistep imports to helper.
gomvpkg -from "github.com/mitchellh/multistep" -to "github.com/hashicorp/packer/helper/multistep"
2018-01-24 17:09:15 -08:00
Matthew Hooker f3dea272d9
Set session in state bag
We can construct connections to arbitrary regions from it
2017-12-19 11:32:33 -08:00
Matthew Hooker f6c401a143
don't pass in accessconfig 2017-12-19 11:32:32 -08:00
Rong Chen 3e92b1374e issue5606: follow the convention to use AccessConfig to create new aws session for step_create_tags etc. 2017-12-19 10:34:12 -08:00
Rickard von Essen 63f791570a
Merge pull request #5665 from kwilczynski/remove-sts-token-from-logging
amazon: Remove Session Token (STS) from being shown in the log.
2017-12-10 06:50:42 +01:00
Matthew Hooker f216330ba3
spot instance ena/sriov checking for all other builders 2017-12-08 14:56:19 -08:00
Matthew Hooker 4b1d8e3fe8
don't allow enhanced networking flags for spot instances. 2017-12-08 13:38:49 -08:00
Krzysztof Wilczynski 05327b7524
amazon: Remove Session Token (STS) from being shown in the log.
This commit adds a change which ensures that the Session Token
config struct item is removed from log output.

Signed-off-by: Krzysztof Wilczynski <kw@linux.com>
2017-12-04 22:14:38 +01:00
John Davies-Colley 707ec675b2 ssh interface for amazon builders 🔨 2017-11-22 15:49:38 +13:00
Matthew Hooker 95e4ae251e
WIP 2017-11-02 09:03:44 -07:00
Matthew Hooker c65fa8490d
fix various bugs deregistering AMIs
always deregister ami in session region
validate that session region does not appear in ami_regions
2017-10-30 14:17:19 -07:00
Matthew Hooker f7e269945e
add cidr validation and rename option 2017-10-12 17:05:56 -07:00
Matthew Hooker c74be87187 Merge pull request #5384 from dave2/sg-source-cidr
[WIP] Allow temporary security group to have source CIDR block explicit
2017-10-12 17:05:45 -07:00
Mark Meyer dbb3c76032 Remove redundant step and its usage
StepTagEBSVolumes is no longer needed, since this functionality is now
taken over by StepRunSourceInstance and StepRunSpotInstance. So remove
this functionality from the codebase.
2017-10-12 21:40:21 +02:00
Mark Meyer 2661fd7869 Move building of spot instances into its own step 2017-10-04 12:38:21 +02:00
Mark Meyer cfd6b6fed5 Change EBS builder to do tag-on-creation
The EBS builder will now use the tag-on-creation pattern, so
that it's possible to restrict packer to only create volumes that are
properly tagged by using an AWS policy.
2017-10-03 00:13:02 +02:00
Zanetti, David 8a8b51be97 Allow temporary security group to have source CIDR block explicitly provided 2017-09-26 00:57:56 +00:00
Megan Marsh 2d4bc70d7b use ami prefix to make it clear that these variables are amazon specific
add fixer, fixer tests
2017-08-28 11:21:27 -07:00
Megan Marsh 9e2e467b31 Allow granular modification of sriov and ena enhanced networking options in amazon amis. Maintain old functionality. 2017-08-25 14:49:53 -07:00
Megan Marsh 14b2d04332 make force_deregister work in all regions, not just original region. 2017-08-14 12:31:58 -07:00
Megan Marsh e9ec705497 use block device mappings to know whether a snapshot existed before packer's current run. If yes, don't delete the unencrypted snapshot. 2017-08-08 13:59:55 -07:00
Matthew Hooker 56c513088b
I think this was the intention 2017-06-09 10:34:49 -07:00
Megan Marsh 1944f38985 update docs and clean up 2017-05-31 13:41:32 -07:00
Megan Marsh f9d7292566 builder/amazon regional KMS keys 2017-05-31 11:23:18 -07:00
Paul Thrasher 0be5a28080 Fix issue 4693 - Derive vpc_id from subnet_id
In AWS we can derive the `VpcId` and AZ from the `SubnetId`, so now we do. In the config you can now only specify the `SubnetId`.

This fixes issue #4693.
2017-05-15 13:46:05 -07:00
Matthew Hooker 81522dced0
move packer to hashicorp 2017-04-04 13:39:01 -07:00
Matthew Hooker 50d9546658
builder/amazon: set force_deregister to true on -force 2017-03-09 14:24:49 -08:00
James Nugent e856339309 build/amazon-ebssurrogate: Add region copy, attributes, tags steps
As pointed out in the initial code review of #4351, some of the steps
from the standard EBS builder were (intetionally) omitted. It turns out
that these actually are useful, and the original rationale for the
omission was wrong. Consequently, this commit adds in the following
steps:

- `StepPrevalidate`
- `StepTagEBSVolumes`
- `StepDeregisterAMI`
- `StepCreateEncryptedAMICopy`
- `StepAMIRegionCopy`
- `StepModifyAMIAttribute`
- `StepCreateTags`

We also fix the interpolation filter and documentation to reflect these
additions, though the majority were already documented and just not
functional.
2017-02-27 09:05:39 -06:00
Matthew Hooker 6a7c37beb6
builder/amazon-chroot: support encrypted boot volume 2017-02-26 17:48:13 -08:00
Rickard von Essen ba9cae5078 builder/amazon: Add SourceAMI and BuildRegion template
Added {{ .SourceAMI }} and {{ .BuildRegion }} template values availible
in `ami_description`, `run_tags`, `run_volume_tags`, `tags`, and
`snapshot_tags`.
2017-01-21 20:39:58 +01:00
Rickard von Essen c3a352955e amazon: Added snapshot_users and snapshot_groups 2016-12-04 16:03:01 +01:00
Matthew Hooker 832de2104b
Revert "Merge pull request #4230 from mitchellh/revert-4223-closes_3320"
This reverts commit bda89b0c42, reversing
changes made to f9f9d82419.
2016-11-30 13:28:34 -08:00
Matthew Hooker f5c3c25d2b Revert "Adds `force_delete_snapshot` flag" 2016-11-30 13:10:22 -08:00
Matthew Hooker f9f9d82419 Merge pull request #4223 from artburkart/closes_3320
Adds `force_delete_snapshot` flag
2016-11-30 12:53:34 -08:00
Arthur Burkart 5b59f56cdb Adds `force_delete_snapshot` flag
This PR adds the ability for Packer to clean up snapshots in addition to
deregistering AMIs at build time.

To test this, I used the following `test.json` file:

```json
{
  "builders": [
    {
      "type": "amazon-ebs",
      "region": "us-east-1",
      "source_ami": "ami-fce3c696",
      "ami_name": "packer-test",
      "instance_type": "m3.medium",
      "ssh_username": "ubuntu",
      "vpc_id": "some-vpc-id",
      "subnet_id": "some-subnet-routed-through-igw",
      "security_group_id": "some-security-group-with-port-22-access",
      "force_delete_snapshot": true
    }
  ],
  "provisioners": [
    {
      "type": "shell-local",
      "command": "echo 'hello'"
    }
  ]
}

```

I appreciate any constructive feedbakc that can be given. Cheers!
2016-11-29 23:39:36 -05:00
Rickard von Essen 99ee610afa
builder/amazon: Fix SSH Agent
Closes #4220
2016-11-29 22:33:53 +01:00
Arthur Burkart 2e65867cba Closes #3908: Adds snapshot tag overrides
This commit adds the ability to configure unique tags on snapshots
that are separate from the tags defined on the AMI. Anything applied
to the AMI will also be applied to the snapshots, but `snapshot_tags`
will override and append tags to the tags already applied to the snapshots
2016-11-15 23:16:27 -05:00
Matthew Hooker 60c459d6c8 Merge pull request #4098 from mitchellh/newsession
aws-sdk: use session.NewSession
2016-11-02 12:25:14 -07:00
James Nugent b1ff8c3bfc builder/amazon: Add `ebs-volume` builder
This commit adds a builder that works like EBS builders, except does not
create an AMI, and instead is intended to create EBS volumes in an
initialized state. For example, the following template can be used to
create and export a set of 3 EBS Volumes in a ZFS zpool named `data` for
importing by instances running production systems:

```
{
	"variables": {
		"aws_access_key_id": "{{ env `AWS_ACCESS_KEY_ID` }}",
		"aws_secret_access_key": "{{ env `AWS_SECRET_ACCESS_KEY` }}",
		"region": "{{ env `AWS_REGION` }}",
		"source_ami": "{{ env `PACKER_SOURCE_AMI` }}",
		"vpc_id": "{{ env `PACKER_VPC_ID` }}",
		"subnet_id": "{{ env `PACKER_SUBNET_ID` }}"
	},
	"builders": [{
		"type": "amazon-ebs-volume",
		"access_key": "{{ user `aws_access_key_id` }}",
		"secret_key": "{{ user `aws_secret_access_key` }}",
		"region": "{{user `region`}}",
		"spot_price_auto_product": "Linux/UNIX (Amazon VPC)",

		"ssh_pty": true,
		"instance_type": "t2.medium",
		"vpc_id": "{{user `vpc_id` }}",
		"subnet_id": "{{user `subnet_id` }}",
		"associate_public_ip_address": true,
		"source_ami": "{{user `source_ami` }}",
		"ssh_username": "ubuntu",
		"ssh_timeout": "5m",

		"ebs_volumes": [
			{
				"device_name": "/dev/xvdf",
				"delete_on_termination": false,
				"volume_size": 10,
				"volume_type": "gp2",
				"tags": {
					"Name": "TeamCity-Data1",
					"zpool": "data",
					"Component": "TeamCity"
				}
			},
			{
				"device_name": "/dev/xvdg",
				"delete_on_termination": false,
				"volume_size": 10,
				"volume_type": "gp2",
				"tags": {
					"Name": "TeamCity-Data2",
					"zpool": "data",
					"Component": "TeamCity"
				}
			},
			{
				"device_name": "/dev/xvdh",
				"delete_on_termination": false,
				"volume_size": 10,
				"volume_type": "gp2",
				"tags": {
					"Name": "TeamCity-Data3",
					"zpool": "data",
					"Component": "TeamCity"
				}
			}
		]
	}],
	"provisioners": [
	{
		"type": "shell",
		"start_retry_timeout": "10m",
		"inline": [
			"DEBIAN_FRONTEND=noninteractive sudo apt-get update",
			"DEBIAN_FRONTEND=noninteractive sudo apt-get install -y zfs",
			"lsblk",
			"sudo parted /dev/xvdf --script mklabel GPT",
			"sudo parted /dev/xvdg --script mklabel GPT",
			"sudo parted /dev/xvdh --script mklabel GPT",
			"sudo zpool create -m none data raidz xvdf xvdg xvdh",
			"sudo zpool status",
			"sudo zpool export data",
			"sudo zpool status"
		]
	}
	]
}
```

StepModifyInstance and StepStopInstance are now shared between EBS and
EBS-Volume builders - move them into the AWS common directory and rename
them to indicate that they only apply to EBS-backed builders.
2016-11-02 12:56:39 -04:00
Matthew Hooker 970b37077e Merge pull request #4050 from jen20/ssh-agent
builder/amazon: Allow use of local SSH Agent
2016-11-01 17:27:37 -07:00
Matthew Hooker 48bdae93d1 aws-sdk: use session.NewSession 2016-11-01 15:53:04 -07:00
Rickard von Essen d16d5d9686 Merge pull request #3817 from ChrisLundquist/dynamic-source-ami
Dynamic source ami
2016-10-25 21:24:41 +02:00
James Nugent 7425fef2c7 builder/amazon: Allow use of local SSH Agent
This commit adds an option to use the local SSH Agent to authenticate
connections to source instances started by the the EBS and Instance
Store builders.

This is of use when the source AMI _already_ has configuration for
authorized SSH keys - for example if one uses an SSH certificate
authority.

A further extension (not implemented in this commit) is to allow SSH
agent use with a pre-defined key pair, in order to allow keys with
passphrases to be used without giving the passphrase to Packer.
2016-10-23 21:43:47 -05:00
Rickard von Essen 0be3034e82 Merge pull request #3953 from rickard-von-essen/amz-ssh
builder/amazon: Allow using ssh_private_key_file and ssh_password
2016-10-03 10:22:15 +02:00