From a418a22caea93e98d24bcfbb1f8b5c35321ca41f Mon Sep 17 00:00:00 2001 From: DanHam Date: Thu, 29 Aug 2019 15:40:37 +0100 Subject: [PATCH 1/7] Docs: Fix missing required option `ami_name` for ebssurrogate builder --- .../source/docs/builders/amazon-ebssurrogate.html.md.erb | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/website/source/docs/builders/amazon-ebssurrogate.html.md.erb b/website/source/docs/builders/amazon-ebssurrogate.html.md.erb index e34e9ed1b..37148385f 100644 --- a/website/source/docs/builders/amazon-ebssurrogate.html.md.erb +++ b/website/source/docs/builders/amazon-ebssurrogate.html.md.erb @@ -38,6 +38,11 @@ builder. - `access_key` (string) - The access key used to communicate with AWS. [Learn how to set this](/docs/builders/amazon.html#specifying-amazon-credentials) +- `ami_name` (string) - The name of the resulting AMI that will appear when + managing AMIs in the AWS console or via APIs. This must be unique. To help + make this unique, use a function like `timestamp` (see [template + engine](../templates/engine.html) for more info). + - `instance_type` (string) - The EC2 instance type to use while building the AMI, such as `m1.small`. @@ -591,4 +596,4 @@ volumes created by this builder, any volumes inn the source AMI which are not marked for deletion on termination will remain in your account. -<%= partial "partials/builders/aws-ssh-differentiation-table" %> \ No newline at end of file +<%= partial "partials/builders/aws-ssh-differentiation-table" %> From 06b049987babc4edc5e11ea6b6dd1aeb375b133b Mon Sep 17 00:00:00 2001 From: DanHam Date: Thu, 29 Aug 2019 16:44:36 +0100 Subject: [PATCH 2/7] Docs: Formatting, fixes, nits, and improvements to the ebsvolume builder doc * Fix or remove references to AMI creation copied in from other docs * Remove unsupported `snapshot_users` and `snapshot_groups` options --- .../builders/amazon-ebsvolume.html.md.erb | 64 ++++++++----------- 1 file changed, 28 insertions(+), 36 deletions(-) diff --git a/website/source/docs/builders/amazon-ebsvolume.html.md.erb b/website/source/docs/builders/amazon-ebsvolume.html.md.erb index c9e5d885c..e11321e4d 100644 --- a/website/source/docs/builders/amazon-ebsvolume.html.md.erb +++ b/website/source/docs/builders/amazon-ebsvolume.html.md.erb @@ -1,7 +1,7 @@ --- description: | - The amazon-ebsvolume Packer builder is like the EBS builder, but is intended to - create EBS volumes rather than a machine image. + The amazon-ebsvolume Packer builder is like the EBS builder, but is + intended to create EBS volumes rather than a machine image. layout: docs page_title: 'Amazon EBS Volume - Builders' sidebar_current: 'docs-builders-amazon-ebsvolume' @@ -14,9 +14,11 @@ Type: `amazon-ebsvolume` The `amazon-ebsvolume` Packer builder is able to create Amazon Elastic Block Store volumes which are prepopulated with filesystems or data. -This builder builds EBS volumes by launching an EC2 instance from a source AMI, -provisioning that running machine, and then destroying the source machine, -keeping the volumes intact. +This builder creates EBS volumes by launching an EC2 instance from a source +AMI. One or more EBS volumes are attached to the running instance, allowing +them to be provisioned into from the running machine. Once provisioning is +complete the source machine is destroyed. The provisioned volumes are kept +intact. This is all done in your own AWS account. The builder will create temporary key pairs, security group rules, etc. that provide it temporary access to the @@ -44,11 +46,11 @@ builder. - `access_key` (string) - The access key used to communicate with AWS. [Learn how to set this.](/docs/builders/amazon.html#specifying-amazon-credentials) -- `instance_type` (string) - The EC2 instance type to use while building the - AMI, such as `m1.small`. +- `instance_type` (string) - The EC2 instance type to use while creating + the EBS volumes, such as `m1.small`. - `region` (string) - The name of the region, such as `us-east-1`, in which - to launch the EC2 instance to create the AMI. + to launch the EC2 instance to create the new EBS volumes. - `secret_key` (string) - The secret key used to communicate with AWS. [Learn how to set this.](/docs/builders/amazon.html#specifying-amazon-credentials) @@ -60,7 +62,7 @@ builder. ### Optional: - `ebs_volumes` (array of block device mappings) - Add the block device - mappings to the AMI. The block device mappings allow for keys: + mappings to the running instance. The block device mappings allow for keys: - `device_name` (string) - The device name exposed to the instance (for example, `/dev/sdh` or `xvdh`). Required for every device in the block @@ -82,24 +84,24 @@ builder. - `iops` (number) - The number of I/O operations per second (IOPS) that the volume supports. See the documentation on [IOPs](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_EbsBlockDevice.html) - for more information + for more information. - `no_device` (boolean) - Suppresses the specified device included in the - block device mapping of the AMI + block device mapping of the AMI. - - `snapshot_id` (string) - The ID of the snapshot + - `snapshot_id` (string) - The ID of the snapshot. - `virtual_name` (string) - The virtual device name. See the documentation on [Block Device Mapping](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_BlockDeviceMapping.html) - for more information + for more information. - `volume_size` (number) - The size of the volume, in GiB. Required if - not specifying a `snapshot_id` + not specifying a `snapshot_id`. - `volume_type` (string) - The volume type. `gp2` for General Purpose (SSD) volumes, `io1` for Provisioned IOPS (SSD) volumes, and `standard` - for Magnetic volumes + for Magnetic volumes. - `tags` (map) - Tags to apply to the volume. These are retained after the builder completes. This is a [template @@ -108,7 +110,7 @@ builder. - `associate_public_ip_address` (boolean) - If using a non-default VPC, public IP addresses are not provided by default. If this is `true`, your - new instance will get a Public IP. default: `false` + new instance will get a Public IP. Default: `false`. - `availability_zone` (string) - Destination availability zone to launch instance in. Leave this empty to allow Amazon to auto-assign. @@ -158,11 +160,9 @@ builder. - `ena_support` (boolean) - Enable enhanced networking (ENA but not SriovNetSupport) on HVM-compatible AMIs. If set, add - `ec2:ModifyInstanceAttribute` to your AWS IAM policy. If false, this will - disable enhanced networking in the final AMI as opposed to passing the - setting through unchanged from the source. Note: you must make sure - enhanced networking is enabled on your instance. See [Amazon's - documentation on enabling enhanced + `ec2:ModifyInstanceAttribute` to your AWS IAM policy. Note: you must + make sure enhanced networking is enabled on your instance. See + [Amazon's documentation on enabling enhanced networking](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html#enabling_enhanced_networking). - `enable_t2_unlimited` (boolean) - Enabling T2 Unlimited allows the source @@ -206,8 +206,7 @@ builder. for more details. - `run_tags` (object of key/value strings) - Tags to apply to the instance - that is *launched* to create the AMI. These tags are *not* applied to the - resulting AMI unless they're duplicated in `tags`. This is a [template + that is *launched* to create the EBS volumes. This is a [template engine](/docs/templates/engine.html), see [Build template data](#build-template-data) for more information. @@ -250,16 +249,6 @@ builder. - `skip_region_validation` (boolean) - Set to `true` if you want to skip validation of the region configuration option. Defaults to `false`. -- `snapshot_groups` (array of strings) - A list of groups that have access to - create volumes from the snapshot(s). By default no groups have permission - to create volumes from the snapshot(s). `all` will make the snapshot - publicly accessible. - -- `snapshot_users` (array of strings) - A list of account IDs that have - access to create volumes from the snapshot(s). By default no additional - users other than the user creating the AMI has permissions to create - volumes from the backing snapshot(s). - - `source_ami_filter` (object) - Filters used to populate the `source_ami` field. Example: @@ -376,9 +365,12 @@ builder. `packer_`, where <UUID> is a 36 character unique identifier. - `temporary_security_group_source_cidrs` (list of string) - A list of IPv4 - CIDR blocks to be authorized access to the instance, when packer is creating a temporary security group. + CIDR blocks to be authorized access to the instance, when packer is + creating a temporary security group. - The default is [`0.0.0.0/0`] (i.e., allow any IPv4 source). This is only used when `security_group_id` or `security_group_ids` is not specified. + The default is [`0.0.0.0/0`] (i.e., allow any IPv4 source). This is + only used when `security_group_id` or `security_group_ids` is not + specified. - `token` (string) - The access token to use. This is different from the access key and secret key. If you're not sure what this is, then you @@ -551,4 +543,4 @@ volumes created by this builder, any volumes inn the source AMI which are not marked for deletion on termination will remain in your account. -<%= partial "partials/builders/aws-ssh-differentiation-table" %> \ No newline at end of file +<%= partial "partials/builders/aws-ssh-differentiation-table" %> From 30070ffb30ee5e118abf3e401fd5164da9cc06fc Mon Sep 17 00:00:00 2001 From: DanHam Date: Wed, 28 Aug 2019 09:31:34 +0100 Subject: [PATCH 3/7] Sort fields and options alphabetically --- builder/amazon/ebsvolume/builder.go | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/builder/amazon/ebsvolume/builder.go b/builder/amazon/ebsvolume/builder.go index 7047c1734..941b2f958 100644 --- a/builder/amazon/ebsvolume/builder.go +++ b/builder/amazon/ebsvolume/builder.go @@ -23,9 +23,9 @@ type Config struct { awscommon.AccessConfig `mapstructure:",squash"` awscommon.RunConfig `mapstructure:",squash"` - VolumeMappings []BlockDevice `mapstructure:"ebs_volumes"` AMIENASupport *bool `mapstructure:"ena_support"` AMISriovNetSupport bool `mapstructure:"sriov_support"` + VolumeMappings []BlockDevice `mapstructure:"ebs_volumes"` launchBlockDevices awscommon.BlockDevices ctx interpolate.Context @@ -120,8 +120,8 @@ func (b *Builder) Run(ctx context.Context, ui packer.Ui, hook packer.Hook) (pack AssociatePublicIpAddress: b.config.AssociatePublicIpAddress, BlockDevices: b.config.launchBlockDevices, BlockDurationMinutes: b.config.BlockDurationMinutes, - Ctx: b.config.ctx, Comm: &b.config.RunConfig.Comm, + Ctx: b.config.ctx, Debug: b.config.PackerDebug, EbsOptimized: b.config.EbsOptimized, ExpectedRootDevice: "ebs", @@ -129,8 +129,8 @@ func (b *Builder) Run(ctx context.Context, ui packer.Ui, hook packer.Hook) (pack InstanceInitiatedShutdownBehavior: b.config.InstanceInitiatedShutdownBehavior, InstanceType: b.config.InstanceType, SourceAMI: b.config.SourceAmi, - SpotPrice: b.config.SpotPrice, SpotInstanceTypes: b.config.SpotInstanceTypes, + SpotPrice: b.config.SpotPrice, SpotTags: b.config.SpotTags, Tags: b.config.RunTags, UserData: b.config.UserData, From 430d41fbf9f5b8deeb976abcdf31b394af70b7c4 Mon Sep 17 00:00:00 2001 From: DanHam Date: Wed, 28 Aug 2019 09:34:16 +0100 Subject: [PATCH 4/7] Add option to enable tagging of the running instances volumes * Currently this results in *all* volumes attached to the instance being tagged with the `run_volume_tags`. This includes any `ebs_volumes` for which the user may have configured other tags. * This issue will be addressed in a later commit --- builder/amazon/ebsvolume/builder.go | 9 ++++++--- 1 file changed, 6 insertions(+), 3 deletions(-) diff --git a/builder/amazon/ebsvolume/builder.go b/builder/amazon/ebsvolume/builder.go index 941b2f958..8a5b840ae 100644 --- a/builder/amazon/ebsvolume/builder.go +++ b/builder/amazon/ebsvolume/builder.go @@ -23,9 +23,10 @@ type Config struct { awscommon.AccessConfig `mapstructure:",squash"` awscommon.RunConfig `mapstructure:",squash"` - AMIENASupport *bool `mapstructure:"ena_support"` - AMISriovNetSupport bool `mapstructure:"sriov_support"` - VolumeMappings []BlockDevice `mapstructure:"ebs_volumes"` + AMIENASupport *bool `mapstructure:"ena_support"` + AMISriovNetSupport bool `mapstructure:"sriov_support"` + VolumeMappings []BlockDevice `mapstructure:"ebs_volumes"` + VolumeRunTags awscommon.TagMap `mapstructure:"run_volume_tags"` launchBlockDevices awscommon.BlockDevices ctx interpolate.Context @@ -135,6 +136,7 @@ func (b *Builder) Run(ctx context.Context, ui packer.Ui, hook packer.Hook) (pack Tags: b.config.RunTags, UserData: b.config.UserData, UserDataFile: b.config.UserDataFile, + VolumeTags: b.config.VolumeRunTags, } } else { instanceStep = &awscommon.StepRunSourceInstance{ @@ -154,6 +156,7 @@ func (b *Builder) Run(ctx context.Context, ui packer.Ui, hook packer.Hook) (pack Tags: b.config.RunTags, UserData: b.config.UserData, UserDataFile: b.config.UserDataFile, + VolumeTags: b.config.VolumeRunTags, } } From 61b1605a85839d8757e7f1734234863250463509 Mon Sep 17 00:00:00 2001 From: DanHam Date: Wed, 28 Aug 2019 17:20:36 +0100 Subject: [PATCH 5/7] More accurate error description; Add comments; Add output --- builder/amazon/ebsvolume/step_tag_ebs_volumes.go | 12 +++++++++--- 1 file changed, 9 insertions(+), 3 deletions(-) diff --git a/builder/amazon/ebsvolume/step_tag_ebs_volumes.go b/builder/amazon/ebsvolume/step_tag_ebs_volumes.go index e58cb98de..29320ea2e 100644 --- a/builder/amazon/ebsvolume/step_tag_ebs_volumes.go +++ b/builder/amazon/ebsvolume/step_tag_ebs_volumes.go @@ -4,6 +4,7 @@ import ( "context" "fmt" + "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/service/ec2" "github.com/hashicorp/packer/helper/multistep" "github.com/hashicorp/packer/packer" @@ -45,15 +46,19 @@ func (s *stepTagEBSVolumes) Run(ctx context.Context, state multistep.StateBag) m continue } + ui.Message(fmt.Sprintf("Compiling list of tags to apply to volume on %s...", mapping.DeviceName)) tags, err := mapping.Tags.EC2Tags(s.Ctx, *ec2conn.Config.Region, state) if err != nil { - err := fmt.Errorf("Error tagging device %s with %s", mapping.DeviceName, err) + err := fmt.Errorf("Error generating tags for device %s: %s", mapping.DeviceName, err) state.Put("error", err) ui.Error(err.Error()) return multistep.ActionHalt } tags.Report(ui) + // Generate the map of volumes and associated tags to apply. + // Looping over the instance block device mappings allows us to + // obtain the volumeId for _, v := range instance.BlockDeviceMappings { if *v.DeviceName == mapping.DeviceName { toTag[*v.Ebs.VolumeId] = tags @@ -61,9 +66,11 @@ func (s *stepTagEBSVolumes) Run(ctx context.Context, state multistep.StateBag) m } } + // Apply the tags for volumeId, tags := range toTag { + ui.Message(fmt.Sprintf("Applying tags to EBS Volume: %s", volumeId)) _, err := ec2conn.CreateTags(&ec2.CreateTagsInput{ - Resources: []*string{&volumeId}, + Resources: aws.StringSlice([]string{volumeId}), Tags: tags, }) if err != nil { @@ -72,7 +79,6 @@ func (s *stepTagEBSVolumes) Run(ctx context.Context, state multistep.StateBag) m ui.Error(err.Error()) return multistep.ActionHalt } - } } From bbeb1a3528538159dca808162be64ce80169bcd6 Mon Sep 17 00:00:00 2001 From: DanHam Date: Thu, 29 Aug 2019 13:01:20 +0100 Subject: [PATCH 6/7] Delete any `run_volume_tags` applied to EBS volumes at instance creation * Allows `run_volume_tags` to be set without requiring a major rewrite of the (common) amazon builder code used to start an instance. The common start up code tags the instance and *all attached volumes at creation*. If `run_volume_tags` are set this means that any volumes specified in `ebs_volumes` will *initially* be tagged with the `run_volume_tags` rather than the tags set in the `ebs_volumes` section * Once the instance is reported to be 'ready' the step to tag the EBS volumes is run. Once complete all volumes should have the tags requested by the user: * Volumes associated with the source instance should be tagged with the tags set in `run_volume_tags` (if any) * Each EBS volumes specified in the `ebs_volumes` section of the template should only be tagged with its associated tags (if any) --- .../amazon/ebsvolume/step_tag_ebs_volumes.go | 46 ++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/builder/amazon/ebsvolume/step_tag_ebs_volumes.go b/builder/amazon/ebsvolume/step_tag_ebs_volumes.go index 29320ea2e..d98848d70 100644 --- a/builder/amazon/ebsvolume/step_tag_ebs_volumes.go +++ b/builder/amazon/ebsvolume/step_tag_ebs_volumes.go @@ -3,6 +3,7 @@ package ebsvolume import ( "context" "fmt" + "strings" "github.com/aws/aws-sdk-go/aws" "github.com/aws/aws-sdk-go/service/ec2" @@ -20,6 +21,7 @@ func (s *stepTagEBSVolumes) Run(ctx context.Context, state multistep.StateBag) m ec2conn := state.Get("ec2").(*ec2.EC2) instance := state.Get("instance").(*ec2.Instance) ui := state.Get("ui").(packer.Ui) + config := state.Get("config").(*Config) volumes := make(EbsVolumes) for _, instanceBlockDevices := range instance.BlockDeviceMappings { @@ -37,8 +39,50 @@ func (s *stepTagEBSVolumes) Run(ctx context.Context, state multistep.StateBag) m state.Put("ebsvolumes", volumes) if len(s.VolumeMapping) > 0 { - ui.Say("Tagging EBS volumes...") + // If run_volume_tags were set in the template any attached EBS + // volume will have had these tags applied when the instance was + // created. We now need to remove these tags to ensure only the EBS + // volume tags are applied (if any) + if config.VolumeRunTags.IsSet() { + ui.Say("Removing any tags applied to EBS volumes when the source instance was created...") + ui.Message("Compiling list of existing tags to remove...") + existingTags, err := config.VolumeRunTags.EC2Tags(s.Ctx, *ec2conn.Config.Region, state) + if err != nil { + err := fmt.Errorf("Error generating list of tags to remove: %s", err) + state.Put("error", err) + ui.Error(err.Error()) + return multistep.ActionHalt + } + existingTags.Report(ui) + + // Generate the list of volumes with tags to delete. + // Looping over the instance block device mappings allows us to + // obtain the volumeId + volumeIds := []string{} + for _, mapping := range s.VolumeMapping { + for _, v := range instance.BlockDeviceMappings { + if *v.DeviceName == mapping.DeviceName { + volumeIds = append(volumeIds, *v.Ebs.VolumeId) + } + } + } + + // Delete the tags + ui.Message(fmt.Sprintf("Deleting 'run_volume_tags' on EBS Volumes: %s", strings.Join(volumeIds, ", "))) + _, err = ec2conn.DeleteTags(&ec2.DeleteTagsInput{ + Resources: aws.StringSlice(volumeIds), + Tags: existingTags, + }) + if err != nil { + err := fmt.Errorf("Error deleting tags on EBS Volumes %s: %s", strings.Join(volumeIds, ", "), err) + state.Put("error", err) + ui.Error(err.Error()) + return multistep.ActionHalt + } + } + + ui.Say("Tagging EBS volumes...") toTag := map[string][]*ec2.Tag{} for _, mapping := range s.VolumeMapping { if len(mapping.Tags) == 0 { From 6b80075db0a2846572264623d7d85cc35b42abf4 Mon Sep 17 00:00:00 2001 From: DanHam Date: Thu, 29 Aug 2019 17:53:06 +0100 Subject: [PATCH 7/7] Docs: Add `run_volume_tags` option to the ebsvolume docs * Add note explaining temp override of `ebs_volumes` tags when the instance is initially created --- .../builders/amazon-ebsvolume.html.md.erb | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/website/source/docs/builders/amazon-ebsvolume.html.md.erb b/website/source/docs/builders/amazon-ebsvolume.html.md.erb index e11321e4d..30e52cfab 100644 --- a/website/source/docs/builders/amazon-ebsvolume.html.md.erb +++ b/website/source/docs/builders/amazon-ebsvolume.html.md.erb @@ -108,6 +108,12 @@ builder. engine](/docs/templates/engine.html), see [Build template data](#build-template-data) for more information. + Note: The tags specified here may be *temporarily* overridden by + those specified in `run_volume_tags` - but only while the instance + is being created. Packer will replace all tags on the volume with + those configured here as soon as the instance is reported as + 'ready'. + - `associate_public_ip_address` (boolean) - If using a non-default VPC, public IP addresses are not provided by default. If this is `true`, your new instance will get a Public IP. Default: `false`. @@ -210,6 +216,19 @@ builder. engine](/docs/templates/engine.html), see [Build template data](#build-template-data) for more information. +- `run_volume_tags` (object of key/value strings) - Tags to apply to the + volumes of the instance that is *launched* to create EBS Volumes. These + tags will *not* appear in the tags of the resulting EBS volumes unless + they're duplicated under `tags` in the `ebs_volumes` setting. This is a + [template engine](/docs/templates/engine.html), see [Build template + data](#build-template-data) for more information. + + Note: The tags specified here will be *temporarily* applied to volumes + specified in `ebs_volumes` - but only while the instance is being + created. Packer will replace all tags on the volume with the tags + configured in the `ebs_volumes` section as soon as the instance is + reported as 'ready'. + - `security_group_id` (string) - The ID (*not* the name) of the security group to assign to the instance. By default this is not set and Packer will automatically create a new temporary security group to allow SSH access.