Commit Graph

325 Commits

Author SHA1 Message Date
Svetlin Zamfirov d827f4f757 Adding custom shielded images support (GCP)
Adds `enable_secure_boot`, `enable_vtpm` and `enable_integrity_monitoring`
config options to enable building of custom Shielded GCP Compute images.

Feedback on this is more than welcome as this is my first attempt in
contributing to anything Packer related.

Packer is great for us to build custom images on top of GCP but we would
like to enhance that to support Shielded VM images. This will allow us
to have more secure and trusted images which our team(s) will be using.
2020-04-16 16:22:41 -07:00
Adrien Delorme 840e67f775
HCL2: when we see a map generate an attribute spec instead of a block spec (#9035)
* mapstructure-to-hcl2: when we see a map generate an attribute spec and not a block spec

this will alow to do

tags = {
 key = "value"
}

instead of

tags {
  key = "value"
}

This will also enable using variables directly for those tags

* generate code

* update tests
2020-04-14 16:05:13 +02:00
Jeff Escalante 4afd107eeb
patch source files for extensions changes 2020-04-07 17:50:51 -04:00
Jeff Escalante bcc848a83d
were rendering yay 2020-04-07 17:50:47 -04:00
Megan Marsh 0f9415297f
Merge pull request #8679 from kwibus/gcp-fallback-image
[WIP] fix #8589 gcp public fallback image
2020-04-02 16:49:50 -07:00
desolatorxxl 87f7160084 [builder/google] Use "ssh-keys" metadata rather than deprecated "sshKeys" 2020-03-24 08:33:55 +01:00
Megan Marsh 6008f911fd
Merge pull request #8847 from takaishi/support-keyboard-interactive
communicator/ssh: support keyboard-interactive auth to connect bastion
2020-03-13 11:48:12 -07:00
r_takaishi f50ff1d270 make generate 2020-03-13 11:29:42 +09:00
Wilken Rivera 7b705545d9
docs: Various doc clean-ups to fix code block formatting and syntax highlighting (#8868) 2020-03-12 15:05:08 +01:00
Rens Sikma 3d8bda31a4 modify docstring 2020-02-20 12:54:35 +01:00
Rens Sikma c8062cc8b3 fix #8679 with source_image_project_id as list 2020-02-06 10:30:04 +01:00
Megan Marsh d252a24b82 allow user to set winrm_host override like we do ssh_host override 2020-01-30 15:22:22 -08:00
Sylvia Moss dc31bad539
Sharing info with post-processors via artifact (#8632) 2020-01-30 11:27:58 +01:00
Adrien Delorme 4b7132c87c Merge remote-tracking branch 'origin/master' into sharing_info 2019-12-17 11:57:09 +01:00
Adrien Delorme 0785c2f6fc
build using HCL2 (#8423)
This follows #8232 which added the code to generate the code required to parse
HCL files for each packer component.

All old config files of packer will keep on working the same. Packer takes one
argument. When a directory is passed, all files in the folder with a name
ending with  “.pkr.hcl” or “.pkr.json” will be parsed using the HCL2 format.
When a file ending with “.pkr.hcl” or “.pkr.json” is passed it will be parsed
using the HCL2 format. For every other case; the old packer style will be used.

## 1. the hcl2template pkg can create a packer.Build from a set of HCL (v2) files

I had to make the packer.coreBuild (which is our one and only packer.Build ) a public struct with public fields

## 2. Components interfaces get a new ConfigSpec Method to read a file from an HCL file.

  This is a breaking change for packer plugins.

a packer component can be a: builder/provisioner/post-processor

each component interface now gets a `ConfigSpec() hcldec.ObjectSpec`
which allows packer to tell what is the layout of the hcl2 config meant
to configure that specific component.

This ObjectSpec is sent through the wire (RPC) and a cty.Value is now
sent through the already existing configuration entrypoints:

 Provisioner.Prepare(raws ...interface{}) error
 Builder.Prepare(raws ...interface{}) ([]string, error)
 PostProcessor.Configure(raws ...interface{}) error

close #1768


Example hcl files:

```hcl
// file amazon-ebs-kms-key/run.pkr.hcl
build {
    sources = [
        "source.amazon-ebs.first",
    ]

    provisioner "shell" {
        inline = [
            "sleep 5"
        ]
    }

    post-processor "shell-local" {
        inline = [
            "sleep 5"
        ]
    }
}

// amazon-ebs-kms-key/source.pkr.hcl

source "amazon-ebs" "first" {

    ami_name = "hcl2-test"
    region = "us-east-1"
    instance_type = "t2.micro"

    kms_key_id = "c729958f-c6ba-44cd-ab39-35ab68ce0a6c"
    encrypt_boot = true
    source_ami_filter {
        filters {
          virtualization-type = "hvm"
          name =  "amzn-ami-hvm-????.??.?.????????-x86_64-gp2"
          root-device-type = "ebs"
        }
        most_recent = true
        owners = ["amazon"]
    }
    launch_block_device_mappings {
        device_name = "/dev/xvda"
        volume_size = 20
        volume_type = "gp2"
        delete_on_termination = "true"
    }
    launch_block_device_mappings {
        device_name = "/dev/xvdf"
        volume_size = 500
        volume_type = "gp2"
        delete_on_termination = true
        encrypted = true
    }

    ami_regions = ["eu-central-1"]
    run_tags {
        Name = "packer-solr-something"
        stack-name = "DevOps Tools"
    }
    
    communicator = "ssh"
    ssh_pty = true
    ssh_username = "ec2-user"
    associate_public_ip_address = true
}
```
2019-12-17 11:25:56 +01:00
Megan Marsh 0f6d1beccf add an extra string array to the Prepare() return values in the builder interfaces; this sets up the ability for builders to give the provisioners custom user-accessible build-time variables. 2019-12-16 21:23:05 -08:00
Megan Marsh 84063d2132 implement ID and Type values in the provisioner template info sharing 2019-12-13 11:57:01 -08:00
Megan Marsh 6f418d0e54 get data sharing to a working state with the powershell provisioner 2019-12-11 15:43:38 -08:00
Megan Marsh 601007e3e2 pas data into provisioners well 2019-12-11 15:43:38 -08:00
Lars Lehtonen 3571c585bb builder/googlecompute: fix dropped error (#8385) 2019-11-19 11:41:39 +01:00
Megan Marsh 6c704f7046 Revert "fix template imports"
This reverts commit 0e70e0e5a4.
2019-11-04 14:29:00 -08:00
Megan Marsh 517834d528 Revert "Revert "remove clean_ami_name and clean_image_name; complete the deprecation for these functions in favor of clean_resource_name""
This reverts commit 44f91c9afc.
2019-11-04 14:28:40 -08:00
Adrien Delorme 819329228a Change back to make sure all durations are a time.Duration
It is simply the best/simplest solution and trying to prevent users from passing and integer here would be like opening a can of worms. Because:

* we cannot make mapstructure validate our duration string ( with an UnmarshalJSON func etc.)
* we cannot make mapstructure spit a string instead of a duration and packer will decode-encode-decode config.
* the hcl2 generated code asks for a string, so this will be enforced by default.
2019-10-31 16:12:07 +01:00
Adrien Delorme bf3d9841c6 Force durations to be passed a strings
Before this commit it was possible to set a duration using an integer or a float. Go's time.Duration is an int64 internally an mapstructure will take advantage of this and load the number as a int64 but `1` means one ns which is unexpected/confusing. To avoid confusion and enforce readability this forces users to pass a string with a unit for a duration; ex "56s".
2019-10-31 11:47:19 +01:00
Adrien Delorme 078ba7c8c3 commit old code generation tool
for history
2019-10-15 12:56:42 +02:00
Adrien Delorme 17c069baa5 builder.google:
* make CustomerEncryptionKey our own type so that it can be hcl2 generated
* make Account setting unexported so that it doesn't temper with HCL2 generation ( the field is set a bit later after processing )
2019-10-15 12:56:41 +02:00
Megan Marsh 0e70e0e5a4 fix template imports 2019-10-14 13:22:06 -07:00
Megan Marsh 44f91c9afc Revert "remove clean_ami_name and clean_image_name; complete the deprecation for these functions in favor of clean_resource_name"
This reverts commit 0026792d2f.
2019-10-14 13:08:15 -07:00
Adrien Delorme d4600a7716 fix go imports 2019-10-04 11:19:45 +02:00
Megan Marsh 0026792d2f remove clean_ami_name and clean_image_name; complete the deprecation for these functions in favor of clean_resource_name 2019-10-03 15:49:01 -07:00
Lars Lehtonen ad860bf1a8
builder: remove deprecated context usage 2019-09-30 11:33:53 -07:00
Megan Marsh 3c14c50aba Allow user to real oauth token from properly configured Vault instance 2019-09-23 13:26:03 -07:00
Adrien Delorme 3fe9d52e3f Merge remote-tracking branch 'origin/master' into scrape_doc_to_builder_struct_config 2019-09-20 11:15:44 +02:00
Megan Marsh 35357616e0
Merge pull request #8112 from milescrabill/packer-googlecompute-validate-imagename
googlecompute: fail fast when image name is invalid, replace unusable characters w/ -'s
2019-09-17 10:00:33 -07:00
Miles Crabill 2bff60bac8
address review feedback 2019-09-16 13:50:43 -07:00
Miles Crabill 328baced05
add some test cases for image names 2019-09-16 10:18:22 -07:00
Miles Crabill c11a444f77
googlecompute: fail fast when image name is invalid, replace unusable characters w/ -'s 2019-09-16 10:10:41 -07:00
Megan Marsh a6e263adb6 don't fall for nil pointer dereference in account file 2019-09-13 16:37:19 -07:00
Adrien Delorme 2db109d55b Merge branch 'master' into scrape_doc_to_builder_struct_config 2019-08-23 17:37:53 +02:00
Megan Marsh fcef7e2d85 this timeout regularly flakes on Travis, I suspect because the instances running the tests are brutally underpowered. Increase the fudge factor. 2019-08-21 16:36:42 -07:00
Adrien Delorme 4cb7c30987 Merge remote-tracking branch 'origin/master' into scrape_doc_to_builder_struct_config 2019-08-21 16:44:23 +02:00
Mason, Elliot ee484f2dd9 Updated public gcp cloud images 2019-07-31 11:38:35 -05:00
Megan Marsh 7c93a5c51e code changed recently so that sometimes the conf is nil. 2019-07-29 13:19:43 -07:00
Megan Marsh a87ce366b3
Merge pull request #7835 from hashicorp/google_oauth
replace some bespoke google auth code with code from golang's oauth2 …
2019-07-08 09:48:27 -07:00
Megan Marsh 617fd18255 supply complete dummy data for googlecompute tests 2019-07-03 15:58:08 -07:00
Megan Marsh 0dfb3cc56f replace some bespoke google auth code with code from golang's oauth2 library 2019-07-03 14:27:33 -07:00
Megan Marsh c03c3f4410 replace duplicate commHost functions with a single communicator helper that checks for ssh_host and does a statebag lookup for a given key 2019-07-03 13:30:29 -07:00
Megan Marsh eb8c74bf85 allow ssh_host config option to override host logic for all builders 2019-07-02 13:56:28 -07:00
Megan Marsh ad8a88e366 allow ssh_host to override the host ip gathered from the instance for GCP builder 2019-07-02 13:01:41 -07:00
Adrien Delorme a3e3c313a9 docs: use partials in googlecompute after master merge 2019-06-12 15:18:39 +02:00
Adrien Delorme 6e3dc1aa9a Merge remote-tracking branch 'origin/master' into scrape_doc_to_builder_struct_config 2019-06-12 14:28:11 +02:00
Pratyush singhal 92af5847a7 refactor: replace *packer.MultiError from type signature of createInstanceMetadata with generic error interface
Signed-off-by: Pratyush singhal <psinghal20@gmail.com>
2019-06-11 21:01:26 +05:30
Pratyush singhal 6ce6bd8ad3 refactor: add multiError in createInstanceMetadata method to capture multiple errors
Signed-off-by: Pratyush singhal <psinghal20@gmail.com>
2019-06-11 20:09:14 +05:30
Pratyush singhal 4a369b4ef1 chore: add test for MetadataFiles option
Signed-off-by: Pratyush singhal <psinghal20@gmail.com>
2019-06-11 17:45:30 +05:30
Pratyush singhal 99a3e9cf0a chore: update tests for createInstaceMetadata
Signed-off-by: Pratyush singhal <psinghal20@gmail.com>
2019-06-11 16:47:16 +05:30
Pratyush singhal 529dff0abb refactor: add error handling in createInstanceMetadata method
Signed-off-by: Pratyush singhal <psinghal20@gmail.com>
2019-06-11 16:36:16 +05:30
Adrien Delorme 4399684372 make fmt autogenerated docs 2019-06-06 16:29:25 +02:00
Pratyush singhal 1e1af35341 refactor: replace userdata_files with generic metadata_files map
Signed-off-by: Pratyush singhal <psinghal20@gmail.com>
2019-06-06 16:14:57 +05:30
Pratyush singhal 3845186c4c feat: add feature to import user-data from a file
Signed-off-by: Pratyush singhal <psinghal20@gmail.com>
2019-06-06 11:28:59 +05:30
Adrien Delorme e6cbb013ba add // go:generate struct-markdown to all previously edited files 2019-06-05 16:42:18 +02:00
Adrien Delorme f1917edd34 generate the comments for config struct of builders scraping doc website 2019-06-05 16:42:17 +02:00
krisko e96bda6466
Remove config interface. Make CreateImage function to accept config as list of config fields 2019-05-05 17:17:08 +02:00
krisko f1e9664012
Googlebuilder: Add image key encryption 2019-05-02 17:37:43 +02:00
krisko 9b39e3f928
GCP builder add image encryption support 2019-05-02 17:37:36 +02:00
Adrien Delorme d72040f4fa move retry code into the common/retry pkg and make retry context aware 2019-04-09 17:46:38 +02:00
Adrien Delorme a81abd297b Merge remote-tracking branch 'origin/master' into context_provisioner 2019-04-08 20:09:01 +02:00
Adrien Delorme c4f3dccc14 rename interpolation context from ctx to ictx and contexts to ctx to avoid conflicts 2019-04-03 15:56:15 +02:00
Adrien Delorme a4bf94dd3c change Builder to be passed a context for cancellation
we have to to give it to our hook
2019-04-03 15:55:55 +02:00
Adrien Delorme 9836dad0c9 introduce the clean_resource_name to clean image names and deprecate the old ones 2019-04-03 11:25:51 +02:00
Adrien Delorme 9f82b75e57 Use the hashicorp/go-getter to download files
* removed packer.Cache and references since packer.Cache is never used except in the download step. The download step now uses the new func packer.CachePath(targetPath) for this, the behavior is the same.
* removed download code from packer that was reimplemented into the go-getter library: progress bar, http download restart, checksuming from file, skip already downloaded files, symlinking, make a download cancellable by context.
* on windows if packer is running without symlinking rights and we are getting a local file, the file will be copied instead to avoid errors.
* added unit tests for step_download that are now CI tested on windows, mac & linux.
* files are now downloaded under cache dir `sha1(filename + "?checksum=" + checksum) + file_extension`
* since the output dir is based on the source url and the checksum, when the checksum fails, the file is auto deleted.
* a download file is protected and locked by a file lock,
* updated docs
* updated go modules and vendors
2019-03-13 12:11:58 +01:00
Murcurio 60c7d22ffb
Change metadata url to FQDN
If instance has a different DNS/search domain then "metadata" alone won't work. The FQDN of "metadata.google.internal" is included in hosts file, and will work regardless of DNS configuration. Alternatively the IP 169.254.169.254 can be used.
2019-02-01 15:36:40 +11:00
Arnaud Dezandee 91d2cb8c83
googlecompute-import: add service accounts 2019-01-25 11:36:18 +01:00
Mikhail Ushanov 536252683e builders: reuse private key file reading function
Signed-off-by: Mikhail Ushanov <gm.mephisto@gmail.com>
2018-11-08 14:25:50 +03:00
Matthew Aynalem 28eff7303e builder/googlecompute: validate startup_script_file exists 2018-10-15 16:09:33 -07:00
Adrien Delorme b2e2d3b008 removed unused processPrivateKeyFile func & tests 2018-10-09 11:56:43 +02:00
Adrien Delorme 146db5667a fix intermitent TestProcessPrivateKeyFile_encrypted failure
Because of deficiencies in the encrypted-PEM format, it's not always possible to detect an incorrect
password. In these cases no error will be returned but the decrypted DER bytes will be random noise.

this closes #3337
2018-10-01 10:29:30 +02:00
Megan Marsh 38cc525ec7 new option allowing user to clean up the ephemeral ssh key from the authorized_keys file 2018-09-14 11:06:38 -07:00
Megan Marsh a0edaf6c46 Going to revert this change for now, becuase of potential issues that arise from calling Prepare() twice
Revert "use statebag instead of SetSharedState for winRM password"

This reverts commit b35acbd879.
2018-09-10 16:48:42 -07:00
Megan Marsh d00b815c06 Update public google cloud image projects 2018-08-31 13:13:19 -07:00
Adrien Delorme 157691c710 remove unecessary get/pug config in tests; those are being referenced 2018-08-30 12:07:37 +02:00
Adrien Delorme 94d5bb7a3b revert test changes of builder/googlecompute/step_create_ssh_key_test.go 2018-08-30 12:04:21 +02:00
Adrien Delorme 050a98e8bb googlecompute builder: use config from statebag 2018-08-29 18:15:09 +02:00
Adrien Delorme 4982627dd2 builder.gcp: use c.Comm for ssh 2018-08-29 14:40:33 +02:00
Adrien Delorme d49ad12895 update gcp ssh tests 2018-08-29 14:40:32 +02:00
Adrien Delorme 663c8134ef GCP: put ssh public/private key in config 2018-08-29 14:40:32 +02:00
Adrien Delorme 51d2aac9f6 SSHPrivateKey => SSHPrivateKeyFile 2018-08-29 14:40:32 +02:00
Megan Marsh b35acbd879 use statebag instead of SetSharedState for winRM password 2018-08-24 13:51:16 -07:00
Megan Marsh 340363a649
Merge pull request #6613 from hashicorp/merge_ssh_config_funcs
Merge ssh config funcs from builders
2018-08-23 16:03:36 -07:00
Megan Marsh 1f79b430ee
Merge pull request #6610 from hashicorp/filter_logs
Filter logs
2018-08-23 13:30:21 -07:00
Adrien Delorme 41f6e0334d refactor all copy pasted sshConfig with into communicator.Config.SSHConfigFunc
* still need to append the auth methods into an array for gracefullness
2018-08-22 17:02:23 +02:00
Adrien Delorme b038cd10f5 GCP: Allow to set MinCpuPlatform 2018-08-21 10:09:30 +02:00
Adrien Delorme e37f126cf1 update gcp google.golang.org/api/compute/v1/ 2018-08-21 10:09:01 +02:00
Megan Marsh ff6a039d5b replace scrubconfig with packer.LogSecretFilter.Set
filter winrm password from logs
Add new root-level packer template option, sensitive-variables, to tell us what user variables to mark sensitive.
2018-08-20 15:35:55 -07:00
deepuashokan85 10f118b14c
Update driver_gce.go
Added "rhel-sap-cloud" Projects_ID.
2018-05-31 11:29:57 +05:30
M. Marsh e90c87f885
Merge pull request #6210 from hashicorp/add_windows_sql_cloud_to_defaults
add windows-sql-cloud to the list of projects to check for an image name
2018-05-01 12:57:10 -07:00
DanHam 5642f8cb01
Remove tmp files created by builder/googlecompute tests 2018-04-30 20:38:03 +01:00
Megan Marsh ed6d999b1f add windows-sql-cloud to the list of projects to check for an image name 2018-04-27 11:24:08 -07:00
Richard Nienaber db65c99bf4 update code based on master and update docs 2018-04-24 10:35:27 +01:00
Richard Nienaber ec6f8264bf fix winrm password access in google compute 2018-04-16 10:56:26 +01:00
Seth Vargo b17b333e29
Add a common package for specifying useragent and adopt that everywhere
There were 5 different formats for the Packer useragent string. This
fixes that and unifies it into a helper package.

I did not touch oracle's user-agent, because it looked kinda special.
2018-04-05 14:28:50 -04:00