Commit Graph

348 Commits

Author SHA1 Message Date
Adrien Delorme 44616d3bff
refactor initialization out from packer configs + tests (#9627)
The initialization of packer core in JSON also validates that `null` variables were set, except in the case of `packer validate --syntax-only` , but after the refactor to allow to have all commands work with HCL2 and JSON this subtlety was lost.

This refactors the initialisation of the core in order to allow to have `packer validate --syntax-only` not error in case a variable is not set. Since these calls are refactored this works for HCL2 too.

fix #9478
2020-07-24 10:58:03 +02:00
Megan Marsh a687d6fe12
ci/tech-debt: Fix vmware acceptance tests (#9636)
* add kex algorithm option to ssh config

* regenerate code

* This commit fixes old vmware acceptance tests that have not been run in some time. It does this in two parts:
1) It modifies the minimal vmware build configuration to use a custom kex algorithm, which enables the ssh connection to succeed.
2) It modifies logic in reading and defaulting hardware config values, which was crashing.
3) It adds a new acceptance test with a preseed file to test loading from an http directory.
2020-07-23 09:40:57 +02:00
Roger Hu 28f8241a05 update generated files 2020-07-10 16:58:06 -07:00
Moss 8ea64c5b35 undocument ssh_keypair_name and ssh_agent_auth 2020-07-02 14:55:58 +02:00
Moss 41edd09d3c undocument temporary_key_pair_name and ssh_private_key_file 2020-07-01 16:33:18 +02:00
Megan Marsh 4f77bb0c4a add new ssh_ciphers option 2020-06-17 16:48:23 -07:00
Megan Marsh 4178625afb re-add winrm_no_proxy option. Winrm code needs to be modified in order to read no_proxy env var when client is created rather than when net/http library is instantiated. 2020-06-16 10:27:30 -07:00
Joshua Foster d35f342319 add the ability to configure the http bind address. defaults to 0.0.0.0. 2020-06-03 16:40:18 -07:00
Adrien Delorme 40947c2bf6
HCL2: generate hcl tags with go-cty tags too (#9306)
This will allow to generate the config files:

```go
package main

import (
	"fmt"

	"github.com/hashicorp/hcl/v2/gohcl"
	"github.com/hashicorp/hcl/v2/hclwrite"
	"github.com/hashicorp/packer/builder/alicloud/ecs"
)

func main() {
	name := "name"
	app := ecs.FlatConfig{
		AlicloudImageName: &name,
		ECSSystemDiskMapping: &ecs.FlatAlicloudDiskDevice{
			DiskName: &name,
		},
	}

	f := hclwrite.NewEmptyFile()
	block := gohcl.EncodeAsBlock(&app, `source "something" "something"`)
	f.Body().AppendBlock(block)
	fmt.Printf("%s", f.Bytes())
}

```

Will output:

```
source "something" "something" {
  packer_user_variables      = null
  packer_sensitive_variables = null
  image_name                 = "name"
  image_share_account        = null
  image_unshare_account      = null
  image_copy_regions         = null
  image_copy_names           = null
  tags                       = null
  tag                        = null
  system_disk_mapping        = { disk_category = null, disk_delete_with_instance = null, disk_descri
ption = null, disk_device = null, disk_encrypted = null, disk_name = "name", disk_size = null, disk_
snapshot_id = null }
  image_disk_mappings        = null
  ssh_remote_tunnels         = null
  ssh_local_tunnels          = null
  ssh_public_key             = null
  ssh_private_key            = null
}
```

This is a good first step for #9015 and #9282 

fix #9304
2020-05-28 11:19:00 +02:00
Adrien Delorme 0fa60c68fb
Drop the iso_checksum_type & iso_checksum_url fields (#8437)
* Drop the iso_checksum_type & iso_checksum_url fields

In favor of simply using iso_checksum that will know what to do.

* fix after master merge

* Update builder_test.go

* Update builder_test.go

* Update builder_test.go

* Update builder_test.go

* Update builder_test.go

* remove checksum lowercasing tests

* Update builder_test.go

* Update builder_test.go

* better docs

* Update builder_test.go

* even better docs

* Update config.go

* Update builder_test.go

* Update step_create_vmx_test.go

* make generate

* better docs

* fix imports

* up tests

* Update _ISOConfig-required.html.md

* Update builder_test.go

* don't use sha1.Sum("none") as a caching path

* Update builder_test.go

* better docs

* Update iso_config_test.go

remove ISOChecksumType/ISOChecksumURL references

* Update step_download_test.go

* add iso_checksum_url and iso_checksum_type fixers + tests

* add concrete examples of checksum values

* add examples of checksumming from local file

* update go-getter dep

* up deps

* use new go-getter version

* up ESX5Driver.VerifyChecksum: use go-getter's checksumming

* ISOConfig.Prepare: get checksum there in case we need it as a string in ESX5Driver.VerifyChecksum

* Update iso_config.go

* get go-getter from v2 branch

* Update driver_esx5.go

add more comments

* Update driver_esx5.go

* show better error message when the checksum is invalid

* Update builder_test.go

put in a valid checksum to fix tests, checksum is md5("packer")

* Update builder_test.go

test invalid and valid checksum

* more test updating

* fix default md5 string to be a valid md5

* TestChecksumFileNameMixedCaseBug: use 'file:' prefix for file checksumming

* Update iso_config_test.go

* Update iso_config_test.go

* Update builder_test.go

* Update builder_test.go

* Update builder_test.go

* Update CHANGELOG.md

* Update CHANGELOG.md

* Update go.mod

* Update go.mod

* Update CHANGELOG.md
2020-05-28 11:02:09 +02:00
Megan Marsh 7e0cfd8b77 add undocumented tag to struct-markdown generator so that we can leave internal-use-only fields out of docs 2020-05-01 10:31:38 -07:00
Adrien Delorme 840e67f775
HCL2: when we see a map generate an attribute spec instead of a block spec (#9035)
* mapstructure-to-hcl2: when we see a map generate an attribute spec and not a block spec

this will alow to do

tags = {
 key = "value"
}

instead of

tags {
  key = "value"
}

This will also enable using variables directly for those tags

* generate code

* update tests
2020-04-14 16:05:13 +02:00
Jeff Escalante 9165be87da
packer.io prefix removal, html extension remove for in-code errors 2020-04-07 17:53:22 -04:00
Jeff Escalante 4afd107eeb
patch source files for extensions changes 2020-04-07 17:50:51 -04:00
Sylvia Moss b17b211aa9
Add cleanup_remote_cache config option to vmware-iso (#8917) 2020-03-24 16:16:25 +01:00
r_takaishi f50ff1d270 make generate 2020-03-13 11:29:42 +09:00
Megan Marsh dde74232f2
Clearly document all template-engine-enabled options (#8770) 2020-02-20 11:05:27 +01:00
Jason A. Beranek 9cbb49bacb
builder/vmware/iso: Expand `guest_os_type` docs (#8760) 2020-02-18 11:11:20 +01:00
Moss 0566f1f999 Extract http ip discover to a new step 2020-01-27 17:06:56 +01:00
Sylvia Moss d6a351b173
Rename communicator config variables and remove deprecated code (#8584) 2020-01-09 08:37:30 -08:00
Adrien Delorme 4b7132c87c Merge remote-tracking branch 'origin/master' into sharing_info 2019-12-17 11:57:09 +01:00
Adrien Delorme 0785c2f6fc
build using HCL2 (#8423)
This follows #8232 which added the code to generate the code required to parse
HCL files for each packer component.

All old config files of packer will keep on working the same. Packer takes one
argument. When a directory is passed, all files in the folder with a name
ending with  “.pkr.hcl” or “.pkr.json” will be parsed using the HCL2 format.
When a file ending with “.pkr.hcl” or “.pkr.json” is passed it will be parsed
using the HCL2 format. For every other case; the old packer style will be used.

## 1. the hcl2template pkg can create a packer.Build from a set of HCL (v2) files

I had to make the packer.coreBuild (which is our one and only packer.Build ) a public struct with public fields

## 2. Components interfaces get a new ConfigSpec Method to read a file from an HCL file.

  This is a breaking change for packer plugins.

a packer component can be a: builder/provisioner/post-processor

each component interface now gets a `ConfigSpec() hcldec.ObjectSpec`
which allows packer to tell what is the layout of the hcl2 config meant
to configure that specific component.

This ObjectSpec is sent through the wire (RPC) and a cty.Value is now
sent through the already existing configuration entrypoints:

 Provisioner.Prepare(raws ...interface{}) error
 Builder.Prepare(raws ...interface{}) ([]string, error)
 PostProcessor.Configure(raws ...interface{}) error

close #1768


Example hcl files:

```hcl
// file amazon-ebs-kms-key/run.pkr.hcl
build {
    sources = [
        "source.amazon-ebs.first",
    ]

    provisioner "shell" {
        inline = [
            "sleep 5"
        ]
    }

    post-processor "shell-local" {
        inline = [
            "sleep 5"
        ]
    }
}

// amazon-ebs-kms-key/source.pkr.hcl

source "amazon-ebs" "first" {

    ami_name = "hcl2-test"
    region = "us-east-1"
    instance_type = "t2.micro"

    kms_key_id = "c729958f-c6ba-44cd-ab39-35ab68ce0a6c"
    encrypt_boot = true
    source_ami_filter {
        filters {
          virtualization-type = "hvm"
          name =  "amzn-ami-hvm-????.??.?.????????-x86_64-gp2"
          root-device-type = "ebs"
        }
        most_recent = true
        owners = ["amazon"]
    }
    launch_block_device_mappings {
        device_name = "/dev/xvda"
        volume_size = 20
        volume_type = "gp2"
        delete_on_termination = "true"
    }
    launch_block_device_mappings {
        device_name = "/dev/xvdf"
        volume_size = 500
        volume_type = "gp2"
        delete_on_termination = true
        encrypted = true
    }

    ami_regions = ["eu-central-1"]
    run_tags {
        Name = "packer-solr-something"
        stack-name = "DevOps Tools"
    }
    
    communicator = "ssh"
    ssh_pty = true
    ssh_username = "ec2-user"
    associate_public_ip_address = true
}
```
2019-12-17 11:25:56 +01:00
Megan Marsh 0f6d1beccf add an extra string array to the Prepare() return values in the builder interfaces; this sets up the ability for builders to give the provisioners custom user-accessible build-time variables. 2019-12-16 21:23:05 -08:00
Lars Lehtonen 6eefe55e23
builder/vmware/iso: remove unused testState() 2019-11-01 19:11:42 -07:00
Adrien Delorme 819329228a Change back to make sure all durations are a time.Duration
It is simply the best/simplest solution and trying to prevent users from passing and integer here would be like opening a can of worms. Because:

* we cannot make mapstructure validate our duration string ( with an UnmarshalJSON func etc.)
* we cannot make mapstructure spit a string instead of a duration and packer will decode-encode-decode config.
* the hcl2 generated code asks for a string, so this will be enforced by default.
2019-10-31 16:12:07 +01:00
Adrien Delorme bf3d9841c6 Force durations to be passed a strings
Before this commit it was possible to set a duration using an integer or a float. Go's time.Duration is an int64 internally an mapstructure will take advantage of this and load the number as a int64 but `1` means one ns which is unexpected/confusing. To avoid confusion and enforce readability this forces users to pass a string with a unit for a duration; ex "56s".
2019-10-31 11:47:19 +01:00
Adrien Delorme 078ba7c8c3 commit old code generation tool
for history
2019-10-15 12:56:42 +02:00
Adrien Delorme 00e3f97f5b builder.vmware.iso: remove duplicate Format field 2019-10-15 12:56:42 +02:00
Lars Lehtonen 2b795053b4
builder/vmware: fix dropped test errors 2019-09-27 16:05:12 -07:00
Megan Marsh c24693372d
Merge pull request #8132 from scienty/support_floppy_vmx_esxi
Support floppy_files for vmx on esxi
2019-09-23 12:20:49 -07:00
Adrien Delorme 3fe9d52e3f Merge remote-tracking branch 'origin/master' into scrape_doc_to_builder_struct_config 2019-09-20 11:15:44 +02:00
Prakash Sidaraddi 67be007f45 Support floppy_files for vmx on esxi 2019-09-20 00:23:28 +05:30
allinwait 171c2a9a07 add the floppy label option 2019-09-12 12:25:22 +00:00
Adrien Delorme 4a61dc3670 vmware: document using partials 2019-06-20 15:28:27 +02:00
Adrien Delorme 926c8bbaa6 refactor ShutdownConfig into a single struct 2019-06-14 12:17:28 +02:00
Adrien Delorme 4399684372 make fmt autogenerated docs 2019-06-06 16:29:25 +02:00
Adrien Delorme e6cbb013ba add // go:generate struct-markdown to all previously edited files 2019-06-05 16:42:18 +02:00
Adrien Delorme f1917edd34 generate the comments for config struct of builders scraping doc website 2019-06-05 16:42:17 +02:00
Adrien Delorme c4f3dccc14 rename interpolation context from ctx to ictx and contexts to ctx to avoid conflicts 2019-04-03 15:56:15 +02:00
Adrien Delorme a4bf94dd3c change Builder to be passed a context for cancellation
we have to to give it to our hook
2019-04-03 15:55:55 +02:00
Adrien Delorme 9f82b75e57 Use the hashicorp/go-getter to download files
* removed packer.Cache and references since packer.Cache is never used except in the download step. The download step now uses the new func packer.CachePath(targetPath) for this, the behavior is the same.
* removed download code from packer that was reimplemented into the go-getter library: progress bar, http download restart, checksuming from file, skip already downloaded files, symlinking, make a download cancellable by context.
* on windows if packer is running without symlinking rights and we are getting a local file, the file will be copied instead to avoid errors.
* added unit tests for step_download that are now CI tested on windows, mac & linux.
* files are now downloaded under cache dir `sha1(filename + "?checksum=" + checksum) + file_extension`
* since the output dir is based on the source url and the checksum, when the checksum fails, the file is auto deleted.
* a download file is protected and locked by a file lock,
* updated docs
* updated go modules and vendors
2019-03-13 12:11:58 +01:00
Megan Marsh 0371e2a1b4
Merge pull request #7074 from hashicorp/fix_3329
vmware-iso (esxi) Add check that "format" is not being set when remote_type is empty.
2019-02-28 15:30:54 -08:00
Megan Marsh 6ea64bc378
Merge pull request #7191 from arizvisa/GH-7190
Adds support to the vmware builders for specifying the number of cores per socket via the `cores` option.
2019-01-31 15:27:52 -08:00
Ali Rizvi-Santiago 4894873971 Added logic to the vmware-iso builder that appends the number of cores per socket to the vmx in `stepCreateVmx`. 2019-01-17 10:39:19 -06:00
Adrien Delorme 9f7b4ffc17
Merge branch 'master' into pr/6950 2019-01-09 10:11:18 +01:00
Megan Marsh 7d5ea43f41 revert the hddorder being added to the vmx template, since it's breaking things for some of our users. 2018-12-13 09:58:24 -08:00
Adrien Delorme b7222d1f56 Add tmp package that offers Dir & File funcs
this regroups the calls of tmp func calls into one package.
the tmp pkg allows to store tmp files under a common directory for easier managment

Squashed commit of the following:
commit 74b674c01560c6b547e9cab5e8afb486f0fe7d6c
Merge: fc94b5714 66001525d
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Fri Nov 2 02:02:38 2018 -0400

    Merge branch 'remove_TMPDIR_docs' of github.com:tb3088/packer into remove_TMPDIR_docs

commit fc94b571428447aaf2e5171967c95cdf9e5c73ff
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Fri Nov 2 01:42:38 2018 -0400

    update tests to honor configured temporary directory

commit 5a562359f4937af506607dd96ed1c46426093f52
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Thu Nov 1 18:52:38 2018 -0400

    sync straggler to refactor. implement tests at packer/configfile instead of buried in docker.

commit 9d5a4a0935bfd049e2f8f50ad73e4455b42c9410
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Thu Nov 1 15:39:46 2018 -0400

    insert URL to GoLang API for os.TempDir()

commit fbae4c2e051898aecd8705e9d8a20353430ede23
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Thu Nov 1 07:41:29 2018 -0400

    revise documentation to match code

commit 3590fae8bd0afa92a161a49ed794ee32e357d583
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Thu Nov 1 07:21:59 2018 -0400

    refacter config_file. replace all hard-coded os.TempDir with wrapper

commit d5c5306a97b648ded06a8eb76ce728f07c9924f0
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Wed Oct 31 20:11:36 2018 -0400

    close massive file overwrite hole with TempDir

commit 0a72297da86089fbd07e4f7d9472a92fa4d1c733
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Wed Oct 31 01:06:00 2018 -0400

    adjust var declaration scope

commit 20f68228b6c372d984ea5b055cfc8bda9b110ac5
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Wed Oct 31 00:54:35 2018 -0400

    use mktemp() equivalent to create temporary directory

commit c73ebe3d8321664a2685a0baa8e441dd895b0db4
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Tue Oct 30 20:40:58 2018 -0400

    remove extraneous variable declaration, fix FOR loop

commit 63549b8bd745a0090b15ed0b0ebf644162db27db
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Tue Oct 30 19:30:44 2018 -0400

    match styistic convension with rest of docs

commit 976101074942db36f10d3a686d6d29ddb7c01926
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Tue Oct 30 19:08:39 2018 -0400

    revert dangling config file change

commit 95159afbc05eac96ed11c3183af5fb0abe2f2d5c
Author: Matthew Patton <matthew.patton@itgfirm.com>
Date:   Fri Dec 29 23:53:43 2017 -0500

    replace invalid TMPDIR variable wth PACKER_TMP_DIR.
    update ConfigTmpDir() to try common temporary paths first and
    only write to configDir() as a last resort.

commit 66001525d72de56a4cf9339b900c46f59bc6e91a
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Fri Nov 2 01:42:38 2018 -0400

    update tests to honor configured temporary directory

commit e9b6adefeae8c65eb8aa47fef38cbf0aa424338c
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Thu Nov 1 18:52:38 2018 -0400

    sync straggler to refactor. implement tests at packer/configfile instead of buried in docker.

commit 852113ed076e2d14e5dca6815ea680da1e2896bb
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Thu Nov 1 15:39:46 2018 -0400

    insert URL to GoLang API for os.TempDir()

commit 54add38d1d1c6e283cd444b367ed8bd49a5f3699
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Thu Nov 1 07:41:29 2018 -0400

    revise documentation to match code

commit 6b5b8f6d4edad0e187ca13d64ac4118f34eee643
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Thu Nov 1 07:21:59 2018 -0400

    refacter config_file. replace all hard-coded os.TempDir with wrapper

commit c22092c601c33484327674f322c7379fa41506d7
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Wed Oct 31 20:11:36 2018 -0400

    close massive file overwrite hole with TempDir

commit 7a730450916bf8e5dbc2a741ec233a49466ab7cc
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Wed Oct 31 01:06:00 2018 -0400

    adjust var declaration scope

commit 0f2933adb6f6922dfeab78a95371a444ec8918ab
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Wed Oct 31 00:54:35 2018 -0400

    use mktemp() equivalent to create temporary directory

commit d74839ede05dacf712b55a7bb48aec19fe6b007f
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Tue Oct 30 20:40:58 2018 -0400

    remove extraneous variable declaration, fix FOR loop

commit eb65416619437e4a3dec90277770803dd5b2281c
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Tue Oct 30 19:30:44 2018 -0400

    match styistic convension with rest of docs

commit acaa2b31ed463219c4ef099f351eec72406e2989
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Tue Oct 30 19:08:39 2018 -0400

    revert dangling config file change

commit e573fde668dcb418561e61535c1d68b2888f5b0f
Author: Matthew Patton <matthew.patton@itgfirm.com>
Date:   Fri Dec 29 23:53:43 2017 -0500

    replace invalid TMPDIR variable wth PACKER_TMP_DIR.
    update ConfigTmpDir() to try common temporary paths first and
    only write to configDir() as a last resort.

commit 39a9874afc9dea71b36753375fb7feb61ffecf69
Merge: 8a413cfe8 3315812c2
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Tue Oct 30 18:19:26 2018 -0400

    Merge branch 'master' of https://github.com/hashicorp/packer into prestine

commit 8a413cfe83b0a70dbd99c32d936334ca5788ca9b
Merge: e07491de5 4e14710a6
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Mon Oct 1 20:18:10 2018 -0400

    Merge remote-tracking branch 'upstream/master' into prestine

commit e07491de59cead0b337a7f57f4a6c625e1f560ab
Merge: 42610a35d a1fa35dff
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Tue Aug 21 13:26:19 2018 -0400

    Merge remote-tracking branch 'upstream/master' into prestine

commit 42610a35d5ef65af8844a58ad70e2ec75262da6a
Merge: 5298142da 0d63cf7bc
Author: Matthew Patton <matthew.patton@itgfirm.com>
Date:   Tue Jun 19 22:45:05 2018 -0400

    Merge remote-tracking branch 'upstream/master' into prestine

commit 5298142da6da95354844f201deeeef3c599e48b2
Merge: 7bb110bc7 9d9736552
Author: Matthew Patton <mpatton@enquizit.com>
Date:   Mon Jun 11 15:10:09 2018 -0400

    Merge remote-tracking branch 'upstream/master' into prestine

commit 7bb110bc744c9797fcaec25a265d9b85a22d6d6b
Merge: a61c869ca 6189d66e7
Author: Matthew Patton <matthew.patton@itgfirm.com>
Date:   Wed May 9 23:41:22 2018 -0400

    Merge remote-tracking branch 'upstream/master' into prestine

commit a61c869ca7268acf0e3b1e1fa7a8ee7feea65984
Merge: 098101dd9 4be5f072c
Author: Matthew Patton <matthew.patton@itgfirm.com>
Date:   Fri May 4 10:13:04 2018 -0400

    Merge remote-tracking branch 'upstream/master' into prestine

commit 098101dd99f08b0ca110d33eff6904537c6d21de
Merge: fefaf0fa6 554b2b4a5
Author: Matthew Patton <matthew.patton@itgfirm.com>
Date:   Tue May 1 20:52:30 2018 -0400

    Merge branch 'EOL-handling' into prestine

Co-Authored-By: Matthew Patton <pattonme@yahoo.com>
2018-12-12 16:35:57 +01:00
Megan Marsh dbfc1b5aff fix tests 2018-12-06 09:36:16 -08:00
Megan Marsh 0fe4d79bbf add validation to prevent users from being confused about format functionality 2018-12-05 15:57:45 -08:00
Ali Rizvi-Santiago e0d3861e81 Updated StepCreateVMX to add the CDROM device that packer uses to the "temporaryDevices" statebag so that StepCleanVMX can unmount its ISO later. 2018-12-03 17:48:05 -06:00