Commit Graph

59 Commits

Author SHA1 Message Date
Adrien Delorme 9f82b75e57 Use the hashicorp/go-getter to download files
* removed packer.Cache and references since packer.Cache is never used except in the download step. The download step now uses the new func packer.CachePath(targetPath) for this, the behavior is the same.
* removed download code from packer that was reimplemented into the go-getter library: progress bar, http download restart, checksuming from file, skip already downloaded files, symlinking, make a download cancellable by context.
* on windows if packer is running without symlinking rights and we are getting a local file, the file will be copied instead to avoid errors.
* added unit tests for step_download that are now CI tested on windows, mac & linux.
* files are now downloaded under cache dir `sha1(filename + "?checksum=" + checksum) + file_extension`
* since the output dir is based on the source url and the checksum, when the checksum fails, the file is auto deleted.
* a download file is protected and locked by a file lock,
* updated docs
* updated go modules and vendors
2019-03-13 12:11:58 +01:00
Adrien Delorme 541c68aed5 add StackableProgressBar struct that will refresh/show dl status for multiple files
* simplified the downloader interface, and removed the total/current values from them
* downloaders use a proxy reader that will add all read bytes to progress
* removed unused const mtu
* DownloadClient doesn't need a downloader, so I removed it too
2018-09-06 15:53:59 +02:00
Adrien Delorme ddd96c513b first draft at self refreshing loading bar centralized/controlled by Ui 2018-09-06 15:53:59 +02:00
Megan Marsh 1f79b430ee
Merge pull request #6610 from hashicorp/filter_logs
Filter logs
2018-08-23 13:30:21 -07:00
Ali Rizvi-Santiago 0f10032b3d Moved the progress bar from common to the packer.UI interface and refactored it so that the terminal width is calculated based on each interface which returns a custom progressbar specific to its ui. 2018-08-20 21:55:18 -05:00
Ali Rizvi-Santiago 16ecb3ad9a Reverted removal of progress-bar that was done by commit 5d97b105a8 and added some missing arguments that were missed during the rebase. Modified the default progress bar's width to 80 as a result of the conversation on PR #5851. 2018-08-20 21:55:18 -05:00
Megan Marsh ff6a039d5b replace scrubconfig with packer.LogSecretFilter.Set
filter winrm password from logs
Add new root-level packer template option, sensitive-variables, to tell us what user variables to mark sensitive.
2018-08-20 15:35:55 -07:00
Ali Rizvi-Santiago c366a1e160 Inverted the logic of FileExistsLocally as suggested by @SwampDragons as remote URLs are assumed to exist locally. 2018-02-02 20:17:24 -06:00
Ali Rizvi-Santiago c98a074f0d Renamed common/config.go's SupportedURL to SupportedProtocol as suggested by @SwampDragons. 2018-02-02 18:58:42 -06:00
Ali Rizvi-Santiago 97fc9c02a5 Grr...missed the case that actually mattered on linux. 2018-01-18 23:58:24 -06:00
Ali Rizvi-Santiago f9572cb244 Fixed a bug on linux related to forgetting to check the platform for the forward-slash prefix. 2018-01-18 23:48:20 -06:00
Ali Rizvi-Santiago 41f4dc3f3d umm...gofmt -w on common/config{,_test}.go from linux instead of windows(?) 2018-01-18 23:33:44 -06:00
Ali Rizvi-Santiago 7cd5d576d9 Updated common/config.go's FileExistsLocally implementation to use the LocalDownloader interface for determining the real file path. 2018-01-18 23:18:55 -06:00
Ali Rizvi-Santiago 8a102a42a0 gofmt -w on common/config{,_test}.go 2018-01-18 23:09:53 -06:00
Ali Rizvi-Santiago 95f60f6153 Modified common/config.go to accommodate some of the new DownloadableURL policies made by the PR #5761 merge.
common/config.go:
    Added the ability for DownloadableURL to promote UNC paths to the SMB uri.
    Modified DownloadableURL to include the "./" prefix when a relative path is passed to it.
    Fix-up the DownloadableURL argument if on windows and incorrectly prefixed with "/".
2018-01-18 22:43:08 -06:00
Ali Rizvi-Santiago 50e9cd2ca7 Initial fixes of common/config.go after rebase before refactoring of test-cases so that they don't require root to run. 2018-01-16 13:46:27 -06:00
Ali Rizvi-Santiago c17f827e1d Split up DownloadableURL() into it's individual components: SupportedURL(), DownloadableURL(), and ValidatedURL(). Updated all instances of DownloadableURL() to point to ValidatedURL(). Reverted the tests that are based on un-supported protocols. 2018-01-16 13:37:32 -06:00
Ali Rizvi-Santiago 5d97b105a8 Removed implementation of the ftp protocol and the usage of cheggaaa's progress-bar as suggested by @SwampDragons. Replaced some of the old smoke-tests that were based on the ftp-protocol non-existing with a "non-existent://" protocol that's guaranteed to not exist. 2018-01-16 13:37:31 -06:00
Ali Rizvi-Santiago b1ff14714b go fmt 2018-01-16 13:37:31 -06:00
Ali Rizvi-Santiago 2f1104625d Fixed some of the unit-tests in common/ due to the changes made in {config,download}.go
config.go:
Fixed some issues related to the url scheme not being lowercased which broke some of the tests.

config_test.go:
Removed the UNC share test for \\host\share\file since SMB support has been moved to a different uri scheme.

download_test.go:
Explicitly set the CopyFile configuration option for all the unit-tests that test file copying capability.
Removed the UNC share testcase since it's under a different uri scheme now.
Modified the file:// UNC share testcase to explicitly test the smb:// uri.
Changed the incorrect t.Errorf calls to t.Logf so that the tests can pass.
2018-01-16 13:37:30 -06:00
Ali Rizvi-Santiago 6170e24ecb Refactored the code a bit to move the CopyFile hack out of DownloadClient and instead into each protocol.
config.go:
Removed all of the windows-specific net/url hackery since it's now handled mostly by download.go
Removed the replacement of '\' with '/' since url.Parse does it now.
Added knowledge of the other protocols implemented in download.go (ftp, smb)
Removed some modules that were unused in this commit.

download.go:
Moved the file-path conversions for the different protocols into their own internally callable functions.
Shuffled some of the functions around in case someone wants to implement the ability to resume.
Modified DownloadClient.Get to remove the CopyFile special case and trust the protocol implementations if a user doesn't want to copy the file.
Since all the protocols except for HTTPDownloader implement Cancel, added a Resume method as a placeholder for another developer to implement.
Added a few missing names from their function definitions.
Fixed the syntax in a few lines due to my suckage at go.
Adjusted the types for progress and total so that they support 64-bit sizes.
Removed the usage of the bufio library since it wasn't really being used.
2018-01-16 13:37:30 -06:00
Ali Rizvi-Santiago 281dd1258a Added proper support for downloading via a Windows UNC path or a relative uri.
Added proper support for validating a downloadableURL containing a UNC or relative uri.
Removed the workaround for an earlier Go issue that had remained dormant in common/download.go (issue #5927).
When building a .vmx file via the vmware-iso builder, transform the path to the correct os-formatted one (using filepath.FromSlash).
2018-01-16 13:36:17 -06:00
Megan Marsh 3ace5bb91b simplify FileExistsLocally 2018-01-10 16:11:17 -08:00
Megan Marsh 55ddbf4765 sloppy copypasta 2018-01-10 10:08:23 -08:00
Megan Marsh 898dadd53c re-add this block. I still don't think we need it but I don't want to risk breaking things with this bugfix. 2018-01-10 10:03:36 -08:00
Megan Marsh 40f0cc6dfe I don't think this is needed anymore 2018-01-09 15:53:54 -08:00
Megan Marsh 2838a2371d disambiguate url variable from url library 2018-01-09 15:47:18 -08:00
Megan Marsh 216c44b153 fix FileExistsLocally 2018-01-09 15:47:07 -08:00
Megan Marsh 4f3b470804 add helper function to manage validation of filepaths created using DownloadableURL 2018-01-03 16:53:47 -08:00
Megan Marsh 54bd057bb9 fix nasty edge case where we can't find guest additions on windows if they are on a different drive 2018-01-03 14:34:11 -08:00
Megan Marsh 0efcb1bba2 dont error in the downloadableURL function; save validation for preflight steps 2017-11-13 12:42:57 -08:00
Megan Marsh 3a9dfb5b18 better 2017-11-13 12:42:57 -08:00
Megan Marsh e45a006d61 clearly state that url is wrong at validation stage of build 2017-11-13 12:42:57 -08:00
Chris Bednarski ee2d636840 Improve delay between key events
- Can now tune delay using PACKER_KEY_INTERVAL
- Added implementation to all of VMware and QEMU
- Removed double delay for QEMU
- Default key delay of 100ms (as before)
- Added docs to QEMU and VMware pages
2017-01-14 17:56:04 -08:00
Mitchell Hashimoto 1e853f9f1f common: revert some changes from #2121 for Windows 2015-06-13 10:21:09 -04:00
Mitchell Hashimoto 44008c321b Merge pull request #2121 from josharian/no-scrub-empty
common: two minor fixes
2015-05-27 20:17:51 -07:00
Mitchell Hashimoto adb6b43dd8 common: remove unused config methods 2015-05-27 14:58:09 -07:00
Josh Bleecher Snyder 2fe785ed35 common: remove dead code
The referenced bug was fixed in Go 1.2,
and Packer requires Go 1.2+.
2015-05-18 15:13:36 -07:00
Josh Bleecher Snyder 76c8cfd498 common: don't scrub ""
If the access_key or secret_key were loaded from
somewhere other than the packer file then
ScrubConfig can get called to scrub "" and "".

This results in very long output:

<Filtered><<Filtered>F<Filtered>i...

Don't do that.
2015-05-18 15:13:01 -07:00
Grégoire Pineau 97c56347a1 Better error reporting when a config key in template is Unknown
This patch will allow to fix the following bug much faster:

```
1 error(s) occurred:

* Unknown configuration key: output_directory
```

Related configuration:

```
"output_directory ": "build/sl_base/",
```

After the patch, the error reporting will be:

```
1 error(s) occurred:

* Unknown configuration key: "output_directory¤"
```
2015-03-05 10:23:21 +01:00
Mitchell Hashimoto 5c5d62733f fmt 2014-05-01 14:24:19 -07:00
Mitchell Hashimoto 5cd2cfa563 comment so future-mitchell knows what I've done 2014-04-28 16:19:35 -07:00
Mitchell Hashimoto ba05119a75 common: user variable conversion to non-string types works [GH-1079] 2014-04-28 16:18:45 -07:00
Mitchell Hashimoto d5981c69f2 common: config strings to slices [GH-950] 2014-04-27 14:47:16 -07:00
Mitchell Hashimoto 5dec2ddb9c common: weakly decode the PackerConfig 2014-04-26 14:23:22 -07:00
Mitchell Hashimoto 5fac6c79c4 fmt 2014-04-26 11:12:43 -07:00
Mitchell Hashimoto 8395d0e97a common: tests for ChooseSTring 2014-04-21 21:30:49 -07:00
Mitchell Hashimoto 7191c1f250 common: Fix URL parsing issues on Windows 2014-04-21 21:28:47 -07:00
Devin Carlen bef63846af Add support for standard OpenStack environment variables 2014-02-17 15:30:01 -08:00
Mitchell Hashimoto 4c6800f5a3 common: process user variables in non-string config decodes [GH-598] 2013-12-16 17:57:07 -08:00