Commit Graph

2280 Commits

Author SHA1 Message Date
Mitchell Hashimoto 2da5930556 Merge pull request #701 from mitchellh/rpc-refactor
RPC happens over single TCP connection per plugin now

Three benefits:
* Single file descriptor per plugin
* NAT-friendly since plugins don't have to dial back in to the host
* Opens the foundation that we can easily use Unix domain sockets and such

A handful of Packer users were having issues with highly parallel (many builder/provisioner) templates where their systems would quickly reach their default file descriptor limits. This was because the previous mechanism would use a single TCP connection per RPC server, and Packer needs many (one per interface, basically). 

This merges in a MuxConn that multiplexes many "streams" on top of a single io.ReadWriteCloser. The RPC system has been revamped to know about this and use unique stream IDs to send everything over a single connection per plugin.

Previously, the RPC mechanism would sometimes send an address to the remote end and expect the remote end to connect back to it. While Packer shouldn't run remotely, some firewalls were having issues. This should be gone.

Finally, it should be possible now to optimize and use Unix domain sockets on Unix systems, avoiding ports and firewalls altogether.
2013-12-10 17:19:25 -08:00
Mitchell Hashimoto 958f4191cd Test on Go 1.2 2013-12-10 17:10:41 -08:00
Mitchell Hashimoto 8a24c9b177 packer/rpc: fix data race in MuxConn 2013-12-10 17:09:17 -08:00
Mitchell Hashimoto f79daa0b1b packer/rpc: edge-triggerd state changes for faster dial/accept 2013-12-10 17:01:02 -08:00
Mitchell Hashimoto d9f79b0ecc packer/rpc: hard close all streams when underlying conn closes 2013-12-10 16:49:14 -08:00
Mitchell Hashimoto 5dffab7439 packer/rpc: need a real lock for closing 2013-12-10 16:23:47 -08:00
Mitchell Hashimoto 7372c32b6b packer/rpc: implement proper close_wait state 2013-12-10 15:51:22 -08:00
Mitchell Hashimoto e4dbad330d packer/rpc: rename uploadReader to uploadData because that makes sense 2013-12-10 15:30:58 -08:00
Mitchell Hashimoto 3a41500888 packer/rpc: more robust communicator connection cleanup 2013-12-10 15:12:16 -08:00
Mitchell Hashimoto 4c5d61709d packer/plugin: catch interrupts for every server 2013-12-10 14:12:00 -08:00
Mitchell Hashimoto 06d12773eb packer/rpc: improve logging for the MuxConn 2013-12-10 14:11:50 -08:00
Mitchell Hashimoto 82bf5fc79e plugin/*: update to latest API 2013-12-10 13:59:38 -08:00
Mitchell Hashimoto b4567c6380 packer/plugin: use new RPC API 2013-12-10 13:47:18 -08:00
Mitchell Hashimoto ce2304c948 packer/rpc: remove unused methods 2013-12-10 13:26:07 -08:00
Mitchell Hashimoto 8d4ba1fc2b packer/rpc: complete command 2013-12-10 13:23:07 -08:00
Mitchell Hashimoto a8b056e939 packer/rpc: builds 2013-12-10 13:18:48 -08:00
Mitchell Hashimoto bd6fbc05eb packer/rpc: environment 2013-12-10 12:23:42 -08:00
Mitchell Hashimoto 2ba713d705 packer/rpc: Builder 2013-12-10 12:14:08 -08:00
Mitchell Hashimoto e69399380e packer/rpc: start command 2013-12-10 12:02:01 -08:00
Mitchell Hashimoto 5966a6e905 packer/rpc: Provisioner 2013-12-10 11:56:15 -08:00
Mitchell Hashimoto a036bec96e packer/rpc: Hook 2013-12-10 11:50:30 -08:00
Mitchell Hashimoto db06fc7501 packer/rpc: implement Communicator 2013-12-10 11:43:02 -08:00
Mitchell Hashimoto 72fcb566a6 packer/rpc: better close states 2013-12-10 11:40:17 -08:00
Mitchell Hashimoto af22b35a1f packer/rpc: MuxConn writes don't block the whole loop 2013-12-10 10:44:57 -08:00
Mitchell Hashimoto 68e51de0f8 packer/rpc: MuxConn.NextId properly increments 2013-12-10 10:34:35 -08:00
Mitchell Hashimoto 2ac629c949 packer/rpc: get PostProcessor working 2013-12-09 19:07:36 -08:00
Mitchell Hashimoto ec68a3fd39 packer/rpc: MuxConn can return next available stream ID 2013-12-09 16:27:13 -08:00
Mitchell Hashimoto 171781c3c6 packer/rpc: work-in-progress commit 2013-12-09 16:22:11 -08:00
Mitchell Hashimoto 4ba5c2ef46 packer/rpc: post-processors work on a single connection 2013-12-09 15:44:00 -08:00
Mitchell Hashimoto a32cd59c29 packer/rpc: PostProcessor 2013-12-09 14:57:18 -08:00
Mitchell Hashimoto 984dd224f3 packer/rpc: Cache 2013-12-09 14:51:13 -08:00
Mitchell Hashimoto e9f7a1418c packer/rpc: use packer.MockArtifact 2013-12-09 14:46:33 -08:00
Mitchell Hashimoto 105e5f6a6d packer/rpc: tests passing 2013-12-09 14:44:26 -08:00
Mitchell Hashimoto 61fd3f7333 packer/rpc: update docs 2013-12-09 14:29:28 -08:00
Mitchell Hashimoto 50cfb67863 packer/rpc: muxconn is a lot more sane, acts like bsd socket 2013-12-09 14:24:55 -08:00
Mitchell Hashimoto 36a47f5b59 packer/rpc: more fine grained lock access on MuxConn 2013-12-08 18:39:14 -08:00
Mitchell Hashimoto 5c6831080c packer/rpc: close the streams when the underlying rwc closes 2013-12-08 18:30:29 -08:00
Mitchell Hashimoto fe46093bcf packer/rpc: a muxconn... 2013-12-08 18:20:27 -08:00
Mitchell Hashimoto a66f148ede packer/rpc: auto-incrementable ID for endpoints 2013-12-08 11:50:14 -08:00
Mitchell Hashimoto 73d691d319 up version for dev 2013-12-08 10:19:13 -08:00
Mitchell Hashimoto c6e969e41b v0.4.1 2013-12-07 21:05:53 -08:00
Mitchell Hashimoto 2789281285 builder/vmware: clarify guestiphack error message [GH-639] 2013-12-07 10:31:57 -08:00
Mitchell Hashimoto 956f49f439 provisioner/puppet-masterless: check for hiera config path [GH-656] 2013-12-06 19:30:06 -08:00
Mitchell Hashimoto 2b4bdee0a2 website: fix docs for vmware esxi discrep [GH-674] 2013-12-06 19:14:20 -08:00
Mitchell Hashimoto 8766a93245 builder/vmware: correct logic in cehcking if file is still there 2013-12-06 19:12:49 -08:00
Mitchell Hashimoto 6211cd2028 builder/vmware: more resilient to racing deletes [GH-675] 2013-12-06 19:11:40 -08:00
Mitchell Hashimoto b79506e623 Update CHANGELOG 2013-12-06 19:05:19 -08:00
Mitchell Hashimoto f7b1c2dbc6 builder/amazon/instance: add location to upload command [GH-679] 2013-12-06 19:04:40 -08:00
Mitchell Hashimoto 48fb1f9dfb common: fix tests 2013-12-06 18:36:16 -08:00
Mitchell Hashimoto 2110edb690 builder/amazon/chroot: remove unused import 2013-12-06 18:32:21 -08:00