HBASE-25375 Provide a VM-based release environment (#2754)
This adds a Vagrantfile and supporting automation that creates a virtual machine environment suitable for running the create-release scripting. Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Michael Stack <stack@apache.org>
This commit is contained in:
Nick Dimiduk
GitHub
parent
3488c44a21
commit
3cc2468300
|
|
@ -0,0 +1,3 @@
|
|||
.vagrant/
|
||||
*.log
|
||||
*.patch
|
||||
|
|
@ -0,0 +1,141 @@
|
|||
<!---
|
||||
Licensed to the Apache Software Foundation (ASF) under one
|
||||
or more contributor license agreements. See the NOTICE file
|
||||
distributed with this work for additional information
|
||||
regarding copyright ownership. The ASF licenses this file
|
||||
to you under the Apache License, Version 2.0 (the
|
||||
"License"); you may not use this file except in compliance
|
||||
with the License. You may obtain a copy of the License at
|
||||
|
||||
http://www.apache.org/licenses/LICENSE-2.0
|
||||
|
||||
Unless required by applicable law or agreed to in writing,
|
||||
software distributed under the License is distributed on an
|
||||
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
|
||||
KIND, either express or implied. See the License for the
|
||||
specific language governing permissions and limitations
|
||||
under the License.
|
||||
-->
|
||||
|
||||
# HBase Release Env
|
||||
|
||||
This is a vagrant project that provides a virtual machine environment suitable
|
||||
for running an Apache HBase release.
|
||||
|
||||
Requires:
|
||||
* [VirtualBox](http://virtualbox.org)
|
||||
* [Vagrant](http://virtualbox.org)
|
||||
* The private portion of your signing key avilable in the local GPG agent
|
||||
* The private portion of your Github authentication key available in either the local GPG agent or
|
||||
local SSH agent
|
||||
|
||||
## Usage
|
||||
|
||||
Unlock the local keyring before proceeding (this should prompt you for your GPG passphrase). For
|
||||
example, assuming you have an authentication key configured in your keyring, this will do the
|
||||
trick.
|
||||
|
||||
All terminal commands used below are assumed to be run with the current working directory as the
|
||||
location containing the `Vagrantfile`.
|
||||
|
||||
The term "Host" is used to mean the environment that runs the Vagrant process. The term "Guest" is
|
||||
used to mean the virtual machine managed by the Host.
|
||||
|
||||
### Ensure credentials work from the Host OS
|
||||
|
||||
The ssh- and gpg-agent forwarding configuration used here assumes that your credentials work
|
||||
on the Host. Verify both are working before you proceed with the Guest. Additionally, using the
|
||||
credentials requires you to unlock the respective keyring, the state of which is persisted by the
|
||||
agent process or processes running on the Host.
|
||||
|
||||
See instructions in [`create-release`](../create-release/README.txt) regarding proper
|
||||
configuration of ssh- and gpg-agents.
|
||||
|
||||
Assuming the git repo origin is on GitHub, the following command will ensure that your ssh
|
||||
credentials are working. On the Host, run:
|
||||
|
||||
```sh
|
||||
host:~$ ssh -T git@github.com
|
||||
Hi <you>! You've successfully authenticated, but GitHub does not provide shell access.
|
||||
```
|
||||
|
||||
Likewise, ensure you have an encryption key that can be used to sign a file. Again, on the Host,
|
||||
run:
|
||||
|
||||
```sh
|
||||
host:~$ gpg --detach --armor --sign Vagrantfile
|
||||
host:~$ gpg --verify Vagrantfile.asc
|
||||
gpg: assuming signed data in 'Vagrantfile'
|
||||
...
|
||||
host:~$ rm Vagrantfile.asc
|
||||
```
|
||||
|
||||
### Make public keyring available to the VM
|
||||
|
||||
Export the public portion of your signing credentials where the Guest can access it. Vagrant
|
||||
(+VirtualBox) shares the directory of the `Vagrantfile` with the Linux Guest via the `/vagrant`
|
||||
mount point. Any files present in this working directory on the Host are available to the Guest.
|
||||
|
||||
From the Host, run:
|
||||
|
||||
```sh
|
||||
host:~$ gpg --export <you>@apache.org > gpg.<you>.apache.pub
|
||||
```
|
||||
|
||||
### Launch the Guest VM
|
||||
|
||||
Launch the Guest VM by running
|
||||
|
||||
```sh
|
||||
host:~$ vagrant up
|
||||
```
|
||||
|
||||
If anything about the Vagrant or VirtualBox environment have changed since you last used this VM,
|
||||
it's best to `vagrant destroy -f` all local state and `vagrant up` a fresh instance.
|
||||
|
||||
### Verify the Guest VM
|
||||
|
||||
Connect to the Guest. This should forward your ssh- and gpg-agent session, as configured in the
|
||||
`Vagrantfile`.
|
||||
|
||||
```sh
|
||||
host:~$ vagrant ssh
|
||||
```
|
||||
|
||||
Now that you're in the Guest VM, be sure that all `gpg` command you issue include the
|
||||
`--no-autostart` flag. This ensures that the `gpg` process in the Guest communicates with the
|
||||
agent running on the Host OS rather than launching its own process on the Guest OS.
|
||||
|
||||
From the Guest, verify that ssh-agent forwarding is working, using the same test performed on the
|
||||
Host,
|
||||
|
||||
```sh
|
||||
guest:~$ ssh -T git@github.com
|
||||
Hi <you>! You've successfully authenticated, but GitHub does not provide shell access.
|
||||
```
|
||||
|
||||
From the Guest, import your exported public identity and verify the gpg-agent passthrough is
|
||||
working correctly.
|
||||
|
||||
```sh
|
||||
guest:~$ gpg --no-autostart --import /vagrant/gpg.<you>.apache.pub
|
||||
...
|
||||
gpg: Total number processed: 1
|
||||
gpg: imported: 1
|
||||
guest:~$ gpg --no-autostart --detach --armor --sign repos/hbase/pom.xml
|
||||
guest:~$ gpg --no-autostart --verify repos/hbase/pom.xml.asc
|
||||
gpg: assuming signed data in 'repos/hbase/pom.xml'
|
||||
...
|
||||
guest:~$ rm repos/hbase/pom.xml.asc
|
||||
```
|
||||
|
||||
### Build a Release Candidate
|
||||
|
||||
Finally, you can initiate the release build. Follow the instructions in
|
||||
[`create-release`](../create-release/README.txt), i.e.,
|
||||
|
||||
```sh
|
||||
guest:~$ mkdir ~/build-2.3.1-rc0
|
||||
guest:~$ cd repos/hbase
|
||||
guest:~/repos/hbase$ ./dev-support/create-release/do-release-docker.sh -d ~/build-2.3.1-rc0/ ...
|
||||
```
|
||||
|
|
@ -0,0 +1,50 @@
|
|||
# -*- mode: ruby -*-
|
||||
# vi: set ft=ruby :
|
||||
#
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
Vagrant.configure("2") do |config|
|
||||
|
||||
config.vm.define "rmvm" do |rmvm|
|
||||
rmvm.vm.box = "ubuntu/focal64"
|
||||
rmvm.vm.hostname = "rmvm"
|
||||
|
||||
rmvm.vm.provision "shell", path: "provision/focal.sh", run: "once"
|
||||
|
||||
rmvm.vm.provision "puppet", run: "always" do |puppet|
|
||||
puppet.environment = "production"
|
||||
puppet.environment_path = "puppet"
|
||||
puppet.working_directory = "/tmp/vagrant-puppet"
|
||||
puppet.options = "--test"
|
||||
end
|
||||
|
||||
rmvm.vm.provider "virtualbox" do |vb|
|
||||
vb.name = "rmvm"
|
||||
vb.cpus = 2
|
||||
vb.memory = (4 * 1024).to_s
|
||||
end
|
||||
end
|
||||
|
||||
# pass through ssh-agent for github interaction
|
||||
config.ssh.forward_agent = true
|
||||
# pass through gpg-agent for artifact signing
|
||||
config.ssh.extra_args = [
|
||||
"-R", "/run/user/1000/gnupg/S.gpg-agent:#{%x(gpgconf --list-dirs agent-extra-socket).strip}",
|
||||
"-R", "/run/user/1000/gnupg/S.gpg-agent.extra:#{%x(gpgconf --list-dir agent-extra-socket).strip}",
|
||||
]
|
||||
end
|
||||
|
|
@ -0,0 +1,25 @@
|
|||
#!/usr/bin/env bash
|
||||
#
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
# Bootstrap provisioner for a Ubuntu Bionic host.
|
||||
#
|
||||
|
||||
sudo apt-get update -qq
|
||||
# puppet lets us manage the host, librarian-puppet lets us download puppet libraries
|
||||
sudo apt-get install -y --no-install-recommends puppet librarian-puppet
|
||||
cd /tmp/vagrant-puppet/environments/production && sudo librarian-puppet install --verbose
|
||||
|
|
@ -0,0 +1,3 @@
|
|||
.tmp/
|
||||
modules/
|
||||
**/*.lock
|
||||
|
|
@ -0,0 +1,21 @@
|
|||
#
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
---
|
||||
LIBRARIAN_PUPPET_DESTRUCTIVE: 'false'
|
||||
LIBRARIAN_PUPPET_USE_V1_API: '1'
|
||||
LIBRARIAN_PUPPET_TMP: "/tmp/librarian_puppet/tmp"
|
||||
|
|
@ -0,0 +1,27 @@
|
|||
#
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
forge "https://forgeapi.puppetlabs.com"
|
||||
|
||||
mod 'puppet-packages',
|
||||
:git => "https://github.com/greenaar/puppet-packages.git",
|
||||
:ref => '8d6b8a85eea931e4cd045884d5786c1c1ff0df4c'
|
||||
mod 'puppetlabs-docker', '3.10.1'
|
||||
mod 'puppetlabs-stdlib', '5.2.0'
|
||||
mod 'puppetlabs-vcsrepo', '3.1.0'
|
||||
mod 'saz-ssh', '6.2.0'
|
||||
|
|
@ -0,0 +1,31 @@
|
|||
#
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
---
|
||||
classes:
|
||||
- docker
|
||||
- packages::manage
|
||||
- ssh
|
||||
|
||||
packages::latest:
|
||||
- curl
|
||||
- git
|
||||
- gnupg
|
||||
- gnupg-agent
|
||||
|
||||
ssh::server_options:
|
||||
StreamLocalBindUnlink: 'yes'
|
||||
|
|
@ -0,0 +1,20 @@
|
|||
#
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
modulepath = modules:site:$basemodulepath
|
||||
manifest = manifests
|
||||
|
|
@ -0,0 +1,23 @@
|
|||
#
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
---
|
||||
version: 5
|
||||
|
||||
hierarchy:
|
||||
- name: "Per-node data"
|
||||
path: "nodes/%{facts.hostname}.yaml"
|
||||
|
|
@ -0,0 +1,44 @@
|
|||
#
|
||||
# Licensed to the Apache Software Foundation (ASF) under one
|
||||
# or more contributor license agreements. See the NOTICE file
|
||||
# distributed with this work for additional information
|
||||
# regarding copyright ownership. The ASF licenses this file
|
||||
# to you under the Apache License, Version 2.0 (the
|
||||
# "License"); you may not use this file except in compliance
|
||||
# with the License. You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
#
|
||||
|
||||
lookup('classes', Array[String], 'unique').include
|
||||
|
||||
node rmvm {
|
||||
$user = 'vagrant'
|
||||
|
||||
# include the default `vagrant` user in the `docker` group
|
||||
user { $user:
|
||||
groups => ['docker'],
|
||||
require => Package['docker'],
|
||||
}
|
||||
|
||||
# allow managing git repos in puppet
|
||||
vcsrepo { "/home/${user}/repos/hbase":
|
||||
ensure => latest,
|
||||
branch => 'master',
|
||||
group => $user,
|
||||
owner => $user,
|
||||
keep_local_changes => true,
|
||||
provider => git,
|
||||
remote => 'origin',
|
||||
source => {
|
||||
'origin' => 'https://github.com/apache/hbase.git',
|
||||
},
|
||||
depth => 1,
|
||||
}
|
||||
}
|
||||
Loading…
Reference in New Issue