HBASE-25437 Update refguide RC section; point at the dev-tools/create-releases/README.txt rather than repeat its content (#2804)
M dev-support/create-release/README.txt Remove redundant text. Add some extra help around figuring state of gpg-agent. M dev-support/create-release/do-release.sh Undo my mistaken commit where I undid test of gpg signing if under docker M dev-support/create-release/release-build.sh Handle '-h' M src/main/asciidoc/_chapters/developer.adoc Point to the README.txt under dev-tools/create-release rather than repeat the text in here. Be more insistent about using scripts.
This commit is contained in:
Michael Stack
GitHub
parent
1ca49ea50c
commit
2e21da7597
|
|
@ -1,18 +1,10 @@
|
|||
Creates a HBase release candidate. The script will update versions, tag the branch,
|
||||
build HBase binary packages and documentation, and upload maven artifacts to a staging
|
||||
repository. There is also a dry run mode where only local builds are performed, and
|
||||
nothing is uploaded to the ASF repos.
|
||||
|
||||
Run with "-h" for options. For example, running below will do all
|
||||
steps above using the 'rm' dir under Downloads as workspace:
|
||||
|
||||
$ ./do-release-docker.sh -d ~/Downloads/rm
|
||||
Creates an HBase release candidate.
|
||||
|
||||
The scripts in this directory came originally from spark
|
||||
(https://github.com/apache/spark/tree/master/dev/create-release). They were then
|
||||
modified to suite the hbase context. These scripts supercedes the old
|
||||
../make_rc.sh script for making release candidates because what is here is more
|
||||
comprehensive doing more steps of the RM process as well as running in a
|
||||
(https://github.com/apache/spark/tree/master/dev/create-release). They were
|
||||
then modified to suit the hbase context. These scripts supercede the old
|
||||
_../make_rc.sh_ script for making release candidates because what is here is
|
||||
more comprehensive doing more steps of the RM process as well as running in a
|
||||
container so the RM build environment can be a constant.
|
||||
|
||||
It:
|
||||
|
|
@ -24,49 +16,52 @@ It:
|
|||
* Pushes release tgzs to the dev dir in a apache dist.
|
||||
* Pushes to repository.apache.org staging.
|
||||
|
||||
The entry point is here, in the do-release-docker.sh script. Requires a local
|
||||
docker; for example, on mac os x, Docker for Desktop installed and running.
|
||||
The entry point is the do-release-docker.sh script. It requires a local
|
||||
docker; for example, on mac os x, a Docker for Desktop installed and running.
|
||||
|
||||
$ ./do-release-docker.sh -h
|
||||
(To run a build w/o invoking docker (not recommended!), use _do_release.sh_.)
|
||||
|
||||
To run a build w/o invoking docker (not recommended!), use _do_release.sh_.
|
||||
|
||||
Both scripts will query interactively for needed parameters and passphrases.
|
||||
The scripts will query interactively for needed parameters and passphrases.
|
||||
For explanation of the parameters, execute:
|
||||
|
||||
$ release-build.sh --help
|
||||
|
||||
The scripts run in dry-run mode by default where only local builds are
|
||||
performed and nothing is uploaded to the ASF repos. Pass the '-f' flag
|
||||
to remove dry-run mode.
|
||||
|
||||
Before starting the RC build, run a reconciliation of what is in JIRA with
|
||||
what is in the commit log. Make sure they align and that anomalies are
|
||||
explained up in JIRA.
|
||||
|
||||
See http://hbase.apache.org/book.html#maven.release
|
||||
explained up in JIRA. See http://hbase.apache.org/book.html#maven.release
|
||||
for how.
|
||||
|
||||
Regardless of where your release build will run (locally, locally in docker,
|
||||
on a remote machine, etc) you will need a local gpg-agent with access to your
|
||||
secret keys. A quick way to tell gpg to clear out state and start a gpg-agent
|
||||
is via the following command phrase:
|
||||
|
||||
$ gpgconf --kill all && gpg-connect-agent /bye
|
||||
|
||||
Before starting an RC build, make sure your local gpg-agent has configs
|
||||
to properly handle your credentials, especially if you want to avoid
|
||||
typing the passphrase to your secret key.
|
||||
|
||||
e.g. if you are going to run and step away, best to increase the TTL
|
||||
on caching the unlocked secret via ~/.gnupg/gpg-agent.conf
|
||||
secret keys. Before starting an RC build, make sure your local gpg-agent has
|
||||
configs to properly handle your credentials, especially if you want to avoid
|
||||
typing the passphrase to your secret key: e.g. if you are going to run
|
||||
and step away (the RC creation takes ~5 hours), best to increase the TTL on
|
||||
caching the unlocked secret by setting the following into local your
|
||||
~/.gnupg/gpg-agent.conf file:
|
||||
# in seconds, e.g. a day
|
||||
default-cache-ttl 86400
|
||||
max-cache-ttl 86400
|
||||
|
||||
A quick way to tell gpg to clear out state, re-read the gpg-agent.conf file
|
||||
and start a new gpg-agent is via the following command phrase:
|
||||
|
||||
$ gpgconf --kill all && gpg-connect-agent /bye
|
||||
|
||||
You can verify options took hold with '$ gpg --list-options gpg-agent'.
|
||||
|
||||
Similarly, run ssh-agent with your ssh key added if building with docker.
|
||||
|
||||
Running a build on GCE is easy enough. Here are some notes if of use.
|
||||
Create an instance. 4CPU/15G/10G disk seems to work well enough.
|
||||
Create an instance. 4CPU/15G/20G disk seems to work well enough.
|
||||
Once up, run the below to make your machine fit for RC building:
|
||||
|
||||
# Presuming debian-compatible OS, do these steps on the VM
|
||||
# your VM username should be your ASF id, because it will show up in build artifacts.
|
||||
# Presuming debian-compatible OS, do these steps on the VM.
|
||||
# Your VM username should be your ASF id, because it will show up in build artifacts.
|
||||
# Follow the docker install guide: https://docs.docker.com/engine/install/debian/
|
||||
$ sudo apt-get install -y \
|
||||
apt-transport-https \
|
||||
|
|
@ -129,7 +124,3 @@ $ git clone https://github.com/apache/hbase.git
|
|||
$ mkdir ~/build
|
||||
$ cd hbase
|
||||
$ ./dev-support/create-release/do-release-docker.sh -d ~/build
|
||||
|
||||
# for building the main repo specifically you can save an extra download by pointing the build
|
||||
# to the local clone you just made
|
||||
$ ./dev-support/create-release/do-release-docker.sh -d ~/build -r .git
|
||||
|
|
|
|||
|
|
@ -88,20 +88,19 @@ if [ "$RUNNING_IN_DOCKER" = "1" ]; then
|
|||
else
|
||||
# Outside docker, need to ask for information about the release.
|
||||
get_release_info
|
||||
|
||||
# Run this stuff when not in docker to check gpg.
|
||||
gpg_test_file="${TMPDIR}/gpg_test.$$.txt"
|
||||
echo "Testing gpg signing ${GPG} ${GPG_ARGS[@]} --detach --armor --sign ${gpg_test_file}"
|
||||
echo "foo" > "${gpg_test_file}"
|
||||
if ! "${GPG}" "${GPG_ARGS[@]}" --detach --armor --sign "${gpg_test_file}" ; then
|
||||
gpg_agent_help
|
||||
fi
|
||||
# In --batch mode we have to be explicit about what we are verifying
|
||||
if ! "${GPG}" "${GPG_ARGS[@]}" --verify "${gpg_test_file}.asc" "${gpg_test_file}" ; then
|
||||
gpg_agent_help
|
||||
fi
|
||||
fi
|
||||
|
||||
# Check GPG
|
||||
gpg_test_file="${TMPDIR}/gpg_test.$$.txt"
|
||||
echo "Testing gpg signing ${GPG} ${GPG_ARGS[@]} --detach --armor --sign ${gpg_test_file}"
|
||||
echo "foo" > "${gpg_test_file}"
|
||||
if ! "${GPG}" "${GPG_ARGS[@]}" --detach --armor --sign "${gpg_test_file}" ; then
|
||||
gpg_agent_help
|
||||
fi
|
||||
# In --batch mode we have to be explicit about what we are verifying
|
||||
if ! "${GPG}" "${GPG_ARGS[@]}" --verify "${gpg_test_file}.asc" "${gpg_test_file}" ; then
|
||||
gpg_agent_help
|
||||
fi
|
||||
GPG_TTY="$(tty)"
|
||||
export GPG_TTY
|
||||
|
||||
|
|
|
|||
|
|
@ -91,6 +91,10 @@ if [ $# -ne 1 ]; then
|
|||
exit_with_usage
|
||||
fi
|
||||
|
||||
if [[ "$1" == "-h" ]]; then
|
||||
exit_with_usage
|
||||
fi
|
||||
|
||||
if [[ "$*" == *"help"* ]]; then
|
||||
exit_with_usage
|
||||
fi
|
||||
|
|
|
|||
|
|
@ -625,15 +625,95 @@ upgrade protobuf later, then we don't have to add the profile '-Paarch64' anymor
|
|||
.Building against HBase 1.x
|
||||
[NOTE]
|
||||
====
|
||||
HBase 1.x requires Java 7 to build.
|
||||
See <<java,java>> for Java requirements per HBase release.
|
||||
See old refguides for how to build HBase 1.x.
|
||||
The below is for building hbase2.
|
||||
====
|
||||
|
||||
[[maven.release]]
|
||||
=== Making a Release Candidate
|
||||
Only committers can make releases of hbase artifacts.
|
||||
|
||||
.Before You Begin
|
||||
Check to be sure recent builds have been passing for the branch from where you
|
||||
are going to take your release. You should also have tried recent branch tips
|
||||
out on a cluster under load, perhaps by running the `hbase-it` integration test
|
||||
suite for a few hours to 'burn in' the near-candidate bits.
|
||||
|
||||
You will need a published signing key added to the hbase
|
||||
link:https://dist.apache.org/repos/dist/release/hbase/KEYS[KEYS] file.
|
||||
(For how to add a KEY, see _Step 1._ in link:https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease[How To Release],
|
||||
the Hadoop version of this document).
|
||||
|
||||
Next make sure JIRA is properly primed, that all issues targeted against
|
||||
the prospective release have been resolved and are present in git on the
|
||||
particular branch. If any outstanding issues, move them out of the release by
|
||||
adjusting the fix version to remove this pending release as a target.
|
||||
Any JIRA with a fix version that matches the release candidate
|
||||
target release will be included in the generated _CHANGES.md/RELEASENOTES.md_
|
||||
files that ship with the release so make sure JIRA is correct before you begin.
|
||||
|
||||
After doing the above, you can move to the manufacture of an RC.
|
||||
|
||||
Building an RC is involved so we've scripted it. The script builds in a Docker
|
||||
container to ensure we have a consistent environment building. It will ask you
|
||||
for passwords for apache and for your gpg signing key so it can sign and commit
|
||||
on your behalf. The passwords are passed to gpg-agent in the container and
|
||||
purged along with the container when the build is done.
|
||||
|
||||
The script will:
|
||||
|
||||
* Set version to the release version
|
||||
* Updates RELEASENOTES.md and CHANGES.md
|
||||
* Tag the RC
|
||||
* Set version to next SNAPSHOT version.
|
||||
* Builds, signs, and hashes all artifacts.
|
||||
* Generates the api compatibility report
|
||||
* Pushes release tgzs to the dev dir in a apache dist.
|
||||
* Pushes to repository.apache.org staging.
|
||||
* Creates vote email template.
|
||||
|
||||
The _dev-support/create-release/do-release-docker.sh_ Release Candidate (RC)
|
||||
Generating script is maintained in the master branch but can generate RCs
|
||||
for any 2.x+ branch (The script does not work against branch-1). Check out
|
||||
and update the master branch when making RCs. See
|
||||
_dev-support/create-release/README.txt_ for how to configure your
|
||||
environment and run the script.
|
||||
|
||||
[NOTE]
|
||||
====
|
||||
_dev-support/create-release/do-release-docker.sh_ supercedes the previous
|
||||
_dev-support/make_rc.sh_ script. It is more comprehensive automating all
|
||||
steps, rather than a portion, building a RC.
|
||||
====
|
||||
|
||||
==== Release Candidate Procedure
|
||||
Here we outline the steps involved generating a Release Candidate, the steps
|
||||
automated by the _dev-support/create-release/do-release-docker.sh_ script
|
||||
described in the previous section. Running these steps manually tends to
|
||||
be error-prone so is not recommended. The below is informational only.
|
||||
|
||||
The process below makes use of various tools, mainly _git_ and _maven_.
|
||||
|
||||
.Specifying the Heap Space for Maven
|
||||
[NOTE]
|
||||
====
|
||||
You may run into OutOfMemoryErrors building, particularly building the site and
|
||||
documentation. Up the heap for Maven by setting the `MAVEN_OPTS` variable.
|
||||
You can prefix the variable to the Maven command, as in the following example:
|
||||
|
||||
----
|
||||
MAVEN_OPTS="-Xmx4g -XX:MaxPermSize=256m" mvn package
|
||||
----
|
||||
|
||||
You could also set this in an environment variable or alias in your shell.
|
||||
====
|
||||
|
||||
[[maven.settings.xml]]
|
||||
.Example _~/.m2/settings.xml_ File
|
||||
====
|
||||
Publishing to maven requires you sign the artifacts you want to upload.
|
||||
For the build to sign them for you, you a properly configured _settings.xml_ in your local repository under _.m2_, such as the following.
|
||||
For the build to sign them for you, you a properly configured _settings.xml_
|
||||
in your local repository under _.m2_, such as the following.
|
||||
|
||||
[source,xml]
|
||||
----
|
||||
|
|
@ -675,190 +755,6 @@ For the build to sign them for you, you a properly configured _settings.xml_ in
|
|||
----
|
||||
====
|
||||
|
||||
[[maven.release]]
|
||||
=== Making a Release Candidate
|
||||
Only committers can make releases of hbase artifacts.
|
||||
|
||||
.Before You Begin
|
||||
Check to be sure recent builds have been passing for the branch from where you
|
||||
are going to take your release. You should also have tried recent branch tips
|
||||
out on a cluster under load, perhaps by running the `hbase-it` integration test
|
||||
suite for a few hours to 'burn in' the near-candidate bits.
|
||||
|
||||
You will need a published signing key added to the hbase
|
||||
link:https://dist.apache.org/repos/dist/release/hbase/KEYS[KEYS] file.
|
||||
(For how to add a KEY, see _Step 1._ in link:https://cwiki.apache.org/confluence/display/HADOOP2/HowToRelease[How To Release],
|
||||
the Hadoop version of this document).
|
||||
|
||||
Next make sure JIRA is properly primed, that all issues targeted against
|
||||
the prospective release have been resolved and are present in git on the
|
||||
particular branch. If any outstanding issues, move them out of the release by
|
||||
adjusting the fix version to remove this pending release as a target.
|
||||
Any JIRA with a fix version that matches the release candidate
|
||||
target release will be included in the generated _CHANGES.md/RELEASENOTES.md_
|
||||
files that ship with the release so make sure JIRA is correct before you begin.
|
||||
|
||||
After doing the above, you can move to the manufacture of an RC.
|
||||
Building an RC is involved. We've tried to script it. In the next section
|
||||
we describe the script. It is followed by a description of the steps
|
||||
involved which the script automates.
|
||||
|
||||
[[do-release-docker.sh]]
|
||||
==== Release Candidate Generating Script
|
||||
|
||||
The _dev-support/create-release/do-release-docker.sh_ Release Candidate (RC)
|
||||
Generating script is maintained in the master branch but can generate RCs
|
||||
for any 2.x+ branch (The script does not work against branch-1). Check out
|
||||
and update the master branch when making RCs.
|
||||
|
||||
The script builds in a Docker container to ensure we have a consistent
|
||||
environment building. It will ask you for passwords for apache and for your
|
||||
gpg signing key so it can sign and commit on your behalf. The passwords
|
||||
are passed to gpg-agent in the container and purged along with the container
|
||||
when the build is done.
|
||||
|
||||
[NOTE]
|
||||
====
|
||||
_dev-support/create-release/do-release-docker.sh_ supercedes the previous
|
||||
_dev-support/make_rc.sh_ script. It is more comprehensive automating all
|
||||
steps, rather than a portion, building a RC.
|
||||
====
|
||||
|
||||
The script will:
|
||||
|
||||
* Set version to the release version
|
||||
* Updates RELEASENOTES.md and CHANGES.md
|
||||
* Tag the RC
|
||||
* Set version to next SNAPSHOT version.
|
||||
* Builds, signs, and hashes all artifacts.
|
||||
* Generates the api compatibility report
|
||||
* Pushes release tgzs to the dev dir in a apache dist.
|
||||
* Pushes to repository.apache.org staging.
|
||||
* Creates vote email template.
|
||||
|
||||
The RC building script is _dev-support/create-release/do-release-docker.sh_.
|
||||
Pass _-h_ to _dev-support/create-release/do-release-docker.sh_ to
|
||||
see available options:
|
||||
|
||||
```
|
||||
Usage: do-release-docker.sh [options]
|
||||
|
||||
This script runs the release scripts inside a docker image.
|
||||
|
||||
Options:
|
||||
|
||||
-d [path] required. working directory. output will be written to "output" in here.
|
||||
-n dry run mode. Checks and local builds, but does not upload anything.
|
||||
-t [tag] tag for the hbase-rm docker image to use for building (default: "latest").
|
||||
-j [path] path to local JDK installation to use building. By default the script will
|
||||
use openjdk8 installed in the docker image.
|
||||
-s [step] runs a single step of the process; valid steps are: tag, build, publish. if
|
||||
none specified, runs tag, then build, and then publish.
|
||||
```
|
||||
|
||||
Running the below command will do all steps above using the
|
||||
'rm' working directory under Downloads as workspace:
|
||||
```
|
||||
$ ./dev-support/create-release/do-release-docker.sh -d ~/Downloads/rm
|
||||
```
|
||||
|
||||
The script will ask you a set of questions about the release version
|
||||
and branch, the version to generate the compatibility report against,
|
||||
and so on, before it starts executing (If you set the appropriate
|
||||
environment variables, the script will skip asking you questions --
|
||||
which can come in handy if you end up having to re-run the script
|
||||
multiple times).
|
||||
|
||||
On branch 2.1, a Release Candidate (RC) creation can take many hours
|
||||
(~8 hours) so run your build on a machine you know will be
|
||||
around for this swath of time. Start the build inside a _screen_
|
||||
or _tmux_ session in case you become disconnected from your
|
||||
build box.
|
||||
|
||||
The build is made of three stages: tag, build, and
|
||||
publish. If the script fails, you may resort to 'fixing' the
|
||||
failure manually and then asking the script to run the
|
||||
subsequent stage rather than start over.
|
||||
|
||||
When the scripts run, they use the passed working directory.
|
||||
Under the working directory is an _output_ dir. In here is
|
||||
where the checkouts go, where we build up the _svn_ directory
|
||||
to _svn_ commit to _apache/dist/dev_, etc. Each step also
|
||||
dumps a log file in here: e.g. _tag.log_ for the tagging
|
||||
step and _build.log_ for building.
|
||||
|
||||
The _tagging_ step will checkout hbase, set the version number
|
||||
in all the poms – e.g. if branch-2.0 is at 2.0.6-SNAPSHOT
|
||||
and you are making a 2.0.5 RC, it will set the versions in
|
||||
all poms to 2.0.5 – appropriately. It then generate CHANGES.md
|
||||
and RELEASENOTES.md by checking out yetus and then
|
||||
calling its generator scripts. It then commits the poms with
|
||||
their new versions along with the changed CHANGES.md and
|
||||
RELEASENOTES.md, tags, and pushes up all changes to the
|
||||
apache repo.
|
||||
|
||||
The _build_ step will checkout hbase, build all including
|
||||
javadoc and doc (javadoc takes the bulk of the time – 4 hours plus),
|
||||
run assemblies to produce src and bin tarballs, sign and hash it
|
||||
all, and then make a dir under apache dist dev named for the RC.
|
||||
It will copy all artifacts in here including top-level CHANGES.md
|
||||
and RELEASENOTES.md. It will generate api diff docs and put them
|
||||
into this RC dir too. When done, it commits the svn RC.
|
||||
|
||||
The publish step will checkout hbase, build, and then copy up all
|
||||
artifacts to repository.apache.org (signed and hashed). When done,
|
||||
it will dump out an email template with all the correct links in place.
|
||||
|
||||
Check the artifacts pushed to the dev distribution directory and up
|
||||
in repository.apache.org. If all looks good, check the generated
|
||||
email and send to the dev list.
|
||||
|
||||
Under the create-release dir, scripts should make some sense:
|
||||
```
|
||||
do-release-docker.sh # Main entrance.
|
||||
do-release.sh . # More checks. Not usable really other than by setting env variables before running it.
|
||||
release-tag.sh # Does tagging steps.
|
||||
release-build.sh . # Does the build and publish step.
|
||||
release-util.sh # Utility used by all of the above.
|
||||
vote.tmpl # Template for email to send out.
|
||||
hbase-rm # Has docker image we use.
|
||||
```
|
||||
|
||||
If the RC fails, the script will do the right thing when it comes
|
||||
to edit of the _CHANGES.md_ and _RELEASENOTES.md_ removing the old
|
||||
and updating the files with the updated content (No harm verifying
|
||||
though).
|
||||
|
||||
One trick for checking stuff especially in utility is to do as follows:
|
||||
|
||||
```
|
||||
$ source release-util.sh ; generate_api_report ../../ rel/2.1.3 2.14RC1
|
||||
```
|
||||
|
||||
i.e. source the release-util.sh script and then run one of its functions
|
||||
passing args. Helped debugging stuff.
|
||||
|
||||
[[rc_procedure]]
|
||||
==== Release Candidate Procedure
|
||||
Here we describe the steps involved generating a Release Candidate, the steps
|
||||
automated by the script described in the previous section.
|
||||
|
||||
The process below makes use of various tools, mainly _git_ and _maven_.
|
||||
|
||||
.Specifying the Heap Space for Maven
|
||||
[NOTE]
|
||||
====
|
||||
You may run into OutOfMemoryErrors building, particularly building the site and
|
||||
documentation. Up the heap for Maven by setting the `MAVEN_OPTS` variable.
|
||||
You can prefix the variable to the Maven command, as in the following example:
|
||||
|
||||
----
|
||||
MAVEN_OPTS="-Xmx4g -XX:MaxPermSize=256m" mvn package
|
||||
----
|
||||
|
||||
You could also set this in an environment variable or alias in your shell.
|
||||
====
|
||||
|
||||
===== Update the _CHANGES.md_ and _RELEASENOTES.md_ files and the POM files.
|
||||
|
||||
Update _CHANGES.md_ with the changes since the last release. Be careful with where you put
|
||||
|
|
|
|||
Loading…
Reference in New Issue