This adds a Vagrantfile and supporting automation that creates a virtual machine environment
suitable for running the create-release scripting.
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
Exclude hbase-shaded-testing-util*.jar from checkcompatibility; this
jar can not be unzipped on a case-insensitive filesystem. Added
some means of debug into the checkcompatibility to help when
cryptic failures.
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Profiler shows a lot of time spent in the UPDATE SQL statement. Remove the tight loop and let SQL
do a bulk-update instead.
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
M dev-support/create-release/README.txt
Remove redundant text. Add some extra help around figuring state of
gpg-agent.
M dev-support/create-release/do-release.sh
Undo my mistaken commit where I undid test of gpg signing if under docker
M dev-support/create-release/release-build.sh
Handle '-h'
M src/main/asciidoc/_chapters/developer.adoc
Point to the README.txt under dev-tools/create-release rather than
repeat the text in here. Be more insistent about using scripts.
Added logging of timestamp so we can tell where we are spending time.
Added context to the README copied from head of entrance script.
Signed-off-by: Andrew Purtell <apurtell@apache.org>
README
Add note on ssh-agent.
dev-support/create-release/do-release.sh
move gpg check to non-docker context. Also cleanup tmp files when done.
dev-support/create-release/hbase-rm/Dockerfile
dev-support/create-release/mac-sshd-gpg-agent/Dockerfile
Hack to update packages... the old ones no longer available.
dev-support/create-release/release-util.sh
Allow that there are no JIRA changes in a release. Good for testing.
Update the design document to the state as of not. Adds a paragraph
on the preamble. See discussion on end of HBASE-25284.
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Populating the audit database with release tag information from git is
time consuming. Until that's sorted out, give the user a flag for
limiting which branches they want to be reviewed.
Signed-off-by: Andrew Purtell <apurtell@apache.org >
Due to HBASE-23834, HBASE-19256, HADOOP-16152, HBase 2.2.x and 2.3.x
cannot run on Hadoop 3.3.0, or any Hadoop version that has upgraded to
Jetty 9.4.
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Now that Hadoop 3.3.0 is released, let's figure out where it goes in
our testing matrix. Start by adding it to precommit checks.
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Matt Foley <mattf@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
After a bit of research into [0] and [1], and a bit of
experimentation, it seems we can use a partial wild-card expression
for these version strings. Let's try this for now. If it works out, we
should expand this usage to all the version package numbers, pinning
them to their epic:upstream-version components.
[0]: http://manpages.ubuntu.com/manpages/xenial/en/man5/deb-version.5.html
[1]: http://manpages.ubuntu.com/manpages/xenial/man8/apt-get.8.html
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
* put together a docker container that can use host gpg-agent forwarded over ssh.
* use gpg-agent forwarding container on OS X and directly forward the agent on Linux
* clean up the release container on exit
* use docker mounts instead of the deprecated volume syntax
* use image names within our project namespace
* update README to walk through running on GCE with gpg-agent forwarding
closes#1620
Signed-off-by: Matt Foley <mattf@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
By default, `urllib.quote` will skipp over `/` characters, which are
valid for use in passwords.
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Matt Foley <mattf@apache.org>
There is not much value in having this code in the master branch.
This code has it's dedicated git repo at
https://github.com/apache/hbase-native-client.
It was already removed from branch-1 (HBASE-19419) and branch-2
(HBASE-19176)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
* adds a optional -r [repo] arg
* if the passed repo is on the local filesystem, creates a container mount
* when cloning a local repo configure git to share objects with the local repo instead of copying
* when cloning a local repo in a container configure the clone to have a remote that will work back on the host.
closes#1725
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
* non-zero exit for usage
* in non-docker mode prompt for yetus install location
* make sure we exit if YETUS_HOME does not actually point at a yetus install.
closes#1726
Signed-off-by: Matt Foley <mattf@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
* narrow 'dry-run' limits so see svn activity up to just before check-in.
* Fix several typos and, in case of dry run, enable build step to chain from tag step by keeping the tagged git repo.
* Improve Maven settings.xml file, and names of variables related to it. Remove unnecessary use of "-Dmaven.repo.local", put it in settings.xml instead. Stop putting password literals in settings.xml.
* stop deleting maven settings file, now that it doesn't contain password strings
* Merge 'tag' into release-build.sh as another stage. Also found and fixed a couple bugs in the current release-tag.sh.
* Delete redundant release-tag.sh script.
* Small changes to make dev-support/create-release tools less focused
on hbase project only, while retaining special behaviors for
hbase sub-projects which share the hbase dist and jira locations.
* Changed terminology of release publish steps, from <build|publish>
to <publish-dist|publish-release>. In fact, what was formerly called
"build" built the distribution tarballs AND published them to dist,
while "publish" built the maven artifacts AND published them to Nexus.
The new terminology clarifies what's happening, and removes the appearance
of order dependendency.
* Fix publish-snapshot so it does same checks as publish-release.
* Factor out common maven usages, and move them to build-util.sh.
* Change default polarity of DRY_RUN to default to true. Change -n ("no-publish") to -f ("force publish") to actually publish.
* Fix problems in do-release.sh so it runs correctly outside of docker,
including DRY_RUN being exported.
* Have do-release.sh set REPO (shared maven local repository) if doing all three stages.
* Cleaned up REPO directory creation.
* General cleanup of comments and usage.
* fix all 'shellcheck' errors
* use ${BASH_SOURCE[0]} instead of $0 to determine script directory path
* smarter way to read version from pom with mvn
* do maven-gpg-plugin config settings in maven settings file correctly as documented
* fix gpg signing failure on Mac due to gpg-agent timeout
* fix various bugs to enable publish-dist, publish-snapshot, and publish-release to work correctly as individual steps and/or without docker
* improve log reporting from publish-release step
* fix bug in argument to checkcompatibility.py: replace PACKAGE_VERSION with GIT_REF
* demote "PACKAGE_VERSION" to "package_version_name" and undocument it in favor of RELEASE_TAG. Still enable appropriate defaulting in case RELEASE_TAG is undefined.
* unify RELEASE_VERSION with VERSION, to remove ambiguity and allow it to be set when only running 'publish' step without 'tag'
* query confirm RELEASE_TAG
* emphasize that release-build.sh is called for a single action at a time, and should be called from do-release.sh.
* add '-s' option to do-release.sh
* suppress maven "Download from central:" messages
* Replace human wait for tag propagation
Signed-off-by: Cesar Delgado <cdelgado@apple.com>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
On 2.x branches, we need to explicitly activate profiles for H3. On
master, all H2 support is dropped which means no special profiles are
required for H3 (though, there is still a profile there to encapsulate
H3 logic).
We need to make sure that the yetus invocation can correctly pass down
any profile information into the personality, so we activate the exact
profiles we want.
Closes#1609
Co-authored-by: Istvan Toth <stoty@cloudera.com>
Signed-off-by: stack <stack@apache.org>
Down jdk8 forked jvm heap from 2800 to 2200 and the jdk11 heap from
3200 to 2200. Down the mvn size from 4G to 3.6G
Change how many puts done by TestMultiRespectsLimits because made
the test run the forked heap over 2.5G in size.
Signed-off-by: Sean Busbey <busbey@apache.org>
Pass --threads=2 to mvn when yetus runs so some parallelism
when dependencies allow.
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Current code gives me the following; notice default values generated
for `RELEASE_VERSION` and `api_diff_tag`.
```
GIT_BRANCH [branch-2.3]:
Current branch VERSION is 2.3.0-SNAPSHOT.
RELEASE_VERSION [2.3.-1]: 2.3.0
NEXT_VERSION [2.3.0-SNAPSHOT]: 2.3.1-SNAPSHOT
RC_COUNT [0]:
GIT_REF [2.3.0RC0]:
api_diff_tag, [rel/2.2.0)]:
```
With this patch I get
```
GIT_BRANCH [branch-2.3]:
Current branch VERSION is 2.3.0-SNAPSHOT.
RELEASE_VERSION [2.3.0]:
NEXT_VERSION [2.3.1-SNAPSHOT]:
RC_COUNT [0]:
GIT_REF [2.3.0RC0]:
api_diff_tag, [rel/2.2.0]:
```
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
* enhancements to git_jira_release_audit.py
* add aforementioned release branch report
* include default values in help doc output
* swap default db to a file on disk instead of in memory
* set logger to match file name
* add separate sql query log at DEBUG level
* more detailed usage info in README.md, including example audit query
* update entries in fallback_actions.csv
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
This implementation is almost surely incorrect. Personality
initialization parses the `--hadoop-profile` argument and sets
`HADOOP_PROFILE`. That value is then used to build an `extras` value
that is passed along to module initialization. I'm guessing that the
`extras` value need to be honored down in the shadedjars module. I'm
not clear on how to make that work (need to study the interfaces at
play here), so taking the more ham-handed approach of referring to
`HADOOP_PROFILE`. I'm not sure if this will even work, or if it will
only work because the `foo_yetus.sh` scripts happen to use a variable
of the same name.
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: stack <stack@apache.org>
Duo Zhang
Sean Busbey
pustota2009
Nick Dimiduk
Mohammad Arshad
Michael Stack
stack
Lokesh Khurana
z-york
Viraj Jasani
Guanghao Zhang
Qi Yu
Istvan Toth
Abhey
Peter Somogyi
pengmq1
bsglz
Bharath Vissapragada
Matt Foley
Josh Elser
Minwoo Kang