Commit Graph

528 Commits

Author SHA1 Message Date
Bharath Vissapragada 2382f90679
HBASE-24422: Remove hbase-native-client code from the master branch (#1766)
There is not much value in having this code in the master branch.
This code has it's dedicated git repo at
https://github.com/apache/hbase-native-client.

It was already removed from branch-1 (HBASE-19419) and branch-2
(HBASE-19176)

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-05-24 16:19:58 -07:00
Sean Busbey a9fefd7f53
HBASE-24297 release scripts should be able to use a custom git repo
* adds a optional -r [repo] arg
* if the passed repo is on the local filesystem, creates a container mount
* when cloning a local repo configure git to share objects with the local repo instead of copying
* when cloning a local repo in a container configure the clone to have a remote that will work back on the host.

closes #1725

Signed-off-by: stack <stack@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-05-22 22:24:12 -05:00
Sean Busbey 8ff8e70edf
HBASE-24296 install yetus as a part of building the rm docker image.
* non-zero exit for usage
* in non-docker mode prompt for yetus install location
* make sure we exit if YETUS_HOME does not actually point at a yetus install.

closes #1726

Signed-off-by: Matt Foley <mattf@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-05-22 22:24:03 -05:00
Sean Busbey 696894e2cc
HBASE-24393 release scripts should allow for creation of initial CHANGES/RELEASE on new major version.
closes #1728

Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-05-18 12:11:13 -05:00
Matt Foley f3c0a0e06d
HBASE-24318 Create-release scripts fixes and enhancements (#1643)
* narrow 'dry-run' limits so see svn activity up to just before check-in.
* Fix several typos and, in case of dry run, enable build step to chain from tag step by keeping the tagged git repo.
* Improve Maven settings.xml file, and names of variables related to it. Remove unnecessary use of "-Dmaven.repo.local", put it in settings.xml instead. Stop putting password literals in settings.xml. 
* stop deleting maven settings file, now that it doesn't contain password strings
* Merge 'tag' into release-build.sh as another stage. Also found and fixed a couple bugs in the current release-tag.sh.
* Delete redundant release-tag.sh script.
* Small changes to make dev-support/create-release tools less focused
on hbase project only, while retaining special behaviors for
hbase sub-projects which share the hbase dist and jira locations.
* Changed terminology of release publish steps, from <build|publish>
to <publish-dist|publish-release>. In fact, what was formerly called
"build" built the distribution tarballs AND published them to dist,
while "publish" built the  maven artifacts AND published them to Nexus.
The new terminology clarifies what's happening, and removes the appearance
of order dependendency.
* Fix publish-snapshot so it does same checks as publish-release.
* Factor out common maven usages, and move them to build-util.sh.
* Change default polarity of DRY_RUN to default to true. Change -n ("no-publish") to -f ("force publish") to actually publish.
* Fix problems in do-release.sh so it runs correctly outside of docker,
  including DRY_RUN being exported.
* Have do-release.sh set REPO (shared maven local repository) if doing all three stages.
* Cleaned up REPO directory creation.
* General cleanup of comments and usage.
*  fix all 'shellcheck' errors
* use ${BASH_SOURCE[0]} instead of $0 to determine script directory path
* smarter way to read version from pom with mvn
* do maven-gpg-plugin config settings in maven settings file correctly as documented
* fix gpg signing failure on Mac due to gpg-agent timeout
* fix various bugs to enable publish-dist, publish-snapshot, and publish-release to work correctly as individual steps and/or without docker
* improve log reporting from publish-release step
*  fix bug in argument to checkcompatibility.py: replace PACKAGE_VERSION with GIT_REF
* demote "PACKAGE_VERSION" to "package_version_name" and undocument it in favor of RELEASE_TAG. Still enable appropriate defaulting in case RELEASE_TAG is undefined.
* unify RELEASE_VERSION with VERSION, to remove ambiguity and allow it to be set when only running 'publish' step without 'tag'
* query confirm RELEASE_TAG
* emphasize that release-build.sh is called for a single action at a time, and should be called from do-release.sh.
* add '-s' option to do-release.sh
* suppress maven "Download from central:" messages
* Replace human wait for tag propagation

Signed-off-by: Cesar Delgado <cdelgado@apple.com>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-05-09 10:46:41 -05:00
Josh Elser e54eec71c8 HBASE-24280 Skip explicit hadoop3 profile activation on master
On 2.x branches, we need to explicitly activate profiles for H3. On
master, all H2 support is dropped which means no special profiles are
required for H3 (though, there is still a profile there to encapsulate
H3 logic).

We need to make sure that the yetus invocation can correctly pass down
any profile information into the personality, so we activate the exact
profiles we want.

Closes #1609

Co-authored-by: Istvan Toth <stoty@cloudera.com>
Signed-off-by: stack <stack@apache.org>
2020-05-01 18:19:11 -04:00
Peter Somogyi 910b0caaf4
HBASE-24254 Update openjdk-8-jdk version in create-release (#1580)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-24 18:02:12 +02:00
Peter Somogyi 4578533d3a
HBASE-24251 PreCommit build is failing because of can not build docker image; addendum (#1579)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-24 13:40:20 +02:00
Duo Zhang ec8d582d80
HBASE-24253 HBASE-Find-Flaky-Tests job is failing because of can not build docker image (#1578)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-24 19:39:19 +08:00
Duo Zhang 023c6fe56b
HBASE-24251 PreCommit build is failing because of can not build docker image (#1575)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-24 15:39:18 +08:00
Duo Zhang af17b3ccd6
HBASE-24218 Add hadoop 3.2.x in hadoop check (#1559)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-22 10:51:46 +08:00
Duo Zhang 2846ea4700 HBASE-24198 Addendum add echo_and_redirect back when running hadoopcheck 2020-04-18 10:54:17 +08:00
Duo Zhang b02064f936
HBASE-24198 Skip hadoop2 check for master branch (#1536) 2020-04-18 09:20:16 +08:00
Peter Somogyi e130ffb7a0
HBASE-24203 Bump git version in Dockerfile; addendum (#1543)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-04-17 23:29:52 +02:00
Peter Somogyi 6d4e15a7ac
HBASE-24203 Bump git version in Dockerfile (#1538)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-17 13:06:46 +02:00
Duo Zhang 1f66806c96
HBASE-24170 Remove hadoop-2.0 profile (#1495)
Signed-off-by: stack <stack@apache.org>
2020-04-16 18:57:40 +08:00
Michael Stack 5f08311b23
HBASE-24134 Down forked JVM heap size from 2800m to 2200m for jdk8 and jdk11 (#1451) (#1503)
Down jdk8 forked jvm heap from 2800 to 2200 and the jdk11 heap from
3200 to 2200. Down the mvn size from 4G to 3.6G

Change how many puts done by TestMultiRespectsLimits because made
the test run the forked heap over 2.5G in size.

Signed-off-by: Sean Busbey <busbey@apache.org>
2020-04-13 21:35:43 -07:00
Michael Stack c2f2553490
HBASE-24126 Up the container nproc uplimit from 10000 to 12500 (#1504) 2020-04-13 14:59:16 -07:00
Michael Stack b843edb141 HBASE-24150 Allow module tests run in parallel (#1464)
Pass --threads=2 to mvn when yetus runs so some parallelism
when dependencies allow.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-13 11:54:16 -07:00
stack 3cf2df2ca0 Revert "HBASE-24126 Up the container nproc uplimit from 10000 to 12500 (#1450)"
This reverts commit 14342b6527.
2020-04-12 18:29:43 -07:00
Michael Stack 14342b6527
HBASE-24126 Up the container nproc uplimit from 10000 to 12500 (#1450)
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-04-10 15:57:47 -07:00
Nick Dimiduk b3488fcdec HBASE-24160 create-release fails to process x.y.0 version info correctly
Current code gives me the following; notice default values generated
for `RELEASE_VERSION` and `api_diff_tag`.

```
GIT_BRANCH [branch-2.3]:
Current branch VERSION is 2.3.0-SNAPSHOT.
RELEASE_VERSION [2.3.-1]: 2.3.0
NEXT_VERSION [2.3.0-SNAPSHOT]: 2.3.1-SNAPSHOT
RC_COUNT [0]:
GIT_REF [2.3.0RC0]:
api_diff_tag, [rel/2.2.0)]:
```

With this patch I get

```
GIT_BRANCH [branch-2.3]:
Current branch VERSION is 2.3.0-SNAPSHOT.
RELEASE_VERSION [2.3.0]:
NEXT_VERSION [2.3.1-SNAPSHOT]:
RC_COUNT [0]:
GIT_REF [2.3.0RC0]:
api_diff_tag, [rel/2.2.0]:
```

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-10 12:02:24 -07:00
Nick Dimiduk b4af0a0943 HBASE-24136 Add release branch report to git-jira-release-audit tool
* enhancements to git_jira_release_audit.py
  * add aforementioned release branch report
  * include default values in help doc output
  * swap default db to a file on disk instead of in memory
  * set logger to match file name
  * add separate sql query log at DEBUG level
* more detailed usage info in README.md, including example audit query
* update entries in fallback_actions.csv

Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-09 09:28:36 -07:00
Minwoo Kang e71c44203e
HBASE-24130 rat plugin complains about having an unlicensed file (#1445)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-04-07 09:16:10 -07:00
Nick Dimiduk 15b94cc306 HBASE-24081 Provide documentation for running Yetus with HBase
Signed-off-by: stack <stack@apache.org>
2020-04-06 16:35:57 -07:00
stack 1a627f0501 HBASE-24122 Change machine ulimit-l to ulimit-a so dumps full ulimit rather than just 'max locked memory' 2020-04-06 10:14:21 -07:00
Michael Stack 24ad05486f HBASE-24113 Upgrade the maven we use from 3.5.4 to 3.6.3 in nightlies (#1430)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-05 12:54:43 -07:00
Nick Dimiduk ea69b8711f HBASE-24049 use hadoop-2.10.0 for "packaging and integration" check
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-03 16:57:09 -07:00
stack 4565f74af2 HBASE-24072 Nightlies reporting OutOfMemoryError: unable to create new native thread
Set the forkcount down from 0.5C to 0.25C.
2020-04-02 21:07:25 -07:00
Nick Dimiduk 040a0a7ac5 HBASE-24071 [JDK11] Remove `unit` filter from nightly and precommit jobs
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-01 12:28:10 -07:00
Nick Dimiduk e85dfdc511 HBASE-24092 Fix links to build reports generated by nightly job
Signed-off-by: stack <stack@apache.org>
2020-04-01 12:16:53 -07:00
Nick Dimiduk e5a8435b48 HBASE-24078 SpotBugs check automatically skip inapplicable modules
Without patch, the spotbugs results include warnings for inapplicable
modules:

+---------------------------------------------------------------------------
|      |                |            | Compile Tests
+---------------------------------------------------------------------------
|   0  |        mvndep  |   0m 11s   | Maven dependency ordering
|  +1  |    mvninstall  |   2m 47s   | the source passed
|   0  |      spotbugs  |   0m 11s   | branch/hbase-annotations no spotbugs
|      |                |            | output file (spotbugsXml.xml)
|   0  |      spotbugs  |   0m 14s   | branch/hbase-assembly no spotbugs output
|      |                |            | file (spotbugsXml.xml)
|   0  |      spotbugs  |   0m 07s   |
|      |                |            | branch/hbase-archetypes/hbase-archetype-bu
|      |                |            | ilder no spotbugs output file
|      |                |            | (spotbugsXml.xml)
+---------------------------------------------------------------------------

With patch,

+---------------------------------------------------------------------------
|      |                |            | Compile Tests
+---------------------------------------------------------------------------
|   0  |        mvndep  |   0m 13s   | Maven dependency ordering
|  +1  |    mvninstall  |   3m 09s   | the source passed
|  +1  |      spotbugs  |  17m 09s   | the source passed
+---------------------------------------------------------------------------

Signed-off-by: stack <stack@apache.org>
2020-03-31 09:22:46 -07:00
Nick Dimiduk d05fccfc24 HBASE-24045 Support setting `-Dhadoop.profile` in adhoc_run_tests.sh ADDENDUM
* Correctly handle the hadoop2 profile.
* Create the output directory if it doesn't exist
2020-03-26 08:43:50 -07:00
Nick Dimiduk 5e63509781 HBASE-24045 Support setting `-Dhadoop.profile` in adhoc_run_tests.sh
Signed-off-by: stack <stack@apache.org>
2020-03-25 13:46:11 -07:00
Nick Dimiduk aa53493ae8 HBASE-24002 shadedjars check does not propagate --hadoop-profile
This implementation is almost surely incorrect. Personality
initialization parses the `--hadoop-profile` argument and sets
`HADOOP_PROFILE`. That value is then used to build an `extras` value
that is passed  along to module initialization. I'm guessing that the
`extras` value need to be honored down in the shadedjars module. I'm
not clear on how to make that work (need to study the interfaces at
play here), so taking the more ham-handed approach of referring to
`HADOOP_PROFILE`. I'm not sure if this will even work, or if it will
only work because the `foo_yetus.sh` scripts happen to use a variable
of the same name.

Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: stack <stack@apache.org>
2020-03-24 09:31:40 -07:00
stack de967502ab HBASE-24017 Turn down flakey rerun rate on all but hot branches
On master branch run twice a day instead of every 4 hours.
2020-03-18 15:50:39 -07:00
stack add2d2ac57 HBASE-24016 Change nightly poll from cron @daily to pollSCM @daily; i.e. run nightly if a change ONLY 2020-03-18 14:39:53 -07:00
Duo Zhang 9190a6b59a
HBASE-23989 Consider 2.10.0 in our support matrix (#1290)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-18 09:58:16 +08:00
Nick Dimiduk 2e1485f505 HBASE-24004 Include hadoop version in Nightly report name 2020-03-17 14:49:59 -07:00
Nick Dimiduk c088d2edc9 HBASE-23978 Dockerfiles reusing stale apt sources info
Signed-off-by: stack <stack@apache.org>
2020-03-12 17:09:06 -07:00
Duo Zhang c5aa2bc121
HBASE-23077 move entirely to spotbugs (#1265)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-12 10:18:09 +08:00
Nick Dimiduk 9c1f2dd0ca HBASE-23945 Dockerfiles showing hadolint check failures
Signed-off-by: stack <stack@apache.org>
2020-03-10 12:30:56 -07:00
Nick Dimiduk 31484f007f
HBASE-23946 [JDK11] Yetus should vote -0 for known JDK11 issues (#1246)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-09 09:10:33 -07:00
Nick Dimiduk 1f77b1ffa5 Revert "HBASE-23945 Dockerfiles showing hadolint check failures"
This reverts commit c61f95022d.
2020-03-07 10:41:54 -08:00
Nick Dimiduk c61f95022d HBASE-23945 Dockerfiles showing hadolint check failures
Signed-off-by: stack <stack@apache.org>
2020-03-06 09:52:15 -08:00
Nick Dimiduk 4c9d14a788 HBASE-23876 Add JDK11 compilation and unit test support to nightly job
Builds on the Dockerfile changes provided by HBASE-23767.

closes #1195

Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-05 11:46:15 -08:00
Nick Dimiduk 055c07c7b7 HBASE-23767 Add JDK11 compilation and unit test support to Github precommit
Rebuild our Dockerfile with support for multiple JDK versions. Use
multiple stages in the Jenkinsfile instead of yetus's multijdk because
of YETUS-953. Run those multiple stages in parallel to speed up
results.

Note that multiple stages means multiple Yetus invocations means
multiple comments on the PreCommit. This should become more obvious to
users once we can make use of GitHub Checks API, HBASE-23902.

closes #1183

Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-05 11:46:15 -08:00
Sean Busbey aff0ff5d97 HBASE-23198 Update ref guide for distributed MOB compaction.
* add design doc for original MOB changes as they were when HBase 2.0 came out
* add design doc for distributed MOB compaction
* remove configuration and commands no longer relevant after distributed MOB compaction
* add in discussion of configuration options
* allow asciimath formulas since we use them in the discussion

closes #1232

Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
2020-03-05 11:40:42 -06:00
stack 9bd39786df HBASE-23926 [Flakey Tests] Down the flakies re-run ferocity; it makes for too many fails.
Change the forkcount from 1.0C to 0.25C
2020-03-04 16:21:27 -08:00
Duo Zhang 29ea96fad8 HBASE-23921 Findbugs is OOM on master 2020-03-04 12:23:36 +08:00