Commit Graph

523 Commits

Author SHA1 Message Date
Josh Elser e54eec71c8 HBASE-24280 Skip explicit hadoop3 profile activation on master
On 2.x branches, we need to explicitly activate profiles for H3. On
master, all H2 support is dropped which means no special profiles are
required for H3 (though, there is still a profile there to encapsulate
H3 logic).

We need to make sure that the yetus invocation can correctly pass down
any profile information into the personality, so we activate the exact
profiles we want.

Closes #1609

Co-authored-by: Istvan Toth <stoty@cloudera.com>
Signed-off-by: stack <stack@apache.org>
2020-05-01 18:19:11 -04:00
Peter Somogyi 910b0caaf4
HBASE-24254 Update openjdk-8-jdk version in create-release (#1580)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-24 18:02:12 +02:00
Peter Somogyi 4578533d3a
HBASE-24251 PreCommit build is failing because of can not build docker image; addendum (#1579)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-24 13:40:20 +02:00
Duo Zhang ec8d582d80
HBASE-24253 HBASE-Find-Flaky-Tests job is failing because of can not build docker image (#1578)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-24 19:39:19 +08:00
Duo Zhang 023c6fe56b
HBASE-24251 PreCommit build is failing because of can not build docker image (#1575)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-24 15:39:18 +08:00
Duo Zhang af17b3ccd6
HBASE-24218 Add hadoop 3.2.x in hadoop check (#1559)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-22 10:51:46 +08:00
Duo Zhang 2846ea4700 HBASE-24198 Addendum add echo_and_redirect back when running hadoopcheck 2020-04-18 10:54:17 +08:00
Duo Zhang b02064f936
HBASE-24198 Skip hadoop2 check for master branch (#1536) 2020-04-18 09:20:16 +08:00
Peter Somogyi e130ffb7a0
HBASE-24203 Bump git version in Dockerfile; addendum (#1543)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-04-17 23:29:52 +02:00
Peter Somogyi 6d4e15a7ac
HBASE-24203 Bump git version in Dockerfile (#1538)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-17 13:06:46 +02:00
Duo Zhang 1f66806c96
HBASE-24170 Remove hadoop-2.0 profile (#1495)
Signed-off-by: stack <stack@apache.org>
2020-04-16 18:57:40 +08:00
Michael Stack 5f08311b23
HBASE-24134 Down forked JVM heap size from 2800m to 2200m for jdk8 and jdk11 (#1451) (#1503)
Down jdk8 forked jvm heap from 2800 to 2200 and the jdk11 heap from
3200 to 2200. Down the mvn size from 4G to 3.6G

Change how many puts done by TestMultiRespectsLimits because made
the test run the forked heap over 2.5G in size.

Signed-off-by: Sean Busbey <busbey@apache.org>
2020-04-13 21:35:43 -07:00
Michael Stack c2f2553490
HBASE-24126 Up the container nproc uplimit from 10000 to 12500 (#1504) 2020-04-13 14:59:16 -07:00
Michael Stack b843edb141 HBASE-24150 Allow module tests run in parallel (#1464)
Pass --threads=2 to mvn when yetus runs so some parallelism
when dependencies allow.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-13 11:54:16 -07:00
stack 3cf2df2ca0 Revert "HBASE-24126 Up the container nproc uplimit from 10000 to 12500 (#1450)"
This reverts commit 14342b6527.
2020-04-12 18:29:43 -07:00
Michael Stack 14342b6527
HBASE-24126 Up the container nproc uplimit from 10000 to 12500 (#1450)
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-04-10 15:57:47 -07:00
Nick Dimiduk b3488fcdec HBASE-24160 create-release fails to process x.y.0 version info correctly
Current code gives me the following; notice default values generated
for `RELEASE_VERSION` and `api_diff_tag`.

```
GIT_BRANCH [branch-2.3]:
Current branch VERSION is 2.3.0-SNAPSHOT.
RELEASE_VERSION [2.3.-1]: 2.3.0
NEXT_VERSION [2.3.0-SNAPSHOT]: 2.3.1-SNAPSHOT
RC_COUNT [0]:
GIT_REF [2.3.0RC0]:
api_diff_tag, [rel/2.2.0)]:
```

With this patch I get

```
GIT_BRANCH [branch-2.3]:
Current branch VERSION is 2.3.0-SNAPSHOT.
RELEASE_VERSION [2.3.0]:
NEXT_VERSION [2.3.1-SNAPSHOT]:
RC_COUNT [0]:
GIT_REF [2.3.0RC0]:
api_diff_tag, [rel/2.2.0]:
```

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-10 12:02:24 -07:00
Nick Dimiduk b4af0a0943 HBASE-24136 Add release branch report to git-jira-release-audit tool
* enhancements to git_jira_release_audit.py
  * add aforementioned release branch report
  * include default values in help doc output
  * swap default db to a file on disk instead of in memory
  * set logger to match file name
  * add separate sql query log at DEBUG level
* more detailed usage info in README.md, including example audit query
* update entries in fallback_actions.csv

Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-09 09:28:36 -07:00
Minwoo Kang e71c44203e
HBASE-24130 rat plugin complains about having an unlicensed file (#1445)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-04-07 09:16:10 -07:00
Nick Dimiduk 15b94cc306 HBASE-24081 Provide documentation for running Yetus with HBase
Signed-off-by: stack <stack@apache.org>
2020-04-06 16:35:57 -07:00
stack 1a627f0501 HBASE-24122 Change machine ulimit-l to ulimit-a so dumps full ulimit rather than just 'max locked memory' 2020-04-06 10:14:21 -07:00
Michael Stack 24ad05486f HBASE-24113 Upgrade the maven we use from 3.5.4 to 3.6.3 in nightlies (#1430)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-05 12:54:43 -07:00
Nick Dimiduk ea69b8711f HBASE-24049 use hadoop-2.10.0 for "packaging and integration" check
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-03 16:57:09 -07:00
stack 4565f74af2 HBASE-24072 Nightlies reporting OutOfMemoryError: unable to create new native thread
Set the forkcount down from 0.5C to 0.25C.
2020-04-02 21:07:25 -07:00
Nick Dimiduk 040a0a7ac5 HBASE-24071 [JDK11] Remove `unit` filter from nightly and precommit jobs
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-01 12:28:10 -07:00
Nick Dimiduk e85dfdc511 HBASE-24092 Fix links to build reports generated by nightly job
Signed-off-by: stack <stack@apache.org>
2020-04-01 12:16:53 -07:00
Nick Dimiduk e5a8435b48 HBASE-24078 SpotBugs check automatically skip inapplicable modules
Without patch, the spotbugs results include warnings for inapplicable
modules:

+---------------------------------------------------------------------------
|      |                |            | Compile Tests
+---------------------------------------------------------------------------
|   0  |        mvndep  |   0m 11s   | Maven dependency ordering
|  +1  |    mvninstall  |   2m 47s   | the source passed
|   0  |      spotbugs  |   0m 11s   | branch/hbase-annotations no spotbugs
|      |                |            | output file (spotbugsXml.xml)
|   0  |      spotbugs  |   0m 14s   | branch/hbase-assembly no spotbugs output
|      |                |            | file (spotbugsXml.xml)
|   0  |      spotbugs  |   0m 07s   |
|      |                |            | branch/hbase-archetypes/hbase-archetype-bu
|      |                |            | ilder no spotbugs output file
|      |                |            | (spotbugsXml.xml)
+---------------------------------------------------------------------------

With patch,

+---------------------------------------------------------------------------
|      |                |            | Compile Tests
+---------------------------------------------------------------------------
|   0  |        mvndep  |   0m 13s   | Maven dependency ordering
|  +1  |    mvninstall  |   3m 09s   | the source passed
|  +1  |      spotbugs  |  17m 09s   | the source passed
+---------------------------------------------------------------------------

Signed-off-by: stack <stack@apache.org>
2020-03-31 09:22:46 -07:00
Nick Dimiduk d05fccfc24 HBASE-24045 Support setting `-Dhadoop.profile` in adhoc_run_tests.sh ADDENDUM
* Correctly handle the hadoop2 profile.
* Create the output directory if it doesn't exist
2020-03-26 08:43:50 -07:00
Nick Dimiduk 5e63509781 HBASE-24045 Support setting `-Dhadoop.profile` in adhoc_run_tests.sh
Signed-off-by: stack <stack@apache.org>
2020-03-25 13:46:11 -07:00
Nick Dimiduk aa53493ae8 HBASE-24002 shadedjars check does not propagate --hadoop-profile
This implementation is almost surely incorrect. Personality
initialization parses the `--hadoop-profile` argument and sets
`HADOOP_PROFILE`. That value is then used to build an `extras` value
that is passed  along to module initialization. I'm guessing that the
`extras` value need to be honored down in the shadedjars module. I'm
not clear on how to make that work (need to study the interfaces at
play here), so taking the more ham-handed approach of referring to
`HADOOP_PROFILE`. I'm not sure if this will even work, or if it will
only work because the `foo_yetus.sh` scripts happen to use a variable
of the same name.

Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: stack <stack@apache.org>
2020-03-24 09:31:40 -07:00
stack de967502ab HBASE-24017 Turn down flakey rerun rate on all but hot branches
On master branch run twice a day instead of every 4 hours.
2020-03-18 15:50:39 -07:00
stack add2d2ac57 HBASE-24016 Change nightly poll from cron @daily to pollSCM @daily; i.e. run nightly if a change ONLY 2020-03-18 14:39:53 -07:00
Duo Zhang 9190a6b59a
HBASE-23989 Consider 2.10.0 in our support matrix (#1290)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-18 09:58:16 +08:00
Nick Dimiduk 2e1485f505 HBASE-24004 Include hadoop version in Nightly report name 2020-03-17 14:49:59 -07:00
Nick Dimiduk c088d2edc9 HBASE-23978 Dockerfiles reusing stale apt sources info
Signed-off-by: stack <stack@apache.org>
2020-03-12 17:09:06 -07:00
Duo Zhang c5aa2bc121
HBASE-23077 move entirely to spotbugs (#1265)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-12 10:18:09 +08:00
Nick Dimiduk 9c1f2dd0ca HBASE-23945 Dockerfiles showing hadolint check failures
Signed-off-by: stack <stack@apache.org>
2020-03-10 12:30:56 -07:00
Nick Dimiduk 31484f007f
HBASE-23946 [JDK11] Yetus should vote -0 for known JDK11 issues (#1246)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-09 09:10:33 -07:00
Nick Dimiduk 1f77b1ffa5 Revert "HBASE-23945 Dockerfiles showing hadolint check failures"
This reverts commit c61f95022d.
2020-03-07 10:41:54 -08:00
Nick Dimiduk c61f95022d HBASE-23945 Dockerfiles showing hadolint check failures
Signed-off-by: stack <stack@apache.org>
2020-03-06 09:52:15 -08:00
Nick Dimiduk 4c9d14a788 HBASE-23876 Add JDK11 compilation and unit test support to nightly job
Builds on the Dockerfile changes provided by HBASE-23767.

closes #1195

Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-05 11:46:15 -08:00
Nick Dimiduk 055c07c7b7 HBASE-23767 Add JDK11 compilation and unit test support to Github precommit
Rebuild our Dockerfile with support for multiple JDK versions. Use
multiple stages in the Jenkinsfile instead of yetus's multijdk because
of YETUS-953. Run those multiple stages in parallel to speed up
results.

Note that multiple stages means multiple Yetus invocations means
multiple comments on the PreCommit. This should become more obvious to
users once we can make use of GitHub Checks API, HBASE-23902.

closes #1183

Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-05 11:46:15 -08:00
Sean Busbey aff0ff5d97 HBASE-23198 Update ref guide for distributed MOB compaction.
* add design doc for original MOB changes as they were when HBase 2.0 came out
* add design doc for distributed MOB compaction
* remove configuration and commands no longer relevant after distributed MOB compaction
* add in discussion of configuration options
* allow asciimath formulas since we use them in the discussion

closes #1232

Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
2020-03-05 11:40:42 -06:00
stack 9bd39786df HBASE-23926 [Flakey Tests] Down the flakies re-run ferocity; it makes for too many fails.
Change the forkcount from 1.0C to 0.25C
2020-03-04 16:21:27 -08:00
Duo Zhang 29ea96fad8 HBASE-23921 Findbugs is OOM on master 2020-03-04 12:23:36 +08:00
Duo Zhang 346d087f40 HBASE-23911 Attach the new rsgroup implementation design doc to our code base (#1224)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Nick Dimiduk 3e21dc73de HBASE-23874 Move Jira-attached file precommit definition from script in Jenkins config to dev-support (#1191) (ADDENDUM)
Addendum -- set new script executable
2020-02-24 10:07:23 -08:00
Nick Dimiduk a5916c8e8e
HBASE-23874 Move Jira-attached file precommit definition from script in Jenkins config to dev-support (#1191)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-02-24 09:18:31 -08:00
Bharath Vissapragada fe0d49bc52 HBASE-23331: Documentation for HBASE-18095 2020-02-19 20:10:17 -08:00
stack 86e76c2bd9 HBASE-23865 Up flakey history from 5 to 10 2020-02-18 16:32:36 -08:00