Commit Graph

106 Commits

Author SHA1 Message Date
Duo Zhang 9e03253447 HBASE-26995 Remove ref guide check in pre commit and nightly for branches other than master (#4399)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
(cherry picked from commit 9a41117ca5)
2022-05-05 18:39:34 +08:00
Duo Zhang 386869a428
HBASE-26892 Add spotless:check in our pre commit general check (#4393)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2022-05-03 21:14:24 +08:00
Duo Zhang f4eb9071fd
HBASE-26917 Do not add --threads when running 'mvn site' (#4354)
Signed-off-by: GeorryHuang <huangzhuoyue@apache.org>
2022-04-29 08:05:00 +08:00
Duo Zhang c791c8181a
HBASE-26929 Upgrade surefire plugin to 3.0.0-M6 (#4319)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2022-04-07 11:29:27 +08:00
Duo Zhang caea843e94 HBASE-26848 Addendum make tmpdir absolute 2022-03-17 13:02:14 +08:00
Duo Zhang fec5478226 HBASE-26828 Addendum fix syntax error 2022-03-16 21:48:58 +08:00
Duo Zhang cc13624232
HBASE-26848 Set java.io.tmpdir on mvn command when running jenkins job (#4231)
Signed-off-by: GeorryHuang <huangzhuoyue@apache.org>
2022-03-16 17:52:36 +08:00
Duo Zhang a49d147d49
HBASE-26828 Increase the concurrency when running UTs in pre commit job (#4213)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2022-03-14 12:01:51 +08:00
Nick Dimiduk 96c971f18b
HBASE-26549 hbaseprotoc plugin should initialize maven (#3927)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-12-14 23:01:14 +08:00
Duo Zhang 8bc61f30ba
HBASE-26428 Add hadoop 2.10.1, 3.2.2 and 3.3.1 to our hadoop checks (#3821)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2021-11-08 01:05:08 +08:00
Duo Zhang 69656e0df8
HBASE-26362 Upload mvn site artifacts for nightly build to nightlies (#3778)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-10-21 23:07:30 +08:00
Peter Somogyi 6a1382a503
HBASE-26176 Correct regex in hbase-personality.sh (#3567)
Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org>
2021-08-09 10:53:41 +02:00
Sean Busbey ca129e99ce
HBASE-25083 further HBase 1.y releases should have Hadoop 2.10 as a minimum version. (#2656) 2020-11-17 08:49:00 -06:00
Duo Zhang 200343c59c
HBASE-25203 Change the reference url to flaky list in our jenkins jobs (#2566)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-10-20 17:00:06 +08:00
Nick Dimiduk bc20203965
"HBASE-25144 Add Hadoop-3.3.0 to personality hadoopcheck" try again (#2536)
Due to HBASE-23834, HBASE-19256, HADOOP-16152, HBase 2.2.x and 2.3.x
cannot run on Hadoop 3.3.0, or any Hadoop version that has upgraded to
Jetty 9.4.

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-13 10:18:47 +08:00
Nick Dimiduk a8c49a6463
HBASE-25144 Add Hadoop-3.3.0 to personality hadoopcheck (#2492)
Now that Hadoop 3.3.0 is released, let's figure out where it goes in
our testing matrix. Start by adding it to precommit checks.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Matt Foley <mattf@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-10-06 12:00:56 -07:00
Nick Dimiduk 16251dbb53
HBASE-25143 Remove branch-1.3 from precommit and docs (#2491)
Following the announcement [0] to EOL branch-1.3, update the precommit
script to not consider this branch any longer, and refresh mentions of
this branch in the doc.

[0]: https://lists.apache.org/thread.html/r9552e9085aaac2a43f8b26b866d34825a84a9be7f19118ac560d14de%40%3Cuser.hbase.apache.org%3E

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-10-05 11:51:59 -07:00
Istvan Toth d1a3b66074
HBASE-25081 Up the container nproc uplimit to 30000 (#2439) 2020-09-22 08:53:15 -07:00
Guanghao Zhang d2eb69df77 HBASE-24876 Fix the flaky job url in hbase-personality.sh (#2253)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-08-13 16:50:25 +08:00
Josh Elser e54eec71c8 HBASE-24280 Skip explicit hadoop3 profile activation on master
On 2.x branches, we need to explicitly activate profiles for H3. On
master, all H2 support is dropped which means no special profiles are
required for H3 (though, there is still a profile there to encapsulate
H3 logic).

We need to make sure that the yetus invocation can correctly pass down
any profile information into the personality, so we activate the exact
profiles we want.

Closes #1609

Co-authored-by: Istvan Toth <stoty@cloudera.com>
Signed-off-by: stack <stack@apache.org>
2020-05-01 18:19:11 -04:00
Duo Zhang af17b3ccd6
HBASE-24218 Add hadoop 3.2.x in hadoop check (#1559)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-22 10:51:46 +08:00
Duo Zhang 2846ea4700 HBASE-24198 Addendum add echo_and_redirect back when running hadoopcheck 2020-04-18 10:54:17 +08:00
Duo Zhang b02064f936
HBASE-24198 Skip hadoop2 check for master branch (#1536) 2020-04-18 09:20:16 +08:00
Duo Zhang 1f66806c96
HBASE-24170 Remove hadoop-2.0 profile (#1495)
Signed-off-by: stack <stack@apache.org>
2020-04-16 18:57:40 +08:00
Michael Stack c2f2553490
HBASE-24126 Up the container nproc uplimit from 10000 to 12500 (#1504) 2020-04-13 14:59:16 -07:00
Michael Stack b843edb141 HBASE-24150 Allow module tests run in parallel (#1464)
Pass --threads=2 to mvn when yetus runs so some parallelism
when dependencies allow.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-13 11:54:16 -07:00
stack 3cf2df2ca0 Revert "HBASE-24126 Up the container nproc uplimit from 10000 to 12500 (#1450)"
This reverts commit 14342b6527.
2020-04-12 18:29:43 -07:00
Michael Stack 14342b6527
HBASE-24126 Up the container nproc uplimit from 10000 to 12500 (#1450)
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-04-10 15:57:47 -07:00
Nick Dimiduk e5a8435b48 HBASE-24078 SpotBugs check automatically skip inapplicable modules
Without patch, the spotbugs results include warnings for inapplicable
modules:

+---------------------------------------------------------------------------
|      |                |            | Compile Tests
+---------------------------------------------------------------------------
|   0  |        mvndep  |   0m 11s   | Maven dependency ordering
|  +1  |    mvninstall  |   2m 47s   | the source passed
|   0  |      spotbugs  |   0m 11s   | branch/hbase-annotations no spotbugs
|      |                |            | output file (spotbugsXml.xml)
|   0  |      spotbugs  |   0m 14s   | branch/hbase-assembly no spotbugs output
|      |                |            | file (spotbugsXml.xml)
|   0  |      spotbugs  |   0m 07s   |
|      |                |            | branch/hbase-archetypes/hbase-archetype-bu
|      |                |            | ilder no spotbugs output file
|      |                |            | (spotbugsXml.xml)
+---------------------------------------------------------------------------

With patch,

+---------------------------------------------------------------------------
|      |                |            | Compile Tests
+---------------------------------------------------------------------------
|   0  |        mvndep  |   0m 13s   | Maven dependency ordering
|  +1  |    mvninstall  |   3m 09s   | the source passed
|  +1  |      spotbugs  |  17m 09s   | the source passed
+---------------------------------------------------------------------------

Signed-off-by: stack <stack@apache.org>
2020-03-31 09:22:46 -07:00
Nick Dimiduk aa53493ae8 HBASE-24002 shadedjars check does not propagate --hadoop-profile
This implementation is almost surely incorrect. Personality
initialization parses the `--hadoop-profile` argument and sets
`HADOOP_PROFILE`. That value is then used to build an `extras` value
that is passed  along to module initialization. I'm guessing that the
`extras` value need to be honored down in the shadedjars module. I'm
not clear on how to make that work (need to study the interfaces at
play here), so taking the more ham-handed approach of referring to
`HADOOP_PROFILE`. I'm not sure if this will even work, or if it will
only work because the `foo_yetus.sh` scripts happen to use a variable
of the same name.

Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: stack <stack@apache.org>
2020-03-24 09:31:40 -07:00
Duo Zhang 9190a6b59a
HBASE-23989 Consider 2.10.0 in our support matrix (#1290)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-18 09:58:16 +08:00
Duo Zhang c5aa2bc121
HBASE-23077 move entirely to spotbugs (#1265)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-12 10:18:09 +08:00
Nick Dimiduk 055c07c7b7 HBASE-23767 Add JDK11 compilation and unit test support to Github precommit
Rebuild our Dockerfile with support for multiple JDK versions. Use
multiple stages in the Jenkinsfile instead of yetus's multijdk because
of YETUS-953. Run those multiple stages in parallel to speed up
results.

Note that multiple stages means multiple Yetus invocations means
multiple comments on the PreCommit. This should become more obvious to
users once we can make use of GitHub Checks API, HBASE-23902.

closes #1183

Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-05 11:46:15 -08:00
Duo Zhang 29ea96fad8 HBASE-23921 Findbugs is OOM on master 2020-03-04 12:23:36 +08:00
stack d554689cc4 HBASE-23779 Up the default fork count; make count relative to CPU count (#1108)
FINAL ADDENDUM. Removes changes to dev-support/hbase-personality
 leaving it as it was. All else about HBASE-23779 change remains
 changing surefire fork counts to be dependent on cpu count.
2020-02-07 15:53:15 -08:00
stack ef380e0a2e HBASE-23779 Up the default fork count; make count relative to CPU count (#1108)
ADDENDUM: Refactor that comes of discussion up on https://github.com/apache/yetus/pull/86
 because what I committed originally, and amended in a subsequent
 ADDENDUM is not taking effect.
2020-02-07 09:40:51 -08:00
stack 867b1e9cbc HBASE-23779 Up the default fork count; make count relative to CPU count (#1108)
ADDENDUM: Try this way of setting MAVEN_ARGS
2020-02-05 21:47:34 -08:00
Michael Stack b49ec58073 HBASE-23779 Up the default fork count; make count relative to CPU count (#1108)
Set the fork count for first and second parts to be 0.5C. Add a bit of
doc too on this as well as some qualification on our test categories.
Also adds -T0.5C to MAVEN_ARGS in the hbase personality.
2020-02-05 11:09:44 -08:00
Nick Dimiduk 1cacf27d5c
HBASE-23793 Increase maven heap allocation to 4G in Yetus personality (#1122)
I saw this over on
https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-2/2447/console. Looks
like we need to bump the memory allocation for maven. I wonder if this
is the underlying cause of HBASE-22470.

```
6:38:47  ============================================================================
16:38:47  ============================================================================
16:38:47                                Finished build.
16:38:47  ============================================================================
16:38:47  ============================================================================
16:38:47
16:38:47
Post stage
[Pipeline] stash
16:38:48  Warning: overwriting stash 'hadoop2-result'
16:38:48  Stashed 1 file(s)
[Pipeline] junit
16:38:48  Recording test results
16:38:54  Remote call on H2 failed
Error when executing always post condition:
java.io.IOException: Remote call on H2 failed
	at hudson.remoting.Channel.call(Channel.java:963)
	at hudson.FilePath.act(FilePath.java:1072)
	at hudson.FilePath.act(FilePath.java:1061)
	at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:114)
	at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:137)
	at hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:167)
	at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:52)
	at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:25)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at com.sun.org.apache.xerces.internal.util.XMLStringBuffer.append(XMLStringBuffer.java:208)
	at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.scanData(XMLEntityScanner.java:1515)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanCDATASection(XMLDocumentFragmentScannerImpl.java:1654)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:3014)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
	at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:842)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771)
	at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
	at org.dom4j.io.SAXReader.read(SAXReader.java:465)
	at org.dom4j.io.SAXReader.read(SAXReader.java:343)
	at hudson.tasks.junit.SuiteResult.parse(SuiteResult.java:178)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:348)
	at hudson.tasks.junit.TestResult.parsePossiblyEmpty(TestResult.java:281)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:206)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:178)
	at hudson.tasks.junit.TestResult.<init>(TestResult.java:143)
	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:146)
	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:118)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3052)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	... 4 more

[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
16:38:54  Failed in branch yetus jdk8 hadoop2 checks
```

Signed-off-by: stack <stack@apache.org>
2020-02-04 09:14:01 -08:00
Sean Busbey d5dcc090c1
HBASE-23228 Allow for jdk8 specific modules on branch-1 in precommit/nightly testing (#804)
Signed-off-by: Josh Elser <elserj@apache.org>
2019-11-08 10:07:24 -06:00
Sean Busbey bcad0d9f98 HBASE-22955 Properly set jdk7 as JAVA_HOME for branches-1 precommit and nightly. (#621)
master/branches-2 specific changes: work around yetus overwriting JAVA_HOME
in the container with the host JAVA_HOME.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
(cherry picked from commit 41990ba20a)
2019-09-16 22:51:24 -05:00
Peter Somogyi 51b6ce391b
HBASE-22981 Remove unused flags for Yetus (#591)
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-09-06 14:56:05 +02:00
Duo Zhang da144888d0
HBASE-22724 Add a emoji on the vote table for pre commit result on github (#568)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Stack <stack@apache.org>
2019-09-04 08:52:28 +08:00
Sean Busbey b4d5a9a8e1 HBASE-22518 yetus personality is treating branch-1.4 like earlier branches for hadoopcheck
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-06-03 09:16:34 -05:00
Duo Zhang e4f4ae835f HBASE-22500 Modify pom and jenkins jobs for hadoop versions
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-05-30 22:31:32 +08:00
Guanghao Zhang de579cd243 HBASE-22462 (addendum) Should run a 'mvn install' at the end of hadoop check in pre commit job 2019-05-24 09:23:20 +08:00
Guanghao Zhang 6b899ccbbc HBASE-22462 Should run a 'mvn install' at the end of hadoop check in pre commit job 2019-05-23 20:57:23 +08:00
zhangduo b74df9b747 HBASE-22442 Nightly build is failing with hadoop 3.x 2019-05-19 07:54:29 +08:00
Duo Zhang 5a578f953b HBASE-20970 Update hadoop check versions for hadoop3 in hbase-personality
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-05-17 10:01:51 +08:00
Duo Zhang 083605df8f HBASE-22399 Change default hadoop-two.version to 2.8.x and remove the 2.7.x hadoop checks 2019-05-14 16:45:20 +08:00