Commit Graph

77 Commits

Author SHA1 Message Date
Duo Zhang 15a46388f1 HBASE-24218 Add hadoop 3.2.x in hadoop check (#1559)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-22 10:52:53 +08:00
Duo Zhang 6e95575669
HBASE-24213 Backport the pre commit changes for HBASE-24169 to all branches (#1548)
Signed-off-by: stack <stack@apache.org>
2020-04-22 10:26:10 +08:00
Michael Stack 62a488b946 HBASE-24126 Up the container nproc uplimit from 10000 to 12500 (#1504) 2020-04-14 17:36:06 -07:00
Michael Stack 9eb49172d0
HBASE-24150 Allow module tests run in parallel (#1464)
Pass --threads=2 to mvn when yetus runs so some parallelism
when dependencies allow.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-09 14:41:41 -07:00
stack 953c326498 HBASE-24126 Up the container nproc uplimit from 10000 to 12500
Addendum number 3; set the yetus proclimit.
2020-04-07 10:22:24 -07:00
stack ba87ffed09 Revert "HBASE-24072 Nightlies reporting OutOfMemoryError: unable to create new native thread."
Revert because applied with wrong JIRA number.

This reverts commit 9c60c5fddb.
2020-04-07 10:17:48 -07:00
stack 9c60c5fddb HBASE-24072 Nightlies reporting OutOfMemoryError: unable to create new native thread.
Up the nproc in hbase-personality. Add test that prints out env.
2020-04-07 07:37:48 -07:00
stack 01715fbbf6 HBASE-24126 Up the container nproc uplimit from 10000 to 12500
Addendum number 2; docker doesn't like the ulimit arg as is.
2020-04-06 18:24:09 -07:00
stack 25773c681b HBASE-24126 Up the container nproc uplimit from 10000 to 12500
ADDENDUM: Add to hbase-personality
2020-04-06 16:29:46 -07:00
Nick Dimiduk 593f44c928 HBASE-24078 SpotBugs check automatically skip inapplicable modules
Without patch, the spotbugs results include warnings for inapplicable
modules:

+---------------------------------------------------------------------------
|      |                |            | Compile Tests
+---------------------------------------------------------------------------
|   0  |        mvndep  |   0m 11s   | Maven dependency ordering
|  +1  |    mvninstall  |   2m 47s   | the source passed
|   0  |      spotbugs  |   0m 11s   | branch/hbase-annotations no spotbugs
|      |                |            | output file (spotbugsXml.xml)
|   0  |      spotbugs  |   0m 14s   | branch/hbase-assembly no spotbugs output
|      |                |            | file (spotbugsXml.xml)
|   0  |      spotbugs  |   0m 07s   |
|      |                |            | branch/hbase-archetypes/hbase-archetype-bu
|      |                |            | ilder no spotbugs output file
|      |                |            | (spotbugsXml.xml)
+---------------------------------------------------------------------------

With patch,

+---------------------------------------------------------------------------
|      |                |            | Compile Tests
+---------------------------------------------------------------------------
|   0  |        mvndep  |   0m 13s   | Maven dependency ordering
|  +1  |    mvninstall  |   3m 09s   | the source passed
|  +1  |      spotbugs  |  17m 09s   | the source passed
+---------------------------------------------------------------------------

Signed-off-by: stack <stack@apache.org>
2020-03-31 09:33:01 -07:00
Nihal Jain 6f675e269c HBASE-20467 Precommit personality should only run checkstyle once if we're going to run it at the root
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-31 09:33:01 -07:00
Nick Dimiduk aba0737acb HBASE-24002 shadedjars check does not propagate --hadoop-profile
This implementation is almost surely incorrect. Personality
initialization parses the `--hadoop-profile` argument and sets
`HADOOP_PROFILE`. That value is then used to build an `extras` value
that is passed  along to module initialization. I'm guessing that the
`extras` value need to be honored down in the shadedjars module. I'm
not clear on how to make that work (need to study the interfaces at
play here), so taking the more ham-handed approach of referring to
`HADOOP_PROFILE`. I'm not sure if this will even work, or if it will
only work because the `foo_yetus.sh` scripts happen to use a variable
of the same name.

Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: stack <stack@apache.org>
2020-03-24 09:35:31 -07:00
Duo Zhang 83388609d0
HBASE-24009 Backport the personality changes in HBASE-23989 to all active branches (#1304)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-20 14:18:07 +08:00
Duo Zhang 7eeb6a0815 HBASE-23077 move entirely to spotbugs (#1265)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-12 11:42:23 +08:00
Nick Dimiduk e0f9d0f562 HBASE-23767 Add JDK11 compilation and unit test support to Github precommit
Rebuild our Dockerfile with support for multiple JDK versions. Use
multiple stages in the Jenkinsfile instead of yetus's multijdk because
of YETUS-953. Run those multiple stages in parallel to speed up
results.

Note that multiple stages means multiple Yetus invocations means
multiple comments on the PreCommit. This should become more obvious to
users once we can make use of GitHub Checks API, HBASE-23902.

closes #1183

Signed-off-by: Sean Busbey <busbey@apache.org>

Conflicts:
  dev-support/Jenkinsfile_GitHub
  dev-support/hbase-personality.sh
2020-03-05 13:59:01 -08:00
stack 450503989f HBASE-23779 Up the default fork count to make builds complete faster; make count relative to CPU count
ADDENDUM: Purge changes made to hbase-personality as they disstabilize
jenkins build. All else remains, the changing of forkcount for tests.
2020-02-07 16:00:50 -08:00
Nick Dimiduk 5cda68899a HBASE-23793 Increase maven heap allocation to 4G in Yetus personality (#1122)
I saw this over on
https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-2/2447/console. Looks
like we need to bump the memory allocation for maven. I wonder if this
is the underlying cause of HBASE-22470.

```
6:38:47  ============================================================================
16:38:47  ============================================================================
16:38:47                                Finished build.
16:38:47  ============================================================================
16:38:47  ============================================================================
16:38:47
16:38:47
Post stage
[Pipeline] stash
16:38:48  Warning: overwriting stash 'hadoop2-result'
16:38:48  Stashed 1 file(s)
[Pipeline] junit
16:38:48  Recording test results
16:38:54  Remote call on H2 failed
Error when executing always post condition:
java.io.IOException: Remote call on H2 failed
	at hudson.remoting.Channel.call(Channel.java:963)
	at hudson.FilePath.act(FilePath.java:1072)
	at hudson.FilePath.act(FilePath.java:1061)
	at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:114)
	at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:137)
	at hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:167)
	at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:52)
	at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:25)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at com.sun.org.apache.xerces.internal.util.XMLStringBuffer.append(XMLStringBuffer.java:208)
	at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.scanData(XMLEntityScanner.java:1515)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanCDATASection(XMLDocumentFragmentScannerImpl.java:1654)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:3014)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
	at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:842)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771)
	at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
	at org.dom4j.io.SAXReader.read(SAXReader.java:465)
	at org.dom4j.io.SAXReader.read(SAXReader.java:343)
	at hudson.tasks.junit.SuiteResult.parse(SuiteResult.java:178)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:348)
	at hudson.tasks.junit.TestResult.parsePossiblyEmpty(TestResult.java:281)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:206)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:178)
	at hudson.tasks.junit.TestResult.<init>(TestResult.java:143)
	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:146)
	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:118)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3052)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	... 4 more

[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
16:38:54  Failed in branch yetus jdk8 hadoop2 checks
```

Signed-off-by: stack <stack@apache.org>
2020-02-05 14:20:34 -08:00
Michael Stack 4c8ac72194
HBASE-23779 Up the default fork count; make count relative to CPU count (#1108)
Set the fork count for first and second parts to be 0.5C. Add a bit of
doc too on this as well as some qualification on our test categories.
Also adds -T0.5C to MAVEN_ARGS in the hbase personality.
2020-02-04 20:47:02 -08:00
Sean Busbey 82037ee3f2 HBASE-23228 Allow for jdk8 specific modules on branch-1 in precommit/nightly testing (#804)
Signed-off-by: Josh Elser <elserj@apache.org>
2019-11-08 10:08:56 -06:00
Sean Busbey 5cedb09b65 HBASE-22955 Properly set jdk7 as JAVA_HOME for branches-1 precommit and nightly. (#621)
master/branches-2 specific changes: work around yetus overwriting JAVA_HOME
in the container with the host JAVA_HOME.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
(cherry picked from commit 41990ba20a)
(cherry picked from commit bcad0d9f98)
2019-09-16 22:57:16 -05:00
Peter Somogyi af86b7bd06 HBASE-22981 Remove unused flags for Yetus (#591)
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-09-06 14:58:21 +02:00
Duo Zhang 48f0b1805e HBASE-22724 Add a emoji on the vote table for pre commit result on github (#568)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Stack <stack@apache.org>
2019-09-04 09:04:59 +08:00
Sean Busbey 72caf79bb3 HBASE-22518 yetus personality is treating branch-1.4 like earlier branches for hadoopcheck
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit b4d5a9a8e1)
2019-06-03 09:26:46 -05:00
Duo Zhang 59973d2024 HBASE-22500 Modify pom and jenkins jobs for hadoop versions
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-05-30 22:31:47 +08:00
Guanghao Zhang c00cc4d9d6 HBASE-22462 (addendum) Should run a 'mvn install' at the end of hadoop check in pre commit job 2019-05-24 12:07:15 +08:00
Guanghao Zhang b469d06bd4 HBASE-22462 Should run a 'mvn install' at the end of hadoop check in pre commit job 2019-05-24 12:07:09 +08:00
zhangduo 6a28e76f7e HBASE-22442 Nightly build is failing with hadoop 3.x 2019-05-19 07:56:02 +08:00
Duo Zhang 5537de38b3 HBASE-20970 Update hadoop check versions for hadoop3 in hbase-personality
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-05-17 09:51:59 +08:00
Duo Zhang 6bd6a953a7 HBASE-22399 Change default hadoop-two.version to 2.8.x and remove the 2.7.x hadoop checks 2019-05-14 16:45:32 +08:00
zhangduo 4562bf608c HBASE-22174 Remove error prone from our precommit javac check 2019-04-06 19:12:51 +08:00
zhangduo a33b3401cf HBASE-22100 False positive for error prone warnings in pre commit job
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-03-28 13:06:53 +08:00
Peter Somogyi 0b086087b2 HBASE-21547 Precommit uses master flaky list for other branches
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-01-02 08:31:06 +01:00
Sean Busbey f9a3d8acc4 HBASE-21058 When testing branches one, the PDF ref guide is named 'book.pdf'
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2018-08-21 09:46:03 -05:00
Sean Busbey 98da781ecf HBASE-21074 JDK7 builds need to be done with TLSv1.2.
* for master and branches-2 we skip the jdk7 stuff

Signed-off-by: Michael Stack <stack@apache.org>
2018-08-21 09:15:26 -05:00
Yu Li 492c0fcd2e HBASE-20838 Include hbase-server in precommit test if CommonFSUtils is changed 2018-07-12 10:17:38 +08:00
Mike Drob b04c976fe6 HBASE-20478 Update checkstyle to v8.2
Cannot go to latest (8.9) yet due to
  https://github.com/checkstyle/checkstyle/issues/5279

* move hbaseanti import checks to checkstyle
* implment a few missing equals checks, and ignore one
* fix lots of javadoc errors

Signed-off-by: Sean Busbey <busbey@apache.org>
2018-06-18 14:02:40 -07:00
Sean Busbey a4cb98b1cd HBASE-20733 QABot should run checkstyle tests if the checkstyle configs change
Signed-off-by: Mike Drob <mdrob@apache.org>
2018-06-14 20:26:42 -05:00
Sean Busbey 9431ee3f49 HBASE-20501 update minimum supported Hadoop version to 2.7.1.
Signed-off-by: Andrew Purtell <apurtell@apache.org>

 Conflicts:
	src/main/asciidoc/_chapters/configuration.adoc
2018-05-30 21:28:06 -05:00
Nihal Jain 8c7293c514 HBASE-20438 Add an HBase antipattern check for reintroducing commons-logging
Signed-off-by: Sean Busbey <busbey@apache.org>
2018-04-20 11:58:39 -05:00
Sean Busbey bd2dddae60 HBASE-20379 shadedjars yetus plugin should add a footer link
Signed-off-by: Mike Drob <mdrob@apache.org>
2018-04-13 10:48:05 -05:00
Mike Drob a50d9f4351 HBASE-20356 Make skipping protoc possible 2018-04-12 14:54:03 -05:00
Sean Busbey 1ef2450ca9 HBASE-20068 personality tests for Apache Yetus should use the maven plugin to exec maven.
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Reid Chan <reidddchan@outlook.com>
2018-04-11 01:30:46 -05:00
Sean Busbey 469b57bba0 HBASE-15466 test personality should recognize docs-only patch.
* add a new test to build the refguide specifically instead of site
* check for changes to src/main/asciidoc or src/main/xslt and run that test and only that test
* check for changes to the hbase-default.xml file and build the refguide if found (but maybe other tests too)
* fallback to relying on the yetus default for other changes
* fix some missing start_clock entries that cause longer-than-actual reported test time.

Signed-off-by: Mike Drob <mdrob@apache.org>
2018-04-09 10:37:49 -05:00
Mike Drob d42886f4fe HBASE-20153 Add error prone to precommit 2018-03-12 16:06:08 -05:00
Mike Drob 32c5118f6f HBASE-20164 add hadoopcheck link to table footer 2018-03-09 15:41:12 -06:00
Michael Stack 1dc640c296 HBASE-19901 HBASE-19901 Up yetus proclimit on nightlies; REAPPLY TO TEST 2018-02-01 20:27:05 -08:00
Michael Stack 12a6af0de5 Revert "HBASE-19901 HBASE-19901 Up yetus proclimit on nightlies"
This reverts commit 2e6bc12441.
2018-02-01 18:04:59 -08:00
Michael Stack ce6c3d96ea HBASE-19901 Up yetus proclimit on nightlies; ADDENDUM -- remove docker mem sizing for the moment... 2018-02-01 16:39:05 -08:00
Michael Stack fe6c06ad6c HBASE-19901 Up yetus proclimit on nightlies; ADDENDUM -- up proclimit from 6k to 10k and set Docker memory limit of 20G (default is 4G) 2018-02-01 14:08:24 -08:00
Michael Stack afdfa54cd3 HBASE-19901 HBASE-19901 Up yetus proclimit on nightlies 2018-01-30 22:26:58 -08:00