Commit Graph

81 Commits

Author SHA1 Message Date
Michael Stack b843edb141 HBASE-24150 Allow module tests run in parallel (#1464)
Pass --threads=2 to mvn when yetus runs so some parallelism
when dependencies allow.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-13 11:54:16 -07:00
stack 3cf2df2ca0 Revert "HBASE-24126 Up the container nproc uplimit from 10000 to 12500 (#1450)"
This reverts commit 14342b6527.
2020-04-12 18:29:43 -07:00
Michael Stack 14342b6527
HBASE-24126 Up the container nproc uplimit from 10000 to 12500 (#1450)
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-04-10 15:57:47 -07:00
Nick Dimiduk e5a8435b48 HBASE-24078 SpotBugs check automatically skip inapplicable modules
Without patch, the spotbugs results include warnings for inapplicable
modules:

+---------------------------------------------------------------------------
|      |                |            | Compile Tests
+---------------------------------------------------------------------------
|   0  |        mvndep  |   0m 11s   | Maven dependency ordering
|  +1  |    mvninstall  |   2m 47s   | the source passed
|   0  |      spotbugs  |   0m 11s   | branch/hbase-annotations no spotbugs
|      |                |            | output file (spotbugsXml.xml)
|   0  |      spotbugs  |   0m 14s   | branch/hbase-assembly no spotbugs output
|      |                |            | file (spotbugsXml.xml)
|   0  |      spotbugs  |   0m 07s   |
|      |                |            | branch/hbase-archetypes/hbase-archetype-bu
|      |                |            | ilder no spotbugs output file
|      |                |            | (spotbugsXml.xml)
+---------------------------------------------------------------------------

With patch,

+---------------------------------------------------------------------------
|      |                |            | Compile Tests
+---------------------------------------------------------------------------
|   0  |        mvndep  |   0m 13s   | Maven dependency ordering
|  +1  |    mvninstall  |   3m 09s   | the source passed
|  +1  |      spotbugs  |  17m 09s   | the source passed
+---------------------------------------------------------------------------

Signed-off-by: stack <stack@apache.org>
2020-03-31 09:22:46 -07:00
Nick Dimiduk aa53493ae8 HBASE-24002 shadedjars check does not propagate --hadoop-profile
This implementation is almost surely incorrect. Personality
initialization parses the `--hadoop-profile` argument and sets
`HADOOP_PROFILE`. That value is then used to build an `extras` value
that is passed  along to module initialization. I'm guessing that the
`extras` value need to be honored down in the shadedjars module. I'm
not clear on how to make that work (need to study the interfaces at
play here), so taking the more ham-handed approach of referring to
`HADOOP_PROFILE`. I'm not sure if this will even work, or if it will
only work because the `foo_yetus.sh` scripts happen to use a variable
of the same name.

Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: stack <stack@apache.org>
2020-03-24 09:31:40 -07:00
Duo Zhang 9190a6b59a
HBASE-23989 Consider 2.10.0 in our support matrix (#1290)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-18 09:58:16 +08:00
Duo Zhang c5aa2bc121
HBASE-23077 move entirely to spotbugs (#1265)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-12 10:18:09 +08:00
Nick Dimiduk 055c07c7b7 HBASE-23767 Add JDK11 compilation and unit test support to Github precommit
Rebuild our Dockerfile with support for multiple JDK versions. Use
multiple stages in the Jenkinsfile instead of yetus's multijdk because
of YETUS-953. Run those multiple stages in parallel to speed up
results.

Note that multiple stages means multiple Yetus invocations means
multiple comments on the PreCommit. This should become more obvious to
users once we can make use of GitHub Checks API, HBASE-23902.

closes #1183

Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-05 11:46:15 -08:00
Duo Zhang 29ea96fad8 HBASE-23921 Findbugs is OOM on master 2020-03-04 12:23:36 +08:00
stack d554689cc4 HBASE-23779 Up the default fork count; make count relative to CPU count (#1108)
FINAL ADDENDUM. Removes changes to dev-support/hbase-personality
 leaving it as it was. All else about HBASE-23779 change remains
 changing surefire fork counts to be dependent on cpu count.
2020-02-07 15:53:15 -08:00
stack ef380e0a2e HBASE-23779 Up the default fork count; make count relative to CPU count (#1108)
ADDENDUM: Refactor that comes of discussion up on https://github.com/apache/yetus/pull/86
 because what I committed originally, and amended in a subsequent
 ADDENDUM is not taking effect.
2020-02-07 09:40:51 -08:00
stack 867b1e9cbc HBASE-23779 Up the default fork count; make count relative to CPU count (#1108)
ADDENDUM: Try this way of setting MAVEN_ARGS
2020-02-05 21:47:34 -08:00
Michael Stack b49ec58073 HBASE-23779 Up the default fork count; make count relative to CPU count (#1108)
Set the fork count for first and second parts to be 0.5C. Add a bit of
doc too on this as well as some qualification on our test categories.
Also adds -T0.5C to MAVEN_ARGS in the hbase personality.
2020-02-05 11:09:44 -08:00
Nick Dimiduk 1cacf27d5c
HBASE-23793 Increase maven heap allocation to 4G in Yetus personality (#1122)
I saw this over on
https://builds.apache.org/view/H-L/view/HBase/job/HBase%20Nightly/job/branch-2/2447/console. Looks
like we need to bump the memory allocation for maven. I wonder if this
is the underlying cause of HBASE-22470.

```
6:38:47  ============================================================================
16:38:47  ============================================================================
16:38:47                                Finished build.
16:38:47  ============================================================================
16:38:47  ============================================================================
16:38:47
16:38:47
Post stage
[Pipeline] stash
16:38:48  Warning: overwriting stash 'hadoop2-result'
16:38:48  Stashed 1 file(s)
[Pipeline] junit
16:38:48  Recording test results
16:38:54  Remote call on H2 failed
Error when executing always post condition:
java.io.IOException: Remote call on H2 failed
	at hudson.remoting.Channel.call(Channel.java:963)
	at hudson.FilePath.act(FilePath.java:1072)
	at hudson.FilePath.act(FilePath.java:1061)
	at hudson.tasks.junit.JUnitParser.parseResult(JUnitParser.java:114)
	at hudson.tasks.junit.JUnitResultArchiver.parse(JUnitResultArchiver.java:137)
	at hudson.tasks.junit.JUnitResultArchiver.parseAndAttach(JUnitResultArchiver.java:167)
	at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:52)
	at hudson.tasks.junit.pipeline.JUnitResultsStepExecution.run(JUnitResultsStepExecution.java:25)
	at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.OutOfMemoryError: Java heap space
	at com.sun.org.apache.xerces.internal.util.XMLStringBuffer.append(XMLStringBuffer.java:208)
	at com.sun.org.apache.xerces.internal.impl.XMLEntityScanner.scanData(XMLEntityScanner.java:1515)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanCDATASection(XMLDocumentFragmentScannerImpl.java:1654)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl$FragmentContentDriver.next(XMLDocumentFragmentScannerImpl.java:3014)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentScannerImpl.next(XMLDocumentScannerImpl.java:602)
	at com.sun.org.apache.xerces.internal.impl.XMLNSDocumentScannerImpl.next(XMLNSDocumentScannerImpl.java:112)
	at com.sun.org.apache.xerces.internal.impl.XMLDocumentFragmentScannerImpl.scanDocument(XMLDocumentFragmentScannerImpl.java:505)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:842)
	at com.sun.org.apache.xerces.internal.parsers.XML11Configuration.parse(XML11Configuration.java:771)
	at com.sun.org.apache.xerces.internal.parsers.XMLParser.parse(XMLParser.java:141)
	at com.sun.org.apache.xerces.internal.parsers.AbstractSAXParser.parse(AbstractSAXParser.java:1213)
	at com.sun.org.apache.xerces.internal.jaxp.SAXParserImpl$JAXPSAXParser.parse(SAXParserImpl.java:643)
	at org.dom4j.io.SAXReader.read(SAXReader.java:465)
	at org.dom4j.io.SAXReader.read(SAXReader.java:343)
	at hudson.tasks.junit.SuiteResult.parse(SuiteResult.java:178)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:348)
	at hudson.tasks.junit.TestResult.parsePossiblyEmpty(TestResult.java:281)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:206)
	at hudson.tasks.junit.TestResult.parse(TestResult.java:178)
	at hudson.tasks.junit.TestResult.<init>(TestResult.java:143)
	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:146)
	at hudson.tasks.junit.JUnitParser$ParseResultCallable.invoke(JUnitParser.java:118)
	at hudson.FilePath$FileCallableWrapper.call(FilePath.java:3052)
	at hudson.remoting.UserRequest.perform(UserRequest.java:212)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:369)
	at hudson.remoting.InterceptingExecutorService$1.call(InterceptingExecutorService.java:72)
	... 4 more

[Pipeline] }
[Pipeline] // withEnv
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
16:38:54  Failed in branch yetus jdk8 hadoop2 checks
```

Signed-off-by: stack <stack@apache.org>
2020-02-04 09:14:01 -08:00
Sean Busbey d5dcc090c1
HBASE-23228 Allow for jdk8 specific modules on branch-1 in precommit/nightly testing (#804)
Signed-off-by: Josh Elser <elserj@apache.org>
2019-11-08 10:07:24 -06:00
Sean Busbey bcad0d9f98 HBASE-22955 Properly set jdk7 as JAVA_HOME for branches-1 precommit and nightly. (#621)
master/branches-2 specific changes: work around yetus overwriting JAVA_HOME
in the container with the host JAVA_HOME.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
(cherry picked from commit 41990ba20a)
2019-09-16 22:51:24 -05:00
Peter Somogyi 51b6ce391b
HBASE-22981 Remove unused flags for Yetus (#591)
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-09-06 14:56:05 +02:00
Duo Zhang da144888d0
HBASE-22724 Add a emoji on the vote table for pre commit result on github (#568)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Stack <stack@apache.org>
2019-09-04 08:52:28 +08:00
Sean Busbey b4d5a9a8e1 HBASE-22518 yetus personality is treating branch-1.4 like earlier branches for hadoopcheck
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-06-03 09:16:34 -05:00
Duo Zhang e4f4ae835f HBASE-22500 Modify pom and jenkins jobs for hadoop versions
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-05-30 22:31:32 +08:00
Guanghao Zhang de579cd243 HBASE-22462 (addendum) Should run a 'mvn install' at the end of hadoop check in pre commit job 2019-05-24 09:23:20 +08:00
Guanghao Zhang 6b899ccbbc HBASE-22462 Should run a 'mvn install' at the end of hadoop check in pre commit job 2019-05-23 20:57:23 +08:00
zhangduo b74df9b747 HBASE-22442 Nightly build is failing with hadoop 3.x 2019-05-19 07:54:29 +08:00
Duo Zhang 5a578f953b HBASE-20970 Update hadoop check versions for hadoop3 in hbase-personality
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-05-17 10:01:51 +08:00
Duo Zhang 083605df8f HBASE-22399 Change default hadoop-two.version to 2.8.x and remove the 2.7.x hadoop checks 2019-05-14 16:45:20 +08:00
zhangduo 86f9443eeb HBASE-22174 Remove error prone from our precommit javac check 2019-04-06 13:51:02 +08:00
zhangduo c8d04cf127 HBASE-22100 False positive for error prone warnings in pre commit job
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-03-28 13:06:26 +08:00
Peter Somogyi f0b50a8f9b HBASE-21547 Precommit uses master flaky list for other branches
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-01-02 08:26:57 +01:00
Sean Busbey 588ff921c1 HBASE-21058 When testing branches one, the PDF ref guide is named 'book.pdf'
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2018-08-21 09:45:26 -05:00
Sean Busbey 23d54f8587 HBASE-21074 JDK7 builds need to be done with TLSv1.2.
* for master and branches-2 we skip the jdk7 stuff

Signed-off-by: Michael Stack <stack@apache.org>
2018-08-21 09:14:48 -05:00
Yu Li c55acb0b03 HBASE-20838 Include hbase-server in precommit test if CommonFSUtils is changed 2018-07-12 10:16:11 +08:00
Sean Busbey 04db900772 HBASE-20733 QABot should run checkstyle tests if the checkstyle configs change
Signed-off-by: Mike Drob <mdrob@apache.org>
2018-06-14 20:24:29 -05:00
Sean Busbey c19fbf2be0 HBASE-20501 update minimum supported Hadoop version to 2.7.1.
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2018-05-30 21:22:42 -05:00
Mike Drob a110e1eff5 HBASE-20478 Update checkstyle to v8.2
Cannot go to latest (8.9) yet due to
  https://github.com/checkstyle/checkstyle/issues/5279

* move hbaseanti import checks to checkstyle
* implment a few missing equals checks, and ignore one
* fix lots of javadoc errors

Signed-off-by: Sean Busbey <busbey@apache.org>
2018-05-29 10:12:31 -05:00
Mike Drob e6220c8b60 Revert "HBASE-20478 move hbaseanti import checks to checkstyle"
Need to update version of checkstyle used for new features.

This reverts commit 90d103d6d5.
2018-04-26 13:36:13 -05:00
Mike Drob 90d103d6d5 HBASE-20478 move hbaseanti import checks to checkstyle
Signed-off-by: Sean Busbey <busbey@apache.org>
2018-04-26 09:28:59 -05:00
Nihal Jain d4768114e8 HBASE-20467 Precommit personality should only run checkstyle once if we're going to run it at the root
Signed-off-by: Sean Busbey <busbey@apache.org>
2018-04-26 07:54:25 -05:00
Nihal Jain 8219ec7493 HBASE-20438 Add an HBase antipattern check for reintroducing commons-logging
Signed-off-by: Sean Busbey <busbey@apache.org>
2018-04-20 11:54:01 -05:00
Sean Busbey 2f74afd6f4 HBASE-20379 shadedjars yetus plugin should add a footer link
Signed-off-by: Mike Drob <mdrob@apache.org>
2018-04-13 10:40:35 -05:00
Mike Drob 70d23214fb HBASE-20356 Make skipping protoc possible 2018-04-12 13:31:54 -05:00
Sean Busbey 83ee82d6a0 HBASE-20068 personality tests for Apache Yetus should use the maven plugin to exec maven.
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Reid Chan <reidddchan@outlook.com>
2018-04-11 01:33:14 -05:00
Sean Busbey ee87de9bfd HBASE-15466 test personality should recognize docs-only patch.
* add a new test to build the refguide specifically instead of site
* check for changes to src/main/asciidoc or src/main/xslt and run that test and only that test
* check for changes to the hbase-default.xml file and build the refguide if found (but maybe other tests too)
* fallback to relying on the yetus default for other changes
* fix some missing start_clock entries that cause longer-than-actual reported test time.

Signed-off-by: Mike Drob <mdrob@apache.org>
2018-04-09 10:37:28 -05:00
Mike Drob 9342e0091d HBASE-20153 Add error prone to precommit 2018-03-12 16:06:57 -05:00
Mike Drob 28279697c2 HBASE-20164 add hadoopcheck link to table footer 2018-03-09 15:40:28 -06:00
Michael Stack cb7bfc21da HBASE-19901 HBASE-19901 Up yetus proclimit on nightlies; REAPPLY TO TEST 2018-02-01 20:27:28 -08:00
Michael Stack 18eec8c1a5 Revert "HBASE-19901 HBASE-19901 Up yetus proclimit on nightlies"
This reverts commit 2e6bc12441.
2018-02-01 18:04:23 -08:00
Michael Stack 5f5ddf5541 HBASE-19901 Up yetus proclimit on nightlies; ADDENDUM -- remove docker mem sizing for the moment... 2018-02-01 16:39:27 -08:00
Michael Stack 0db7db3cd1 HBASE-19901 Up yetus proclimit on nightlies; ADDENDUM -- up proclimit from 6k to 10k and set Docker memory limit of 20G (default is 4G) 2018-02-01 14:08:04 -08:00
Michael Stack 2e6bc12441 HBASE-19901 HBASE-19901 Up yetus proclimit on nightlies 2018-01-30 22:25:24 -08:00
Apekshit Sharma a4272a9f08 HBASE-19570 Add hadoop3 tests to Nightly master/branch-2 runs.
Jenkins fails the whole build immediately if any stage fails. Hadoop2 tests run before Hadoop3 tests.
So Hadoop3 tests will run only if hadoop2 tests pass.
2017-12-21 17:44:04 -08:00