Commit Graph

16853 Commits

Author SHA1 Message Date
Nick Dimiduk 593f44c928 HBASE-24078 SpotBugs check automatically skip inapplicable modules
Without patch, the spotbugs results include warnings for inapplicable
modules:

+---------------------------------------------------------------------------
|      |                |            | Compile Tests
+---------------------------------------------------------------------------
|   0  |        mvndep  |   0m 11s   | Maven dependency ordering
|  +1  |    mvninstall  |   2m 47s   | the source passed
|   0  |      spotbugs  |   0m 11s   | branch/hbase-annotations no spotbugs
|      |                |            | output file (spotbugsXml.xml)
|   0  |      spotbugs  |   0m 14s   | branch/hbase-assembly no spotbugs output
|      |                |            | file (spotbugsXml.xml)
|   0  |      spotbugs  |   0m 07s   |
|      |                |            | branch/hbase-archetypes/hbase-archetype-bu
|      |                |            | ilder no spotbugs output file
|      |                |            | (spotbugsXml.xml)
+---------------------------------------------------------------------------

With patch,

+---------------------------------------------------------------------------
|      |                |            | Compile Tests
+---------------------------------------------------------------------------
|   0  |        mvndep  |   0m 13s   | Maven dependency ordering
|  +1  |    mvninstall  |   3m 09s   | the source passed
|  +1  |      spotbugs  |  17m 09s   | the source passed
+---------------------------------------------------------------------------

Signed-off-by: stack <stack@apache.org>
2020-03-31 09:33:01 -07:00
Nihal Jain 6f675e269c HBASE-20467 Precommit personality should only run checkstyle once if we're going to run it at the root
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-31 09:33:01 -07:00
stack b1eff98789 HBASE-24079 [Flakey Tests] Misc fixes and debug; fix BindException in Thrift tests; add waits on quota table to come online; etc.
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientAsyncPrefetchScanner.java
 Refactor to avoid NPE timing issue referencing lock during Construction.

hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
 Comment

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
 Refactor. Catch NPE during startup and return it instead as failed initialization.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplit.java
 Catch IndexOutOfBounds exception and convert to non-split request.

hbase-server/src/test/java/org/apache/hadoop/hbase/TestCachedClusterId.java
 Make less furious. Make it less flakie.

hbase-server/src/test/java/org/apache/hadoop/hbase/TestServerSideScanMetricsFromClientSide.java
 Debug. Catch exception to log, then rethrow.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java
 Guess that waiting longer on compaction to succeed may help make this
 less flakey.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java
 Be explicit about timestamping to avoid concurrent edit landing
 server-side and messing up test expectation.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMasterRegistry.java
 Add wait on meta before proceeding w/ test.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannersFromClientSide.java
 Be explicit that edits are distinct.

hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketCacheRefCnt.java
 Add @Ignore on RAM test... Fails sporadically.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestRegionMoveAndAbandon.java
 Add wait for all RegionServers going down before proceeding; was
 messing up RS accounting.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
 Make balancer test sloppier; less restrictive; would fail on occasion
 by being just outside test limits.

hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaObserverChoreRegionReports.java
 Add wait on quota table coming up; helps make this less flakie.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
 Be explicity about timestamps; see if helps w/ flakie failure.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
 Catch and ignore if issue in shutdown; don't care if after test.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerReportForDuty.java
 Comment.

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
 Add retry to see if helps w/ odd failure; grant hasn't propagated?

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestCellACLWithMultipleVersions.java
 Explicit w/ timestamps so no accidental overlap of puts.

hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftHttpServer.java
hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerCmdLine.java
 Hack to deal w/ BindException on startup.

hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift2/TestThrift2ServerCmdLine.java
 Use loopback.

hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift2/TestThriftHBaseServiceHandler.java
 Disable flakie test.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-03-30 16:46:48 -07:00
huaxiangsun 34ebdd6c9d
HBASE-24073 [flakey test] client.TestAsyncRegionAdminApi messed up compaction state. (#1387) (#1389)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-30 12:26:10 -07:00
Nick Dimiduk 227bcaeb88 HBASE-24007 Get `-PrunLargeTests` passing on JDK11 (#1351)
Minor tweaks required to get passing runs of `-PrunLargeTests`.
* Minimum Hadoop version is 3.2.0 due to
  [HADOOP-12760](https://issues.apache.org/jira/browse/HADOOP-12760).
* JDK11 looks like it consumes more memory than JDK8, so failures due
  to OOME see more common here. Bumping heap allocated to surefire
  forks allows better pass rate.

Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-30 09:48:58 -07:00
Peter Somogyi 244e47ca65 HBASE-24057 Add modules to mapreduce job classpaths (#1359)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-27 11:15:50 +01:00
Duo Zhang 7d842ca886
HBASE-24050 Deprecated PBType on all 2.x branches (#1354)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-27 10:43:14 +08:00
WenFeiYi 0433713b35 HBASE-24040 WALFactory.Providers.multiwal causes StackOverflowError (#1338)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-27 10:11:50 +08:00
huaxiangsun 5d5d845fea
HBASE-23853 [Flakey Test] TestBlockEvictionFromClient#testBlockRefCountAfterSplits (#1363) (#1366)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: <stack@apache.org>
2020-03-26 17:21:01 -07:00
stack 8a26a4e64f HBASE-24052 Add debug to TestMasterShutdown
Addendum 2: Refactor TestMasterShutdown
2020-03-26 15:42:56 -07:00
stack a18f5b1517 HBASE-24052 Add debug to TestMasterShutdown
Addendum
2020-03-26 12:22:22 -07:00
Nick Dimiduk c98ec0c22c HBASE-24045 Support setting `-Dhadoop.profile` in adhoc_run_tests.sh ADDENDUM
* Correctly handle the hadoop2 profile.
* Create the output directory if it doesn't exist
2020-03-26 08:48:03 -07:00
Duo Zhang 30eba2c24e HBASE-24000 Simplify CommonFSUtils after upgrading to hadoop 2.10.0 (#1335)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-26 18:10:03 +08:00
Peter Somogyi 05023846f9 HBASE-22555 Re-enable TestMasterOperationsForRegionReplicas (#1345)
Signed-off-by: stack <stack@apache.org>
2020-03-26 11:02:16 +01:00
stack b71ef1e94a HBASE-24052 Add debug to TestMasterShutdown 2020-03-25 22:42:19 -07:00
stack dcd9a81528 HBASE-24047 [Flakey Tests] Disable TestCustomSaslAuthenticationProvider#testNegativeAuthentication 2020-03-25 15:44:19 -07:00
Esteban Gutierrez a867464f5a HBASE-24041 [regression] Increase RESTServer buffer size back to 64k (#1339)
* HBASE-24041 [regression] Increase RESTServer buffer size back to 64k

* HBASE-24041 [regression] Increase RESTServer buffer size back to 64k
2020-03-25 17:00:40 -05:00
Nick Dimiduk a8606c36c7 HBASE-24045 Support setting `-Dhadoop.profile` in adhoc_run_tests.sh
Signed-off-by: stack <stack@apache.org>
2020-03-25 13:48:15 -07:00
niuyulin 244b308a3e
HBASE-23949 refactor loadBalancer implements for rsgroup balance by table to achieve overallbalanced (#1324)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-25 11:27:32 +08:00
Guanghao Zhang 41baf711ec HBASE-24037 Add ut for root dir and wal root dir are different (#1336)
Signed-off-by: stack <stack@apache.org>
2020-03-25 10:55:58 +08:00
Wei-Chiu Chuang 8521207be4 HBASE-8868. add metric to report client shortcircuit reads. (#1334)
Signed-off-by: stack <stack@apache.net>
2020-03-24 15:31:34 -07:00
stack d7189127fb HBASE-24043 [Flakey Tests] TestAsyncRegionAdminApi, TestRegionMergeTransactionOnCluster fixes and debug
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java
 Edit of log about archiving that shows in middle of a table create;
 try to make it less disorientating.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java
 Loosen assert. Compaction may have produced a single file only. Allow
 for this.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableGetMultiThreaded.java
 Make this test less furious given it is inline w/ a bunch of unit
 tests.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java
 Add debug

hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaObserverChoreRegionReports.java
 Add wait on quota table to show up before moving forward; otherwise,
 attempt at quota setting fails.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
 Debug

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java
 Remove asserts that expected regions to still have a presence in fs
 after merge when a catalogjanitor may have cleaned up parent dirs.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
 Catch exception on way out and log it rather than let it fail test.

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController.java
 Wait on acl table before proceeding.
2020-03-24 14:47:28 -07:00
Nick Dimiduk aba0737acb HBASE-24002 shadedjars check does not propagate --hadoop-profile
This implementation is almost surely incorrect. Personality
initialization parses the `--hadoop-profile` argument and sets
`HADOOP_PROFILE`. That value is then used to build an `extras` value
that is passed  along to module initialization. I'm guessing that the
`extras` value need to be honored down in the shadedjars module. I'm
not clear on how to make that work (need to study the interfaces at
play here), so taking the more ham-handed approach of referring to
`HADOOP_PROFILE`. I'm not sure if this will even work, or if it will
only work because the `foo_yetus.sh` scripts happen to use a variable
of the same name.

Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: stack <stack@apache.org>
2020-03-24 09:35:31 -07:00
Nick Dimiduk 7d89a9870b HBASE-23980 Use enforcer plugin to print JVM info in maven output
Does what it says on the tin. Bound to `initialize` phase so that it
runs early in lifecycle. Uses `<inherited>false</inherited>` so that
the plugin will run only for the base pom's reactor stage and not for
any children.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-24 09:19:04 -07:00
Reid Chan 304edcf1af HBASE-24032 [RSGroup] Assign created tables to respective rsgroup automatically instead of manual operations 2020-03-24 20:12:00 +08:00
Reid Chan 1196e42362 Revert "[RSGroup] Forward-port HBASE-22658 to master branch and branch-2.x (#1326)"
Reason: Invalid, branch-2 and master is different in RSGroup module.

This reverts commit e869a20123.
2020-03-24 14:41:33 +08:00
Reid Chan e869a20123 [RSGroup] Forward-port HBASE-22658 to master branch and branch-2.x (#1326)
Signed-off-by: stack <stack@apache.org>
2020-03-24 13:17:25 +08:00
stack 50161f2de4 HBASE-24034 [Flakey Tests] A couple of fixes and cleanups
hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupMajorCompactionTTL.java
 Remove spurious assert. Just before this it waits an arbitrary 10
 seconds. Compactions could have completed inside this time. The spirit
 of the test remains.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
 Get log cleaner to go down promptly; its sticking around. See if this
 helps with TestMasterShutdown

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
 We get a rare NPE trying to sync. Make local copy of SyncFuture and see
 if that helps.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java
 Compaction  may have completed when not expected; allow for it.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestBlockEvictionFromClient.java
 Add wait before testing. Compaction may not have completed. Let
 compaction complete before progressing and then test for empty cache.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
 Less resources.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java
 Less resources.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
 Wait till online before we try and do compaction (else request is
 ignored)

hbase-server/src/test/java/org/apache/hadoop/hbase/tool/TestCanaryTool.java
 Disable test that fails randomly w/ mockito complaint on some mac os
 x's.

TestMasterShutdown... fix NPE in RSRpcDispatcher... catch it and covert
to false and have master check for successful startup.
2020-03-23 16:21:19 -07:00
stack 1147c23627 HBASE-24035 [Flakey Tests] Disable TestClusterScopeQuotaThrottle#testUserNamespaceClusterScopeQuota 2020-03-23 13:44:00 -07:00
Huaxiang Sun ccc955a4d0
HBASE-23957 [flakey test] client.TestMultiParallel fails to read hbase-site.xml (#1310) (#1327)
Signed-off-by: Nick Dimiduk ndimiduk@apache.org
Signed-off-by: stack <stack@apache.org>
2020-03-23 12:55:59 -07:00
Viraj Jasani 94c06bd8f8
HBASE-23936 : Thrift support for Online SlowLog APIs (#1317)
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
2020-03-23 23:40:45 +05:30
Guanghao Zhang f16cf1dd8d HBASE-23741 Data loss when WAL split to HFile enabled (#1254)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-23 14:42:08 +08:00
Guanghao Zhang 1cede85a53 HBASE-24033 Add ut for loading the corrupt recovered hfiles (#1322)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-22 22:58:35 +08:00
Pankaj 3e4444f4dd HBASE-23633 Find a way to handle the corrupt recovered hfiles (#1233)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-22 16:48:01 +08:00
Toshihiro Suzuki 5104aa80fa HBASE-24030 Add necessary validations to HRegion.checkAndMutate() and HRegion.checkAndRowMutate() (#1315)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-22 11:58:42 +09:00
Mohammad Arshad e6adb79741
HBASE-24019: Correct exception messages for table null and namespace unavailable. (#1312)
Signed-off-by: Pankaj <pankajkumar@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-21 20:34:55 +05:30
Michael Stack 392bce03f6
HBASE-23984 [Flakey Tests] TestMasterAbortAndRSGotKilled fails in teardown (#1311)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 Change parameter name and add javadoc to make it more clear what the
 param actually is.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
 Move postOpenDeployTasks so if it fails to talk to the Master -- which
 can happen on cluster shutdown -- then we will do cleanup of state;
 without this the RS can get stuck and won't go down.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/CloseRegionHandler.java
 Add handleException so CRH looks more like UnassignRegionHandler and
 AssignRegionHandler around exception handling. Add a bit of doc on
 why CRH.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
 Right shift most of the body of process so can add in a finally
 that cleans up rs.getRegionsInTransitionInRS is on exception
 (otherwise outstanding entries can stop a RS going down on cluster
 shutdown)

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-20 15:25:06 -07:00
Viraj Jasani 8320f73c8c
HBASE-23977 : Resolve flakes present in TestSlowLogRecorder (ADDENDUM) 2020-03-20 13:28:28 +05:30
Duo Zhang 83388609d0
HBASE-24009 Backport the personality changes in HBASE-23989 to all active branches (#1304)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-20 14:18:07 +08:00
Nick Dimiduk ffb2359146
HBASE-24013 Bump branch-2 version to 2.4.0-SNAPSHOT (#1309)
Increment version in poms with

```
$ mvn org.codehaus.mojo:versions-maven-plugin:2.7:set -DnewVersion=2.4.0-SNAPSHOT -DgenerateBackupPoms=false
```

Verified no dangling references with

```
$ find . -iname '*pom.xml' -exec grep -n '2.3.0-SNAPSHOT' {} +
```

Verified build with

```
$ JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home mvn clean package -DskipTests
$ JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home mvn clean package -DskipTests -Dhadoop.profile=3.0
```

Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-19 08:01:43 -07:00
Viraj Jasani 481338cc4b
HBASE-23977 : Resolve flakes present in TestSlowLogRecorder (#1286)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-19 15:43:30 +05:30
Huaxiang Sun 5e764f57a8 HBASE-23999 [flakey test] TestTableOutputFormatConnectionExhaust (#1302)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-18 21:53:19 -07:00
stack c8d6ee31b8 HBASE-24017 Turn down flakey rerun rate on all but hot branches
On master branch run twice a day instead of every 4 hours.
2020-03-18 15:54:10 -07:00
stack df77c029ae HBASE-24016 Change nightly poll from cron @daily to pollSCM @daily; i.e. run nightly if a change ONLY 2020-03-18 14:42:48 -07:00
Jan Hentschel 58a0ef0cfe
HBASE-23943 Updated Rubocop configuration to current version
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-18 11:48:36 +01:00
Michael Stack ebd37a314c
HBASE-23993 Use loopback for zk standalone server in minizkcluster (#1291)
hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Have client and server use loopback instead of 'localhost'

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-17 20:14:24 -07:00
wen_yi 292cadf847 HBASE-23971 protoc warns: "no protobuf syntax specified" 2020-03-17 15:38:50 -07:00
Nick Dimiduk b942527b60 HBASE-24004 Include hadoop version in Nightly report name 2020-03-17 14:52:13 -07:00
Nick Dimiduk 33cb9fc932 HBASE-23829 Get `-PrunSmallTests` passing on JDK11
Signed-off-by: stack <stack@apache.org>
2020-03-17 09:01:02 -07:00
BukrosSzabolcs f951913e24
HBASE-23891: Add an option to Actions to filter out meta RS
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-03-17 15:02:33 +01:00