16791 Commits

Author SHA1 Message Date
WenFeiYi
a4c66b3fa6 HBASE-24040 WALFactory.Providers.multiwal causes StackOverflowError (#1338)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-27 10:11:58 +08:00
huaxiangsun
9f53d35e5e
HBASE-23853 [Flakey Test] TestBlockEvictionFromClient#testBlockRefCountAfterSplits (#1363) (#1367)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-03-26 17:20:29 -07:00
Nick Dimiduk
00c8ccd70f HBASE-24045 Support setting -Dhadoop.profile in adhoc_run_tests.sh ADDENDUM
* Correctly handle the hadoop2 profile.
* Create the output directory if it doesn't exist
2020-03-26 08:48:12 -07:00
Duo Zhang
35f7d050ce HBASE-24000 Simplify CommonFSUtils after upgrading to hadoop 2.10.0 (#1335)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-26 18:11:46 +08:00
Peter Somogyi
224b706c6e HBASE-22555 Re-enable TestMasterOperationsForRegionReplicas (#1345)
Signed-off-by: stack <stack@apache.org>
2020-03-26 11:06:47 +01:00
stack
0af208679a HBASE-24047 [Flakey Tests] Disable TestCustomSaslAuthenticationProvider#testNegativeAuthentication 2020-03-25 15:45:22 -07:00
Esteban Gutierrez
11bc006ff3 HBASE-24041 [regression] Increase RESTServer buffer size back to 64k (#1339)
* HBASE-24041 [regression] Increase RESTServer buffer size back to 64k

* HBASE-24041 [regression] Increase RESTServer buffer size back to 64k
2020-03-25 17:06:34 -05:00
Nick Dimiduk
3ef3e758d8 HBASE-24045 Support setting -Dhadoop.profile in adhoc_run_tests.sh
Signed-off-by: stack <stack@apache.org>
2020-03-25 13:50:33 -07:00
niuyulin
0c545cf4d4 HBASE-23949 refactor loadBalancer implements for rsgroup balance by table to achieve overallbalanced (#1324)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-25 11:45:16 +08:00
Guanghao Zhang
88ac6db032 HBASE-24037 Add ut for root dir and wal root dir are different (#1336)
Signed-off-by: stack <stack@apache.org>
2020-03-25 11:22:41 +08:00
Wei-Chiu Chuang
4ec19096df HBASE-8868. add metric to report client shortcircuit reads. (#1334)
Signed-off-by: stack <stack@apache.net>
2020-03-24 15:31:56 -07:00
stack
75a66f15d5 HBASE-24043 [Flakey Tests] TestAsyncRegionAdminApi, TestRegionMergeTransactionOnCluster fixes and debug
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java
 Edit of log about archiving that shows in middle of a table create;
 try to make it less disorientating.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java
 Loosen assert. Compaction may have produced a single file only. Allow
 for this.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableGetMultiThreaded.java
 Make this test less furious given it is inline w/ a bunch of unit
 tests.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java
 Add debug

hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaObserverChoreRegionReports.java
 Add wait on quota table to show up before moving forward; otherwise,
 attempt at quota setting fails.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
 Debug

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java
 Remove asserts that expected regions to still have a presence in fs
 after merge when a catalogjanitor may have cleaned up parent dirs.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
 Catch exception on way out and log it rather than let it fail test.

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController.java
 Wait on acl table before proceeding.
2020-03-24 14:51:37 -07:00
Nick Dimiduk
050383e96d HBASE-24002 shadedjars check does not propagate --hadoop-profile
This implementation is almost surely incorrect. Personality
initialization parses the `--hadoop-profile` argument and sets
`HADOOP_PROFILE`. That value is then used to build an `extras` value
that is passed  along to module initialization. I'm guessing that the
`extras` value need to be honored down in the shadedjars module. I'm
not clear on how to make that work (need to study the interfaces at
play here), so taking the more ham-handed approach of referring to
`HADOOP_PROFILE`. I'm not sure if this will even work, or if it will
only work because the `foo_yetus.sh` scripts happen to use a variable
of the same name.

Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: stack <stack@apache.org>
2020-03-24 09:35:40 -07:00
Nick Dimiduk
035d7cb360 HBASE-23980 Use enforcer plugin to print JVM info in maven output
Does what it says on the tin. Bound to `initialize` phase so that it
runs early in lifecycle. Uses `<inherited>false</inherited>` so that
the plugin will run only for the base pom's reactor stage and not for
any children.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-24 09:19:54 -07:00
Reid Chan
733a1c06a6 HBASE-24032 [RSGroup] Assign created tables to respective rsgroup automatically instead of manual operations 2020-03-24 20:24:46 +08:00
Reid Chan
1d988afc9d Revert "[RSGroup] Forward-port HBASE-22658 to master branch and branch-2.x (#1326)"
Reason: Invalid, branch-2 and master is different in RSGroup module.

This reverts commit 58b63e04c4af99a5730efb0a7e553be4d950e6a5
2020-03-24 14:45:01 +08:00
Reid Chan
58b63e04c4 [RSGroup] Forward-port HBASE-22658 to master branch and branch-2.x (#1326)
Signed-off-by: stack <stack@apache.org>
2020-03-24 13:19:51 +08:00
Huaxiang Sun
da85d347ca
HBASE-23957 [flakey test] client.TestMultiParallel fails to read hbase-site.xml (#1310) (#1328)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-03-23 17:44:31 -07:00
stack
bcc7f7f06d HBASE-24034 [Flakey Tests] A couple of fixes and cleanups
hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupMajorCompactionTTL.java
 Remove spurious assert. Just before this it waits an arbitrary 10
 seconds. Compactions could have completed inside this time. The spirit
 of the test remains.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
 Get log cleaner to go down promptly; its sticking around. See if this
 helps with TestMasterShutdown

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
 We get a rare NPE trying to sync. Make local copy of SyncFuture and see
 if that helps.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java
 Compaction  may have completed when not expected; allow for it.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestBlockEvictionFromClient.java
 Add wait before testing. Compaction may not have completed. Let
 compaction complete before progressing and then test for empty cache.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
 Less resources.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java
 Less resources.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
 Wait till online before we try and do compaction (else request is
 ignored)

hbase-server/src/test/java/org/apache/hadoop/hbase/tool/TestCanaryTool.java
 Disable test that fails randomly w/ mockito complaint on some mac os
 x's.

TestMasterShutdown... fix NPE in RSRpcDispatcher... catch it and covert
to false and have master check for successful startup.
2020-03-23 16:22:37 -07:00
stack
12e5b07258 HBASE-24035 [Flakey Tests] Disable TestClusterScopeQuotaThrottle#testUserNamespaceClusterScopeQuota 2020-03-23 13:44:58 -07:00
Viraj Jasani
bb879272f1
HBASE-23936 : Thrift support for Online SlowLog APIs (#1317)
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
2020-03-23 23:42:53 +05:30
Guanghao Zhang
c519576fcb HBASE-23741 Data loss when WAL split to HFile enabled (#1254)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-23 15:09:20 +08:00
Guanghao Zhang
75655bd3aa HBASE-24033 Add ut for loading the corrupt recovered hfiles (#1322)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-22 23:02:29 +08:00
Pankaj
3c33478608 HBASE-23633 Find a way to handle the corrupt recovered hfiles (#1233)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-22 16:52:43 +08:00
Toshihiro Suzuki
f5e9b3b6f1 HBASE-24030 Add necessary validations to HRegion.checkAndMutate() and HRegion.checkAndRowMutate() (#1315)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-22 12:07:55 +09:00
Mohammad Arshad
8f439786ad
HBASE-24019: Correct exception messages for table null and namespace unavailable. (#1312)
Signed-off-by: Pankaj <pankajkumar@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-21 20:35:22 +05:30
Michael Stack
16441fbd68 HBASE-23984 [Flakey Tests] TestMasterAbortAndRSGotKilled fails in teardown (#1311)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 Change parameter name and add javadoc to make it more clear what the
 param actually is.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
 Move postOpenDeployTasks so if it fails to talk to the Master -- which
 can happen on cluster shutdown -- then we will do cleanup of state;
 without this the RS can get stuck and won't go down.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/CloseRegionHandler.java
 Add handleException so CRH looks more like UnassignRegionHandler and
 AssignRegionHandler around exception handling. Add a bit of doc on
 why CRH.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
 Right shift most of the body of process so can add in a finally
 that cleans up rs.getRegionsInTransitionInRS is on exception
 (otherwise outstanding entries can stop a RS going down on cluster
 shutdown)

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-20 15:26:31 -07:00
Viraj Jasani
2b9cb32f01
HBASE-23977 : Resolve flakes present in TestSlowLogRecorder (ADDENDUM) 2020-03-20 13:29:02 +05:30
Duo Zhang
2bdd4bee20 HBASE-24009 Backport the personality changes in HBASE-23989 to all active branches (#1304)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-20 14:22:43 +08:00
Viraj Jasani
b9aa4ce673
HBASE-23977 : Resolve flakes present in TestSlowLogRecorder (#1286)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-19 15:44:04 +05:30
Huaxiang Sun
3c4cbf18ea HBASE-23999 [flakey test] TestTableOutputFormatConnectionExhaust (#1302)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-18 21:54:14 -07:00
stack
e6b8a19dd1 HBASE-24016 Change nightly poll from cron @daily to pollSCM @daily; i.e. run nightly if a change ONLY 2020-03-18 14:43:28 -07:00
Jan Hentschel
58a0ef0cfe
HBASE-23943 Updated Rubocop configuration to current version
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-18 11:48:36 +01:00
Michael Stack
ebd37a314c
HBASE-23993 Use loopback for zk standalone server in minizkcluster (#1291)
hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Have client and server use loopback instead of 'localhost'

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-17 20:14:24 -07:00
wen_yi
292cadf847 HBASE-23971 protoc warns: "no protobuf syntax specified" 2020-03-17 15:38:50 -07:00
Nick Dimiduk
b942527b60 HBASE-24004 Include hadoop version in Nightly report name 2020-03-17 14:52:13 -07:00
Nick Dimiduk
33cb9fc932 HBASE-23829 Get -PrunSmallTests passing on JDK11
Signed-off-by: stack <stack@apache.org>
2020-03-17 09:01:02 -07:00
BukrosSzabolcs
f951913e24
HBASE-23891: Add an option to Actions to filter out meta RS
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-03-17 15:02:33 +01:00
Wei-Chiu Chuang
7b2fe82be3 HBASE-22103. HDFS-13209 in Hadoop 3.3.0 breaks asyncwal. (#1284)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-17 14:37:40 +08:00
Duo Zhang
3366d64046 HBASE-23986 Bump hadoop-two.version to 2.10.0 on master and branch-2 (#1289)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-17 08:06:50 +08:00
stack
bb8e13fc6e HBASE-23992 Fix TestAdminShell and TestQuotasShell mistakenly broken by parent commit 2020-03-14 15:17:20 -07:00
stack
671772b229 HBASE-23991 [Flakey Tests] Disable TestSecureExport.testVisibilityLabels 2020-03-14 14:04:46 -07:00
Duo Zhang
26ab594d09 HBASE-23987 NettyRpcClientConfigHelper will not share event loop by default which is incorrect (#1288)
Signed-off-by: stack <stack@apache.org>
2020-03-14 14:06:45 +08:00
xuqinya1
6de2925404
HBASE-23967 Improve the accuracy of the method sizeToString (#1273)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-13 19:43:49 +05:30
Wei-Chiu Chuang
ba6d8c8f27
HBASE-23861. Reconcile Hadoop version. (#1179) (#1242)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
(cherry picked from commit 6d9802fc2ea4d0da75164cabe58d620ade5c604a)
2020-03-13 15:16:35 +05:30
Duo Zhang
d6edb56e5f HBASE-23077 Addendum typo fix fingbugs -> findbugs 2020-03-13 08:37:01 +08:00
Nick Dimiduk
dab736428e HBASE-23978 Dockerfiles reusing stale apt sources info
Signed-off-by: stack <stack@apache.org>
2020-03-12 17:28:41 -07:00
stack
93ed6d6dce HBASE-23979 Disable TestSlowLogRecorder until parent addressed 2020-03-12 17:22:55 -07:00
Duo Zhang
d90ff09fe0 HBASE-23077 Addendum fix compile error on hbase-rsgroup module when building with hadoop 3.x
Missed this one as on master we do not have hbase-rsgroup module any more
2020-03-12 17:54:11 +08:00
stack
09c37323e7 HBASE-23974 [Flakey Tests] Allow that server may not yet be cleared from DeadServers in TestHBCKSCP 2020-03-11 23:02:15 -07:00