Commit Graph

17948 Commits

Author SHA1 Message Date
Reid Chan c3a704ea22
HBASE-24032 [RSGroup] Assign created tables to respective rsgroup automatically instead of manual operations (#1319)
Signed-off-by: binlijin <binlijin@gmail.com>
2020-03-24 13:33:00 +08:00
Reid Chan bdcfd6a4a8
[RSGroup] Forward-port HBASE-22658 to master branch and branch-2.x (#1326)
Signed-off-by: stack <stack@apache.org>
2020-03-24 10:34:12 +08:00
stack afc1746757 HBASE-24034 [Flakey Tests] A couple of fixes and cleanups
hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupMajorCompactionTTL.java
 Remove spurious assert. Just before this it waits an arbitrary 10
 seconds. Compactions could have completed inside this time. The spirit
 of the test remains.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
 Get log cleaner to go down promptly; its sticking around. See if this
 helps with TestMasterShutdown

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
 We get a rare NPE trying to sync. Make local copy of SyncFuture and see
 if that helps.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java
 Compaction  may have completed when not expected; allow for it.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestBlockEvictionFromClient.java
 Add wait before testing. Compaction may not have completed. Let
 compaction complete before progressing and then test for empty cache.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
 Less resources.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java
 Less resources.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
 Wait till online before we try and do compaction (else request is
 ignored)

hbase-server/src/test/java/org/apache/hadoop/hbase/tool/TestCanaryTool.java
 Disable test that fails randomly w/ mockito complaint on some mac os
 x's.

TestMasterShutdown... fix NPE in RSRpcDispatcher... catch it and covert
to false and have master check for successful startup.
2020-03-23 16:26:32 -07:00
Nick Dimiduk af4fdcc556 HBASE-23980 Use enforcer plugin to print JVM info in maven output
Does what it says on the tin. Bound to `initialize` phase so that it
runs early in lifecycle. Uses `<inherited>false</inherited>` so that
the plugin will run only for the base pom's reactor stage and not for
any children.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-23 14:27:55 -07:00
stack 2f26d8a623 HBASE-24035 [Flakey Tests] Disable TestClusterScopeQuotaThrottle#testUserNamespaceClusterScopeQuota 2020-03-23 13:44:34 -07:00
Viraj Jasani 7081ccd53c
HBASE-23936 : Thrift support for Online SlowLog APIs (#1317)
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
2020-03-23 22:59:04 +05:30
Guanghao Zhang 8130858199
HBASE-23741 Data loss when WAL split to HFile enabled (#1254)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-23 14:38:47 +08:00
Duo Zhang eface74407
HBASE-23799 Make our core coprocessors use shaded protobuf (#1280)
Signed-off-by: stack <stack@apache.org>
2020-03-23 08:47:02 +08:00
Huaxiang Sun 542244056c
HBASE-23957 [flakey test] client.TestMultiParallel fails to read hbase-site.xml (#1310)
Signed-off-by: Nick Dimiduk ndimiduk@apache.org
Signed-off-by: stack <stack@apache.org>
2020-03-22 17:43:04 -07:00
Guanghao Zhang 2a60a61a73
HBASE-24033 Add ut for loading the corrupt recovered hfiles (#1322)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-22 22:56:30 +08:00
niuyulin d4ab645531
HBASE-23949 refactor loadBalancer implements for rsgroup balance by table to achieve overallbalanced (#1279)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-22 21:54:57 +08:00
Pankaj ca533a77cb
HBASE-23633 Find a way to handle the corrupt recovered hfiles (#1233)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-22 16:39:56 +08:00
Toshihiro Suzuki 9b4e75a7f6
HBASE-24030 Add necessary validations to HRegion.checkAndMutate() and HRegion.checkAndRowMutate() (#1315)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-22 11:55:52 +09:00
Toshihiro Suzuki b53fa3c51f
HBASE-24031 TestHRegion.testCheckAndMutate_WithFilters is flaky (#1316)
Signed-off-by: Viraj Jasani <virajjasani007@gmail.com>
2020-03-22 05:04:16 +09:00
Michael Stack 5592d0c191 HBASE-23984 [Flakey Tests] TestMasterAbortAndRSGotKilled fails in teardown (#1311)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 Change parameter name and add javadoc to make it more clear what the
 param actually is.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
 Move postOpenDeployTasks so if it fails to talk to the Master -- which
 can happen on cluster shutdown -- then we will do cleanup of state;
 without this the RS can get stuck and won't go down.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/CloseRegionHandler.java
 Add handleException so CRH looks more like UnassignRegionHandler and
 AssignRegionHandler around exception handling. Add a bit of doc on
 why CRH.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
 Right shift most of the body of process so can add in a finally
 that cleans up rs.getRegionsInTransitionInRS is on exception
 (otherwise outstanding entries can stop a RS going down on cluster
 shutdown)

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-20 15:26:58 -07:00
Guanghao Zhang 080d8645ed
HBASE-24023 Add 2.2.4 to download page (#1314)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-20 23:57:19 +08:00
Viraj Jasani 44b3ac6457
HBASE-23977 : Resolve flakes present in TestSlowLogRecorder (ADDENDUM) 2020-03-20 13:27:06 +05:30
Viraj Jasani 26b31e351c
HBASE-23977 : Resolve flakes present in TestSlowLogRecorder (#1286)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-19 15:35:49 +05:30
Huaxiang Sun 52a04e69c1
HBASE-23999 [flakey test] TestTableOutputFormatConnectionExhaust (#1302)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-18 21:51:04 -07:00
stack de967502ab HBASE-24017 Turn down flakey rerun rate on all but hot branches
On master branch run twice a day instead of every 4 hours.
2020-03-18 15:50:39 -07:00
stack add2d2ac57 HBASE-24016 Change nightly poll from cron @daily to pollSCM @daily; i.e. run nightly if a change ONLY 2020-03-18 14:39:53 -07:00
Jan Hentschel d7e70bb497
HBASE-23943 Updated Rubocop configuration to current version
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-18 11:09:36 +01:00
Michael Stack 3e1cf00c71 HBASE-23993 Use loopback for zk standalone server in minizkcluster (#1291)
hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Have client and server use loopback instead of 'localhost'

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-17 20:26:06 -07:00
Duo Zhang 9190a6b59a
HBASE-23989 Consider 2.10.0 in our support matrix (#1290)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-18 09:58:16 +08:00
stack 9804f73c66 HBASE-23971 protoc warns: "no protobuf syntax specified" (#1278)
Remove overcommit in TestUsersOperationsWithSecureHadoop
(Noticed by ndimiduk)
2020-03-17 15:28:26 -07:00
Nick Dimiduk 2e1485f505 HBASE-24004 Include hadoop version in Nightly report name 2020-03-17 14:49:59 -07:00
Wei-Chiu Chuang 60de4c2e31
HBASE-22103. HDFS-13209 in Hadoop 3.3.0 breaks asyncwal. (#1284)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-17 14:36:56 +08:00
Duo Zhang 795e9887b5
HBASE-23986 Bump hadoop-two.version to 2.10.0 on master and branch-2 (#1289)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-17 08:05:56 +08:00
Nick Dimiduk 059c189451 HBASE-23829 Get `-PrunSmallTests` passing on JDK11
Signed-off-by: stack <stack@apache.org>
2020-03-16 15:21:37 -07:00
stack f66cbe1a40 HBASE-23992 Fix TestAdminShell and TestQuotasShell mistakenly broken by parent commit 2020-03-14 15:18:21 -07:00
stack eefeeceedf HBASE-23991 [Flakey Tests] Disable TestSecureExport.testVisibilityLabels 2020-03-14 14:05:36 -07:00
Duo Zhang ba3f02629a
HBASE-23987 NettyRpcClientConfigHelper will not share event loop by default which is incorrect (#1288)
Signed-off-by: stack <stack@apache.org>
2020-03-14 13:47:41 +08:00
xuqinya1 b9e74e0b0a
HBASE-23967 Improve the accuracy of the method sizeToString (#1273)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-13 19:41:48 +05:30
Jan Hentschel 1b163d98b9
HBASE-23847 Removed deprecated setStartRow from Scan (#1220)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-13 08:43:17 +08:00
Duo Zhang e5b0455c38 HBASE-23077 Addendum typo fix fingbugs -> findbugs 2020-03-13 08:36:39 +08:00
stack 02b6bbe7ce HBASE-23979 Disable TestSlowLogRecorder until parent addressed 2020-03-12 17:24:21 -07:00
Nick Dimiduk c088d2edc9 HBASE-23978 Dockerfiles reusing stale apt sources info
Signed-off-by: stack <stack@apache.org>
2020-03-12 17:09:06 -07:00
WenFeiYi abf0a22259
HBASE-23971 protoc warns: "no protobuf syntax specified" (#1278)
Signed-off-by: stack <stack@apache.org>
2020-03-12 08:32:22 -07:00
stack 26f3702735 HBASE-23974 [Flakey Tests] Allow that server may not yet be cleared from DeadServers in TestHBCKSCP 2020-03-11 23:04:16 -07:00
stack d42e9768d0 HBASE-23956 Use less resources running tests (#1266)
Addendum: TestNamespaceCommands was using up the limit on
nioeventloopgroup.... so create them per run of client.
2020-03-11 22:48:22 -07:00
stack 20658c5139 HBASE-23956 Use less resources running tests (#1266)
Addendum: Two more tests that had issues because parent cut
down the number of default RegionServers launched.
2020-03-11 21:03:10 -07:00
Duo Zhang f4c576582b HBASE-23077 Addendum remove findbugs annotations dependency from shaded client 2020-03-12 11:41:32 +08:00
stack 4b64558ebc HBASE-23956 Use less resources running tests (#1266)
Addendum: This test was depending on hard-coded number for
log cleaners; the parent patch made them less in test context.
2020-03-11 20:34:29 -07:00
Duo Zhang c5aa2bc121
HBASE-23077 move entirely to spotbugs (#1265)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-12 10:18:09 +08:00
stack cf121227aa HBASE-23930 Shell should attempt to format `timestamp` attributes as ISO-8601
Make display of Cell timestamp be ISO8601 format instead of pure milliseconds.
2020-03-11 17:15:15 -07:00
Minwoo Kang 6f1e8abc30
HBASE-23561 acquire lock before update regionsMap (#1263)
Addendum: tie add and delete of regionMap and encodedRegionsMap

Signed-off-by: stack <stack@apache.org>
2020-03-11 16:57:21 -07:00
stack 0b18a7dc26 HBASE-23929 Shell formatter for for meta table should pretty-print values of info:merge columns 2020-03-11 16:41:30 -07:00
stack 64e67fb98b HBASE-23956 Use less resources running tests (#1266)
Addendum to cleanup pom warning.
2020-03-11 14:35:09 -07:00
Michael Stack aaf9a148a0 HBASE-23956 Use less resources running tests (#1266)
Add being able to configure netty thread counts. Enable socket reuse
(should not have any impact).

hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java
 Rename the threads we create in here so they are NOT named same was
 threads created by Hadoop RPC.

hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java
 Allow configuring eventloopgroup thread count (so can override for
 tests)

hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java
 Enable socket resuse.

hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java
 Enable socket resuse and config for how many threads to use.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java
 Thread name edit; drop the redundant 'Thread' suffix.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java
 Make closeable and shutdown executor when called.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
 Call close on HFileReplicator

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
 HDFS creates lots of threads. Use less of it so less threads overall.

hbase-server/src/test/resources/hbase-site.xml
hbase-server/src/test/resources/hdfs-site.xml
 Constrain resources when running in test context.

hbase-server/src/test/resources/log4j.properties
 Enable debug on netty to see netty configs in our log

pom.xml
 Add system properties when we launch JVMs to constrain thread counts in
 tests

 Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-11 10:37:50 -07:00
stack d019fd2eed HBASE-23963 Split TestFromClientSide; it takes too long to complete timing out
Split TestGetFromClientSide. Means have to also split the superclass
TestFromClientSideWithCoprocessor.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-10 22:13:24 -07:00