Commit Graph

16871 Commits

Author SHA1 Message Date
Guanghao Zhang 1cede85a53 HBASE-24033 Add ut for loading the corrupt recovered hfiles (#1322)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-22 22:58:35 +08:00
Pankaj 3e4444f4dd HBASE-23633 Find a way to handle the corrupt recovered hfiles (#1233)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-22 16:48:01 +08:00
Toshihiro Suzuki 5104aa80fa HBASE-24030 Add necessary validations to HRegion.checkAndMutate() and HRegion.checkAndRowMutate() (#1315)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-22 11:58:42 +09:00
Mohammad Arshad e6adb79741
HBASE-24019: Correct exception messages for table null and namespace unavailable. (#1312)
Signed-off-by: Pankaj <pankajkumar@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-21 20:34:55 +05:30
Michael Stack 392bce03f6
HBASE-23984 [Flakey Tests] TestMasterAbortAndRSGotKilled fails in teardown (#1311)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 Change parameter name and add javadoc to make it more clear what the
 param actually is.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
 Move postOpenDeployTasks so if it fails to talk to the Master -- which
 can happen on cluster shutdown -- then we will do cleanup of state;
 without this the RS can get stuck and won't go down.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/CloseRegionHandler.java
 Add handleException so CRH looks more like UnassignRegionHandler and
 AssignRegionHandler around exception handling. Add a bit of doc on
 why CRH.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
 Right shift most of the body of process so can add in a finally
 that cleans up rs.getRegionsInTransitionInRS is on exception
 (otherwise outstanding entries can stop a RS going down on cluster
 shutdown)

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-20 15:25:06 -07:00
Viraj Jasani 8320f73c8c
HBASE-23977 : Resolve flakes present in TestSlowLogRecorder (ADDENDUM) 2020-03-20 13:28:28 +05:30
Duo Zhang 83388609d0
HBASE-24009 Backport the personality changes in HBASE-23989 to all active branches (#1304)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-20 14:18:07 +08:00
Nick Dimiduk ffb2359146
HBASE-24013 Bump branch-2 version to 2.4.0-SNAPSHOT (#1309)
Increment version in poms with

```
$ mvn org.codehaus.mojo:versions-maven-plugin:2.7:set -DnewVersion=2.4.0-SNAPSHOT -DgenerateBackupPoms=false
```

Verified no dangling references with

```
$ find . -iname '*pom.xml' -exec grep -n '2.3.0-SNAPSHOT' {} +
```

Verified build with

```
$ JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home mvn clean package -DskipTests
$ JAVA_HOME=/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home mvn clean package -DskipTests -Dhadoop.profile=3.0
```

Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-19 08:01:43 -07:00
Viraj Jasani 481338cc4b
HBASE-23977 : Resolve flakes present in TestSlowLogRecorder (#1286)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-19 15:43:30 +05:30
Huaxiang Sun 5e764f57a8 HBASE-23999 [flakey test] TestTableOutputFormatConnectionExhaust (#1302)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-18 21:53:19 -07:00
stack c8d6ee31b8 HBASE-24017 Turn down flakey rerun rate on all but hot branches
On master branch run twice a day instead of every 4 hours.
2020-03-18 15:54:10 -07:00
stack df77c029ae HBASE-24016 Change nightly poll from cron @daily to pollSCM @daily; i.e. run nightly if a change ONLY 2020-03-18 14:42:48 -07:00
Jan Hentschel 58a0ef0cfe
HBASE-23943 Updated Rubocop configuration to current version
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-18 11:48:36 +01:00
Michael Stack ebd37a314c
HBASE-23993 Use loopback for zk standalone server in minizkcluster (#1291)
hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Have client and server use loopback instead of 'localhost'

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-17 20:14:24 -07:00
wen_yi 292cadf847 HBASE-23971 protoc warns: "no protobuf syntax specified" 2020-03-17 15:38:50 -07:00
Nick Dimiduk b942527b60 HBASE-24004 Include hadoop version in Nightly report name 2020-03-17 14:52:13 -07:00
Nick Dimiduk 33cb9fc932 HBASE-23829 Get `-PrunSmallTests` passing on JDK11
Signed-off-by: stack <stack@apache.org>
2020-03-17 09:01:02 -07:00
BukrosSzabolcs f951913e24
HBASE-23891: Add an option to Actions to filter out meta RS
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-03-17 15:02:33 +01:00
Wei-Chiu Chuang 7b2fe82be3 HBASE-22103. HDFS-13209 in Hadoop 3.3.0 breaks asyncwal. (#1284)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-17 14:37:40 +08:00
Duo Zhang 3366d64046 HBASE-23986 Bump hadoop-two.version to 2.10.0 on master and branch-2 (#1289)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-17 08:06:50 +08:00
stack bb8e13fc6e HBASE-23992 Fix TestAdminShell and TestQuotasShell mistakenly broken by parent commit 2020-03-14 15:17:20 -07:00
stack 671772b229 HBASE-23991 [Flakey Tests] Disable TestSecureExport.testVisibilityLabels 2020-03-14 14:04:46 -07:00
Duo Zhang 26ab594d09 HBASE-23987 NettyRpcClientConfigHelper will not share event loop by default which is incorrect (#1288)
Signed-off-by: stack <stack@apache.org>
2020-03-14 14:06:45 +08:00
xuqinya1 6de2925404
HBASE-23967 Improve the accuracy of the method sizeToString (#1273)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-13 19:43:49 +05:30
Wei-Chiu Chuang ba6d8c8f27
HBASE-23861. Reconcile Hadoop version. (#1179) (#1242)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
(cherry picked from commit 6d9802fc2e)
2020-03-13 15:16:35 +05:30
Duo Zhang d6edb56e5f HBASE-23077 Addendum typo fix fingbugs -> findbugs 2020-03-13 08:37:01 +08:00
Nick Dimiduk dab736428e HBASE-23978 Dockerfiles reusing stale apt sources info
Signed-off-by: stack <stack@apache.org>
2020-03-12 17:28:41 -07:00
stack 93ed6d6dce HBASE-23979 Disable TestSlowLogRecorder until parent addressed 2020-03-12 17:22:55 -07:00
Duo Zhang d90ff09fe0 HBASE-23077 Addendum fix compile error on hbase-rsgroup module when building with hadoop 3.x
Missed this one as on master we do not have hbase-rsgroup module any more
2020-03-12 17:54:11 +08:00
stack 09c37323e7 HBASE-23974 [Flakey Tests] Allow that server may not yet be cleared from DeadServers in TestHBCKSCP 2020-03-11 23:02:15 -07:00
stack 933cf33034 HBASE-23956 Use less resources running tests (#1266)
Addendum: TestNamespaceCommands was using up the limit on
nioeventloopgroup.... so create them per run of client.
2020-03-11 22:47:39 -07:00
stack 6a30b4eab4 HBASE-23956 Use less resources running tests (#1266)
Addendum: TestMultiParallel thread counts were off after
parent change.
2020-03-11 22:08:31 -07:00
stack 4214a4f0b9 HBASE-23956 Use less resources running tests (#1266)
Addendum: Two more tests that had issues because parent cut
down the number of default RegionServers launched.
2020-03-11 21:01:33 -07:00
Duo Zhang 7eeb6a0815 HBASE-23077 move entirely to spotbugs (#1265)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-12 11:42:23 +08:00
stack 789a8b7c44 HBASE-23956 Use less resources running tests (#1266)
Addendum: This test was depending on hard-coded number for
log cleaners; the parent patch made them less in test context.
2020-03-11 20:33:00 -07:00
stack 08603f02ea HBASE-23930 Shell should attempt to format `timestamp` attributes as ISO-8601
Make display of Cell timestamp be ISO8601 format instead of pure milliseconds.
2020-03-11 17:14:11 -07:00
stack 0152272435 HBASE-23929 Shell formatter for for meta table should pretty-print values of info:merge columns 2020-03-11 16:40:07 -07:00
stack 4f92f0f807 HBASE-23956 Use less resources running tests (#1266)
Addendum to cleanup pom warning.
2020-03-11 14:36:34 -07:00
Nick Dimiduk 678b142da2 HBASE-23932 Minor improvements to Region Normalizer (#1258)
- consolidate checks made by master on behalf of balancer and
   normalizer: deciding if the master is in a healthy state for
   running any actions at all (skipRegionManagementAction). Normalizer
   now does as balancer did previously.
 - both balancer and normalizer make one final check on above
   conditions between calculating an action plan and executing the
   plan. should make the process more responsive to shutdown
   requests.
 - change normalizer to only consider acting on a region when it is in
   the OPEN state. previously we would normalizer attempt to merge a
   region that was already in a MERGING_NEW,MERGING,MERGED state.
 - fix some typos in variable names.

Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: binlijin <binlijin@gmail.com>
2020-03-11 12:38:52 -07:00
Michael Stack 2655f9647e
HBASE-23956 Use less resources running tests (#1266)
Add being able to configure netty thread counts. Enable socket reuse
(should not have any impact).

hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java
 Rename the threads we create in here so they are NOT named same was
 threads created by Hadoop RPC.

hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java
 Allow configuring eventloopgroup thread count (so can override for
 tests)

hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java
 Enable socket resuse.

hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java
 Enable socket resuse and config for how many threads to use.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java
 Thread name edit; drop the redundant 'Thread' suffix.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java
 Make closeable and shutdown executor when called.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
 Call close on HFileReplicator

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
 HDFS creates lots of threads. Use less of it so less threads overall.

hbase-server/src/test/resources/hbase-site.xml
hbase-server/src/test/resources/hdfs-site.xml
 Constrain resources when running in test context.

hbase-server/src/test/resources/log4j.properties
 Enable debug on netty to see netty configs in our log

pom.xml
 Add system properties when we launch JVMs to constrain thread counts in
 tests

 Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-11 10:25:11 -07:00
stack ee0ea33ad0 HBASE-23963 Split TestFromClientSide; it takes too long to complete timing out
Split TestGetFromClientSide. Means have to also split the superclass
TestFromClientSideWithCoprocessor.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-10 21:25:56 -07:00
niuyulin 6900a6bfde HBASE-23953 SimpleBalancer bug when second pass to fill up to min (#1262)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-11 10:30:51 +08:00
Aman Poonia c9c22fedaa HBASE-22285 A normalizer which merges small size regions with adjacent regions (#978)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-03-10 15:47:18 -07:00
Nick Dimiduk ebfc1a206c HBASE-23945 Dockerfiles showing hadolint check failures
Signed-off-by: stack <stack@apache.org>
2020-03-10 12:33:18 -07:00
Nick Dimiduk 4f76e24755 Revert "HBASE-23891: Add an option to Actions to filter out meta RS (#1217)"
This reverts commit 7d8fa5c818.
2020-03-10 11:48:12 -07:00
niuyulin 30637f2e5f HBASE-23954 SplitParent region should not be balanced (#1264)
Co-authored-by: niuyulin <niuyulin@xiaomi.com>

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-03-10 11:15:26 -07:00
Yuki Tawara 26617df373 HBASE-20289 Fix comparator for NormalizationPlan
Signed-off-by: tedyu <yuzhihong@gmail.com>
2020-03-10 10:59:06 -07:00
Michael Stack a6700bc6b5
HBASE-23851 Log networks and bind addresses when multicast publisher/listener enabled (#1173)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-10 10:55:44 -07:00
Dima Spivak dacceba0c7 HBASE-18418 Remove apache_hbase_topology from dev-support 2020-03-09 09:53:37 -07:00
Nick Dimiduk 84efaed120 HBASE-23946 [JDK11] Yetus should vote -0 for known JDK11 issues (#1246)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-03-09 09:13:06 -07:00