Commit Graph

7889 Commits

Author SHA1 Message Date
Michael Stack 1dd4b13e17
HBASE-24246 Miscellaneous hbck2 fixMeta bulk merge fixes: better logging around merges/overlap-fixing, 'HBCK Report' overlap listing, and configuration (#1572)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
 Fix weird brackets around each region name when logging.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetaFixer.java
  Log when we hit the max merge limit. Also up limit to 64.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
 Make logs make more sense to operator.

hbase-server/src/main/resources/hbase-webapps/master/hbck.jsp
 Make RegionName show when you mouseover so long names don't mess up
 display of holes and overlaps.

Address Mingliang Liu liuml07 feedback

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-05-04 08:21:52 -07:00
niuyulin 0a428e8b48
HBASE-24221 Support bulkLoadHFile by family (#1627)
Co-authored-by: niuyulin <niuyulin@xiaomi.com>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-05-03 13:46:53 +01:00
stack da92a12bf5 HBASE-24199 Procedure related metrics is not consumed in the JMX metric (#1616)
Addendum; fix failing test
2020-05-01 17:31:59 -07:00
xincunSong 0d2b959d20 [HBASE-24288]Allow admin user to create table and do bulkLoad (#1612)
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
Signed-off-by: binlijin <binlijin@gmail.com>
2020-05-01 11:58:06 -07:00
gkanade 1df3bee5f6 HBASE-24199 Procedure related metrics is not consumed in the JMX metric (#1616)
Signed-off-by: stack <stack@apache.org>
2020-05-01 11:49:32 -07:00
bsglz a6aea729c5
HBASE-22710 Wrong result in one case of scan that use raw and version… (#767)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-30 14:11:04 +05:30
xincunSong d59af05b0f HBASE-24282 'scanDetail' log message is missing when responseTooSlow happens on the first scan rpc call (#1604)
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-04-29 16:04:40 -07:00
stack 93f0d4ce06 HBASE-24247 Failed multi-merge because two regions not adjacent (legitimately).
Addendum: Restore old semantic where merge of non-adjacent required
'force'
2020-04-29 14:34:37 -07:00
Duo Zhang 922921ee5f HBASE-24249 Move code in FSHDFSUtils to FSUtils and mark related clas… (#1586)
Signed-off-by: stack <stack@apache.org>
2020-04-29 11:31:32 +08:00
stack 3b0812d3e1 HBASE-24215 [Flakey Tests] [ERROR] TestSecureRESTServer java.lang.NoClassDefFoundError: com/sun/jersey/core/spi/factory/AbstractRuntimeDelegate
Addendum #2: Add jersey-servlet to hadoop3 profile.
2020-04-28 20:18:31 -07:00
Michael Stack 9e697d0ea6
HBASE-24247 Failed multi-merge because two regions not adjacent (legitimately) (#1570)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionInfo.java
 Add new isOverlap method that takes list of RegionInfos checking that
 current RegionInfo is overlapped by the passed in Regions.

    Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
    Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-04-28 13:22:00 -07:00
Duo Zhang 47348f5887 Revert "HBASE-24211: Create table is slow in large cluster when AccessController is enabled. (#1546)"
This breaks TestAccessController.testAccessControllerUserPermsRegexHandling

This reverts commit 66e90dece2.
2020-04-27 22:50:02 +08:00
Duo Zhang 5af807153a HBASE-24264 Disable TestNettyIPC.testHedgedAsyncEcho (#1589) 2020-04-27 09:43:09 +08:00
bsglz ee5283ee82
HBASE-23264 Resolve a TODO of BucketAllocator: "Why we add the extra … (#801)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-26 16:20:40 +05:30
Mohammad Arshad 66e90dece2
HBASE-24211: Create table is slow in large cluster when AccessController is enabled. (#1546)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Pankaj <pankajkumar@apache.org>
2020-04-25 17:45:32 +05:30
niuyulin 90d738107f HBASE-24222 remove FSUtils.checkAccess and replace with FileSystem.access in HBCK (#1557)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-23 23:59:34 +08:00
Viraj Jasani b40381857a
HBASE-24024 : Reject multi() requests with rows higher than threshold (#1560)
Signed-off-by: Reid Chan <reidchan@apache.org>
2020-04-22 12:51:49 +05:30
Beata Sudi 74c033db83
HBASE-24139 : Balancer should avoid leaving idle region servers (#1511)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2020-04-22 10:11:16 +05:30
stack f8ce21f096 HBASE-24226 Address other hard references to '/tmp' found in Configuration 2020-04-21 17:39:30 -07:00
stack 3c1725f2f0 HBASE-24175 [Flakey Tests] TestSecureExportSnapshot FileNotFoundException Addendum #3; catch more '/tmp' references by hadoop. Fix broken build. 2020-04-21 17:39:20 -07:00
stack b262b9a013 HBASE-24175 [Flakey Tests] TestSecureExportSnapshot FileNotFoundException Addendum #3; catch more '/tmp' references by hadoop. 2020-04-21 17:39:04 -07:00
stack 8a1ffa6c43 HBASE-24220 Allow that zk NOTEMPTY multi exception is retryable by running in-series
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReaderImpl.java
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
 Cleanup checkstyle warnings. Don't depend on hbase-client
 ScannerCallable.

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
 Cut down on cluster resource usage.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/AbstractTestDLS.java
 Debug

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController3.java
 Debug

hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftHttpServer.java
 Debug

hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Debug

hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
 Allow that NONEMPTY is retryable by running in series.
2020-04-20 15:43:09 -07:00
Guangxu Cheng 2762ddc6a8 HBASE-23896 Snapshot owner cannot delete snapshot when ACL is enabled and Kerberos is not enabled (#1211)
Signed-off-by: binlijin <binlijin@gmail.com>
2020-04-20 10:26:21 +08:00
Reid Chan eed39202e1
HBASE-24174 Fix findbugs warning on ServiceAuthorizationManager for master branch (#1537) (#1542)
Signed-off-by: binlijin <binlijin@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-04-18 13:36:19 +08:00
XinSun 935c23312f HBASE-24166 Duplicate implementation for acquireLock between CreateTableProcedure and its parent class (#1486)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-18 10:31:46 +08:00
Joseph295 c775adfc06 HBASE-24182 log when the region is set to closing status (#1509)
Signed-off-by: stack <stack@apache.org>
2020-04-17 12:21:12 -07:00
Wei-Chiu Chuang 98ecd584c4
HBASE-23833. The relocated hadoop-thirdparty protobuf breaks HBase asyncwal (#1301) (#1534)
* Use Reflection to access shaded Hadoop protobuf classes.

(cherry picked from commit a321e536989083ca3620bf2c53f12c07740bf5b0)

* Update to improve the code:

1. Added license.
2. Added more comments.
3. Wrap byte array instead of copy to make a ByteString.
4. Moved all reflection instantiation to static class loading time.

* Use LiteralByteString to wrap byte array instead of copying it.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
(cherry picked from commit 72727ff9be)
2020-04-17 12:08:02 -07:00
gkanade 361e81e1f8 MetricsTable#updateFlushTime is wrong (#1539)
Authored-by: Gaurav Kanade <gkanade@microsoft.com>
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
2020-04-17 19:24:52 +05:30
Viraj Jasani a0a769d96f
HBASE-24195 : Admin.getRegionServers() should return live servers exc… (#1523)
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-16 20:16:30 +05:30
BukrosSzabolcs 8498fe7260
HBASE-24186: RegionMover ignores replicationId (#1512)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-16 19:56:52 +05:30
huaxiangsun efb5ff07ae
HBASE-24183 [flakey test] replication.TestAddToSerialReplicationPeer (#1514) (#1525)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-04-15 14:42:38 -07:00
stack 62c5bdcc08 HBASE-24175 [Flakey Tests] TestSecureExportSnapshot FileNotFoundException
Addendum: add test to check for '/tmp' references.
2020-04-15 10:55:08 -07:00
Reid Chan 4c83067e1e
HBASE-24112 [RSGroup] Support renaming rsgroup (#1520)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-04-15 19:04:40 +08:00
GeorryHuang f61aa0292d HBASE-23994: Add WebUI to Canary (#1292)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-04-14 16:09:47 -07:00
binlijin df3768cf2a HBASE-24181 Add region info when log meessages in HRegion. (#1506)
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-14 19:45:19 +08:00
Jianfei Jiang 2f4ae522c3 HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7 (#1453)
* HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7

* HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7

Co-authored-by: 姜建飞 10222269 <jiang.jianfei@zte.com.cn>

Signed-off-by: Mate Szalay-Beko <szalay.beko.mate@gmail.com>
Signed-off-by: Norbert Kalmar <nkalmar@cloudera.com>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangdo@apache.org>
2020-04-13 13:48:47 -07:00
Viraj Jasani 45622abe2f
HBASE-24140 : Move CandidateGenerator and their implementors out of StochasticLoadBalancer (#1458)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-12 19:21:06 +05:30
stack 9067dfe83c HBASE-24175 [Flakey Tests] TestSecureExportSnapshot FileNotFoundException
Overwrite yarn configs that write /tmp dir by default.
2020-04-11 21:23:17 -07:00
stack cc4aaefa52 HBASE-24167 [Flakey Tests] TestHRegionWithInMemoryFlush#testWritesWhileScanning gets stuck MVCC 2020-04-10 18:30:08 -07:00
huaxiangsun 3e4dfa9b31
HBASE-24161 [flakey test] locking.TestEntityLocks.testEntityLockTimeout (#1477) (#1480)
Signed-off-by: stack <stack@apache.org>
2020-04-10 09:42:47 -07:00
huaxiangsun 39187f87dd
HBASE-24159 [flakey test] regionserver.TestRegionMergeTransactionOnCluster (#1472) (#1474)
Delete tables after each test cases to avoid polluting states.

Signed-off-by: stack <stack@apache.org>
2020-04-09 21:09:34 -07:00
Pankaj 823cf97495 HBASE-24074 ConcurrentModificationException occured in ReplicationSourceManager while refreshing the peer (#1471)
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
Signed-off-by: stack <stack@apache.org>
2020-04-09 16:42:22 -07:00
stack 2d11796380 HBASE-24158 [Flakey Tests] TestAsyncTableGetMultiThreaded 2020-04-09 11:03:22 -07:00
Reid Chan 5012241817 HBASE-24121 [Authorization] ServiceAuthorizationManager isn't dynamically updatable. And it should be (#1439)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2020-04-09 16:30:00 +08:00
Sean Busbey 611c62f52a HBASE-24138 log more details about balancer decisions for StochasticLoadBalancer (#1455)
- at DEBUG log messages about RegionCountSkewCostFunction region/server totals
- at DEBUG log messages about the decision to balance or not with total costs
- at TRACE log messages about region count on each server RegionCountSkewCostFunction sees
- at TRACE log message with the individual cost functions used in the decision to balance or not

Signed-off-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 2d78a286b6)
2020-04-08 15:50:38 -05:00
huaxiangsun 6f00714ad3
HBASE-24120 Flakey Test: TestReplicationAdminWithClusters timeout (#1441) (#1456)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-08 12:23:50 -07:00
Andrew Purtell 3f511b0e93
HBASE-24099 Use a fair ReentrantReadWriteLock for the region close lock (#1407)
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-04-08 11:40:42 -07:00
Michael Stack ed54c35cf0
HBASE-24134 Down forked JVM heap size from 2800m to 2200m for jdk8 and jdk11 (#1451)
Down jdk8 forked jvm heap from 2800 to 2200 and the jdk11 heap from
3200 to 2200. Down the mvn size from 4G to 3.6G

Change how many puts done by TestMultiRespectsLimits because made
the test run the forked heap over 2.5G in size.

Signed-off-by: Sean Busbey <busbey@apache.org>
2020-04-08 10:51:03 -07:00
Michael Stack 9da7f95fa7
HBASE-24128 [Flakey Tests] Add retry on thrift cmdline if client fails plus misc debug (#1442)
hbase-server/src/test/java/org/apache/hadoop/hbase/TestClusterPortAssignment.java
 Saw case where Master failed startup but it came out as an IOE so we
 did not trip the retry logic.

hbase-server/src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
 Add some debug and up timeouts. This test fails frequently for me
 locally.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/locking/TestEntityLocks.java
 Up the wait from 2x 200ms to 10x in case a pause on hardware or GC.
 This test fails locally and up on jenkins.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestClearRegionBlockCache.java
 Debug. Have assert say what bad count was.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingToCellFlatMapMemStore.java
 Fails on occasion. Found count is off by a few. Tricky to debug. HBASE-24129 to reenable.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java
 Debug. Add wait and check before moving to assert.

hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftHttpServer.java
 Check for null before shutting; can be null if failed start.

hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerCmdLine.java
 Add retry if client messes up connection. Fails for me locally.
2020-04-07 09:28:05 -07:00
Duo Zhang 74a85e26ee HBASE-24055 Make AsyncFSWAL can run on EC cluster (#1437)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-04-07 23:46:06 +08:00