Apache
/
hbase
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
18,069 Commits 95 Branches 866 Tags 550 MiB
Commit Graph

8630 Commits

Author SHA1 Message Date
Xiaolin Ha e8ecafb22b HBASE-26155 JVM crash when scan (#3553) 
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-08-12 17:17:03 +08:00
Rushabh Shah 6b9114d95e
HBASE-26185 Return mutable list in AssignmentManager#getExcludedServersForSystemTable (#3577) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-08-12 13:47:52 +05:30
stack 243e861985 Revert "HBASE-26122: Implement an optional maximum size for Gets, after which a partial result is returned (#3532)" 
This reverts commit 8f16e34eb2.
 2021-08-11 07:41:22 -07:00
Bryan Beaudreault 8f16e34eb2
HBASE-26122: Implement an optional maximum size for Gets, after which a partial result is returned (#3532) 
Signed-off-by: stack <stack@apache.org>
 2021-08-10 20:38:06 -07:00
Viraj Jasani 40a7a75a19
HBASE-26185 Fix TestMaster#testMoveRegionWhenNotInitialized in the presence of hbase.min.version.move.system.tables (#3573) 
Signed-off-by: Michael Stack <stack@apache.org>
 2021-08-10 10:58:31 +05:30
Richard Marscher 79a6dc6bfd HBASE-6908: Add pluggable rpc queue implementation (#3522) 
Can pass in a FQCN to load as the call queue implementation.

Standardized arguments to the constructor are the max queue length, the
PriorityFunction, and the Configuration.

`PluggableBlockingQueue` abstract class provided to help guide the
correct constructor signature

Hard fails if the class fails to load as a `BlockingQueue<CallRunner>`

Signed-off-by: stack <stack@apache.org>
 2021-08-09 12:55:27 -07:00
Duo Zhang 611ffa7da9 HBASE-26179 TestRequestTooBigException spends too much time to finish (#3571) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-08-07 23:05:53 +08:00
Xiaolin Ha 61fbe6f480 HBASE-26170 handleTooBigRequest in NettyRpcServer didn't skip enough bytes (#3564) 
Signed-off-by: stack <stack@apache.com>
 2021-08-07 23:05:27 +08:00
chenglei 4f0f519f9f HBASE-26142 NullPointerException when set 'hbase.hregion.memstore.mslab.indexchunksize.percent' to zero (#3531) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-08-05 21:04:42 +08:00
caroliney14 effde97a88 HBASE-25469 Add detailed RIT info in JSON format for consumption as metrics (#3555) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2021-08-04 18:23:17 -07:00
Andrew Purtell 8e9253edd8 Revert "Add detailed RIT info in JSON format for consumption as metrics (#3555)" 
Commit is missing JIRA identifier.

This reverts commit 31b5e09ac2.
 2021-08-04 18:23:08 -07:00
caroliney14 31b5e09ac2
Add detailed RIT info in JSON format for consumption as metrics (#3555) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2021-08-04 18:04:44 -07:00
Tak Lon (Stephen) Wu c11a3e1b39 Revert "HBASE-26124 Backport HBASE-25373 "Remove HTrace completely in code base and try to make use of OpenTelemetry" to branch-2 (#3529)" 
This reverts commit f049301606.
 2021-08-04 15:55:13 -07:00
Tak Lon (Stephen) Wu fffdcba5bb Revert "HBASE-26125 Backport HBASE-25401 "Add trace support for async call in rpc client" to branch-2 (#3543)" 
This reverts commit ca096437d7.
 2021-08-04 15:55:13 -07:00
Duo Zhang 620806e3fb HBASE-26150 Let region server also carry ClientMetaService (#3550) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2021-08-05 00:15:49 +08:00
Ruanhui 04d3c7a9d0 HBASE-26166 Fix a minor bug of table list on master ui (#3554) 
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
Signed-off-by: Rushabh Shah <shahrs87@gmail.com>
 2021-08-04 20:42:15 +08:00
Tak Lon (Stephen) Wu ca096437d7
HBASE-26125 Backport HBASE-25401 "Add trace support for async call in rpc client" to branch-2 (#3543) 
2/17 commits of HBASE-22120

Co-authored-by: Duo Zhang <zhangduo@apache.org>

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
 2021-08-02 11:33:19 -07:00
Bryan Beaudreault aa3b07f6bb
HBASE-26154: Adds exception metrics for QuotaExceededException and RpcThrottlingException (#3544) 
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
 2021-08-02 09:51:34 +05:30
Duo Zhang f7baa195e3 HBASE-26151 Reimplement MasterAddressTracker to also cache backup master addresses (#3548) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-07-31 15:54:06 +08:00
Duo Zhang aff880e147 HBASE-26144 The HStore.snapshot method is never called in main code (#3533) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-07-31 14:39:13 +08:00
YutSean f3c022740a HBASE-26105 Rectify the expired TODO comment in CombinedBC. (#3509) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-07-30 21:43:31 +08:00
liever18 6df4cb1cf4
HBASE-26114 when “hbase.mob.compaction.threads.max” is set to a negative number, HMaster cannot start normally (#3541) 
Signed-off-by: Anoop <anoopsamjohn@apache.org>
 2021-07-30 17:13:50 +05:30
Tak Lon (Stephen) Wu f049301606
HBASE-26124 Backport HBASE-25373 "Remove HTrace completely in code base and try to make use of OpenTelemetry" to branch-2 (#3529) 
1/17 commits of HBASE-22120

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
 2021-07-29 09:15:10 -07:00
Duo Zhang dd22fecc2b
HBASE-26120 New replication gets stuck or data loss when multiwal groups more than 10 (#3528) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
 2021-07-27 08:14:15 -07:00
Duo Zhang 20a4aaedcc HBASE-26118 The HStore.commitFile and HStore.moveFileIntoPlace almost have the same logic (#3525) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-07-26 21:47:12 +08:00
Wei-Chiu Chuang 37c3f7b446
HBASE-21946 Use ByteBuffer pread instead of byte[] pread in HFileBlock when applicable (#3434) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 4a3c7d73b0)
 2021-07-26 17:43:10 +08:00
Duo Zhang be0a6cea40 HBASE-26119 Polish TestAsyncNonMetaRegionLocator (#3526) 
Signed-off-by: Xin Sun <ddupgs@gmail.com>
 2021-07-26 17:34:01 +08:00
Rushabh Shah 59e2825c93 HBASE-26093 Replication is stuck due to zero length wal file in oldWALs directory (#3504) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-07-23 09:33:43 -07:00
meiyi 729da5a6d3
HBASE-24734 RegionInfo#containsRange should support check meta table (#3496) (#3517) 
Signed-off-by: zhangduo <zhangduo@apache.org>
 2021-07-22 18:04:24 +08:00
gkanade 8bc180c773
HBASE-24984 WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used with multi operation (#3512) 
Signed-off-by: Anoop <anoopsamjohn@apache.org>
Signed-off-by: zhangduo <zhangduo@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.org>
 2021-07-22 08:30:25 +05:30
Aman Poonia cabf0dc51d
HBASE-25986 set default value of normalization enabled from hbase site (#3476) (#3372) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-07-19 15:26:52 +05:30
clarax 55d93fbf2d
HBASE-25973 Balancer should explain progress in a better way in log -… (#3483) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2021-07-18 13:04:33 -07:00
YutSean fb01a6c276
HBASE-26083 In branch-2 & branch-1 L1 miss metric is always 0 when using CombinedBlockCache (#3473) 
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by Reid Chan <reidchan@apache.org>
 2021-07-16 10:28:27 +08:00
Xiaolin Ha 692d384ec6
HBASE-26036 DBB released too early in HRegion.get() and dirty data for some operations (#3436) (#3486) 
Signed-off-by: Michael Stack <stack@apache.org>
 2021-07-14 16:30:46 +08:00
binlijin 12f2a16280
HBASE-26086 TestHRegionReplayEvents do not pass in branch-2 and throws NullPointerException (#3480) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
 2021-07-14 11:37:25 +08:00
Clara Xiong dfc9ac8ffb HBASE-25739 TableSkewCostFunction need to use aggregated deviation - backport 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@duboce.net>
Reviewed-by: Nick Dimiduk <ndimiduk@apache.org>
 2021-07-13 08:37:29 -07:00
mokai 3294d32f46 HBASE-25700 Enhance znode parent validation when add_peer (#3458) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Pankaj Kumar <pankajkumar@apache.org>
 2021-07-11 21:15:57 +08:00
Duo Zhang b86d2d4020 HBASE-26069 Remove HStore.compactRecentForTestingAssumingDefaultPolic… (#3462) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-07-07 21:57:59 +08:00
Duo Zhang 1ba03cae4b HBASE-26068 The last assertion in TestHStore.testRefreshStoreFilesNotChanged is wrong (#3461) 
Signed-off-by: Yulin Niu <niuyulin@apache.org
 2021-07-07 14:39:03 +08:00
Duo Zhang bde5d98338 HBASE-26065 StripeStoreFileManager does not need to throw IOException for most methods (#3459) 
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
 2021-07-05 22:19:17 +08:00
Viraj Jasani 342a478753
HBASE-22923 Consider minVersionToMoveSysTables while moving region and creating regionPlan (ADDENDUM) (#3455) 
Signed-off-by: David Manning <david.manning@salesforce.com>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2021-07-03 22:44:46 +05:30
Wei-Chiu Chuang db75ef265e
HBASE-26050 Remove the reflection used in FSUtils.isInSafeMode (#3445) 
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 82c44b49f5)
 2021-07-02 14:21:28 +08:00
Viraj Jasani 1381df4c80
HBASE-22923 min version of RegionServer to move system table regions (#3439) (#3438) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2021-07-01 16:24:14 +05:30
Viraj Jasani 75cd25386a
HBASE-25902 Add missing CFs in meta during HBase 1 to 2 Upgrade (#3417) 
Signed-off-by: Michael Stack <stack@apache.org>
 2021-07-01 15:20:38 +05:30
bsglz b90fbb77c6 HBASE-26028 The view as json page shows exception when using TinyLfuBlockCache (#3420) 
(cherry picked from commit 147b030c1f)
 2021-07-01 10:27:24 +08:00
Duo Zhang 748e369629 HBASE-26029 It is not reliable to use nodeDeleted event to track region server's death (#3430) 
Signed-off-by: Xin Sun <ddupgs@gmail.com>
 2021-06-30 11:46:09 +08:00
Duo Zhang f827a4c5f8 HBASE-26039 TestReplicationKillRS is useless after HBASE-23956 (#3440) 
Signed-off-by: Michael Stack <stack@apache.org>
 2021-06-30 08:27:17 +08:00
GeorryHuang 166becfd66
HBASE-25980 Master table.jsp pointed at meta throws 500 when no all r… (#3373) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-06-27 22:30:46 +08:00
GeorryHuang c707a9ffff
HBASE-25914 Provide slow/large logs on RegionServer UI (#3321) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-06-27 22:27:06 +08:00
GeorryHuang c7c5d56084 HBASE-26015 Should implement getRegionServers(boolean) method in Asyn… (#3406) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-06-27 22:13:21 +08:00
YutSean 193a94e523
HBASE-26013 Get operations readRows metrics becomes zero after HBASE-25677 (#3411) 
Signed-off-by: Reid Chan <reidchan@apache.org>
 2021-06-24 15:24:50 +08:00
Duo Zhang 9fe1781037 HBASE-26020 Split TestWALEntryStream.testDifferentCounts out (#3409) 
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
 2021-06-23 22:51:23 +08:00
litao c81a5ab565
HBASE-25934 Add username for RegionScannerHolder (#3325) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-06-23 12:54:41 +05:30
lujiefsi 2ae69a883c HBASE-25877 Add access check for compactionSwitch (#3253) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-06-22 23:45:36 +08:00
belugabehr bda7afe97d HBASE-25937: Clarify UnknownRegionException (#3330) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-06-22 22:44:05 +08:00
Duo Zhang 12d707c880 HBASE-25992 Addendum add missing catch WALEntryFilterRetryableException back 2021-06-21 23:58:42 +08:00
Viraj Jasani 41a68325c2
HBASE-25698 Fixing IllegalReferenceCountException when using TinyLfuBlockCache (#3215) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
 2021-06-21 12:51:59 +05:30
YutSean 029dc81b39
HBASE-26001 When turn on access control, the cell level TTL of Increment and Append operations is invalid (#3403) 
Signed-off-by: Reid Chan <reidchan@apache.org>
 2021-06-21 15:00:50 +08:00
Duo Zhang c800e6c825 HBASE-25992 Polish the ReplicationSourceWALReader code for 2.x after HBASE-25596 (#3376) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-06-20 16:38:13 +08:00
Bharath Vissapragada 3b58919246
HBASE-25998: Redo synchronization in SyncFuture (#3399) 
Currently uses coarse grained synchronized approach that seems to
create a lot of contention. This patch

- Uses a reentrant lock instead of synchronized monitor
- Switches to a condition variable based waiting rather than busy wait
- Removed synchronization for unnecessary fields

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 6bafb59642)
 2021-06-18 10:37:10 -07:00
Bharath Vissapragada 836fd3c52a
HBASE-25984: Avoid premature reuse of sync futures in FSHLog (#3371) (#3392) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 5a19bcfa98)
 2021-06-17 12:27:23 -07:00
Duo Zhang 26c4e86ebd HBASE-25976 Implement a master based ReplicationTracker (#3390) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2021-06-17 18:38:43 +08:00
binlijin 4719a2078e HBASE-25997 NettyRpcFrameDecoder decode request header wrong when han… (#3380) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-06-15 14:39:47 +08:00
Toshihiro Suzuki 6f1e652091
HBASE-26002 MultiRowMutationEndpoint should return the result of the conditional update (#3386) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-06-15 13:55:21 +09:00
Duo Zhang cb5f0dfe1e HBASE-25995 Change the method name for DoubleArrayCost.setCosts (#3381) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-06-15 11:55:54 +08:00
Andrew Purtell a1d59bf1a5 HBASE-25994 Active WAL tailing fails when WAL value compression is enabled (#3377) 
Depending on which compression codec is used, a short read of the
compressed bytes can cause catastrophic errors that confuse the WAL reader.
This problem can manifest when the reader is actively tailing the WAL for
replication. To avoid these issues when WAL value compression is enabled,
BoundedDelegatingInputStream should assume enough bytes are available to
supply a reader up to its bound. This behavior is valid per the contract
of available(), which provides an _estimate_ of available bytes, and
equivalent to IOUtils.readFully but without requiring an intermediate
buffer.

Added TestReplicationCompressedWAL and TestReplicationValueCompressedWAL.
Without the WALCellCodec change TestReplicationValueCompressedWAL will
fail.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2021-06-14 17:18:53 -07:00
bsglz 88f031eeec HBASE-25967 The readRequestsCount does not calculate when the outResu… (#3351) 
* HBASE-25967 The readRequestsCount does not calculate when the outResults is empty

Co-authored-by: Zheng Wang <wangzheng@apache.org>
(cherry picked from commit 329f0baa98)
 2021-06-10 10:24:18 +08:00
Xiaolin Ha b6837295f3 HBASE-25981 JVM crash when displaying RegionServer UI (#3364) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-06-09 18:16:33 +08:00
Duo Zhang e4dd4901a8 HBASE-25963 HBaseCluster should be marked as IA.Public (#3348) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-06-06 21:27:02 +08:00
meiyi 1267bdb1a4 HBASE-25929 RegionServer JVM crash when compaction (#3318) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-06-03 17:20:08 +08:00
Andrew Purtell a4e8ee183e HBASE-25911 Replace calls to System.currentTimeMillis with EnvironmentEdgeManager.currentTime (#3302) 
We introduced EnvironmentEdgeManager as a way to inject alternate clocks
for unit tests. In order for this to be effective, all callers that would
otherwise use System.currentTimeMillis() must call
EnvironmentEdgeManager.currentTime() instead, except the implementers of
EnvironmentEdge.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>

Conflicts:
	hbase-backup/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupManager.java
	hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupBase.java
	hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupManager.java
	hbase-backup/src/test/java/org/apache/hadoop/hbase/backup/TestBackupSystemTable.java
	hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncConnectionTracing.java
	hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionLocatorTracing.java
	hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestBackupRestore.java
	hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestManyRegions.java
	hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/MoveRegionsOfTableAction.java
	hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALRecordReader.java
	hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/replication/TestVerifyReplicationCrossDiffHdfs.java
	hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotV1NoCluster.java
	hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestEnableRSGroups.java
	hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupsAdmin2.java
	hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/CallRunner.java
	hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcServer.java
	hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
	hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
	hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
	hbase-server/src/main/java/org/apache/hadoop/hbase/master/balancer/SimpleLoadBalancer.java
	hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractWALRoller.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/TestMetaTableAccessor.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/TestMetaUpdatesGoToPriorityQueue.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/TestSerialization.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestHFileArchiving.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/client/RestoreSnapshotFromClientSimpleTestBase.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin2.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestConnection.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMobCloneSnapshotFromClientCloneLinksAfterDelete.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMutationGetCellBuilder.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSnapshotMetadata.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestActiveMasterManager.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterMetricsWrapper.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMetaAssignmentWithStopMaster.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestRegionStateStore.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestLogsCleaner.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/master/cleaner/TestReplicationHFileCleaner.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/master/janitor/TestCatalogJanitor.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/MasterProcedureSchedulerPerformanceEvaluation.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestCloneSnapshotProcedure.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestRestoreSnapshotProcedure.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/procedure2/store/region/RegionProcedureStorePerformanceEvaluation.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHStoreFile.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMajorCompaction.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionOpen.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSimpleTimeRangeTracker.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestWideScanner.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestCloseChecker.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/AbstractTestProtobufLog.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/AbstractTestWALReplay.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestCombinedAsyncWriter.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/replication/master/TestRecoverStandbyProcedure.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestSyncReplicationWALProvider.java
	hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServer.java
	hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 2021-06-01 12:41:15 -07:00
Bharath Vissapragada d4285be5c1
HBASE-25932: Ensure replication reads the trailer bytes from WAL. (#3332) (#3345) 
This bug was exposed by the test from HBASE-25924. Since this wal
implementations close the wal asynchronously, replication can potentially
miss the trailer bytes. (see jira comment for detailed analysis).

While this is not a correctness problem (since trailer does not have any entry data),
it erroneously bumps a metric that is used to track skipped bytes in WAL resulting
in false alarms which is something we should avoid.

Reviewed-by: Rushabh Shah <rushabh.shah@salesforce.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
(cherry picked from commit b04c3c7786)
 2021-06-01 08:04:12 -07:00
Michael Stack 1a45b743a0 HBASE-25940 Update Compression/TestCompressionTest: LZ4, SNAPPY, LZO (#3334) 
Undo asserts that LZ4 and SNAPPY fails if their native libs are NOT
loaded; as of hadoop 3.3.1, LZ4 and SNAPPY can work w/o native libs.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-05-29 09:04:08 -07:00
Sandeep Pal 5157521213
HBASE-25927: Fix the log messages by not stringifying the exceptions in log (#3338) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
(cherry picked from commit 9a2027bf71)
 2021-05-28 21:34:14 -07:00
Duo Zhang 2a84fd40d3 HBASE-25938 The SnapshotOfRegionAssignmentFromMeta.initialize call in FavoredNodeLoadBalancer is just a dummy one (#3329) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-05-29 11:10:52 +08:00
Baiqiang Zhao bbe9558040
HBASE-25745 Deprecate/Rename config `hbase.normalizer.min.region.count` to `hbase.normalizer.merge.min.region.count` 
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2021-05-28 13:10:20 -07:00
Victor 5aa39a975c
HBASE-25910 - Fix port assignment test (#3308) 
Signed-off-by: David Manning <david.manning@salesforce.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-05-28 20:53:20 +05:30
Wellington Ramos Chevreuil cad9ae386a HBASE-25933 Log trace raw exception, instead of cause message in NettyRpcServerRequestDecoder (#3323) 
Signed-off-by: Rushabh Shah <shahrs87@gmail.com>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
 2021-05-27 20:03:17 +01:00
Duo Zhang 3cf0678aee HBASE-25926 Cleanup MetaTableAccessor references in FavoredNodeBalancer related code (#3313) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-05-27 16:25:01 +08:00
Rushabh Shah e265eccf20
HBASE-25924 Re-compute size of WAL file while removing from WALEntryStream (#3315) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2021-05-26 10:42:03 -07:00
Baiqiang Zhao 563ebc214a
HBASE-25534 Honor TableDescriptor settings earlier in normalization (#2917) 
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2021-05-25 08:24:05 -07:00
Duo Zhang 2cb6cc8a6c HBASE-25894 Improve the performance for region load and region count related cost functions (#3276) 
Signed-off-by: Yi Mei <myimeiyi@gmail.com>
 2021-05-25 18:07:33 +08:00
GeorryHuang 8754e88fa7
HBASE-25906 UI of master-status to show recent history of balancer desicion (#3303) (#3296) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-05-25 12:39:47 +05:30
Baiqiang Zhao 073f23e2bd HBASE-25513 When the table is turned on normalize, the first region may not be merged even the size is 0 (#2887) 
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2021-05-24 17:27:06 -07:00
Anoop Sam John 1649013b99 HBASE-25898 RS getting aborted due to NPE in Replication WALEntryStream (#3292) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Rushabh Shah <shahrs87@gmail.com>
 2021-05-24 23:44:32 +05:30
Xiaolin Ha 76d56bc42e HBASE-25899 Improve efficiency of SnapshotHFileCleaner (#3280) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-05-24 22:18:43 +08:00
Duo Zhang 1c1acee2fb HBASE-25873 Refactor and cleanup the code for CostFunction (#3274) 
Signed-off-by: Yi Mei <myimeiyi@gmail.com>
 2021-05-24 22:13:01 +08:00
Xiaolin Ha 16f865f65d HBASE-25773 TestSnapshotScannerHDFSAclController.setupBeforeClass is flaky (#3160) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-05-22 22:20:47 +08:00
caoliqing a19dee1eed HBASE-25892: 'False' should be 'True' in auditlog of listLabels (#3273) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-05-22 17:25:43 +08:00
Sandeep Pal 926c1132c0
HBASE-25848: Add flexibility to backup replication in case replication filter throws an exception 
Signed-off-by: Xu Cang <xcang@salesforce.com>
 2021-05-21 12:14:35 -07:00
Andrew Purtell 2d591ab3c4 HBASE-25869 WAL value compression (#3244) 
WAL storage can be expensive, especially if the cell values
represented in the edits are large, consisting of blobs or
significant lengths of text. Such WALs might need to be kept around
for a fairly long time to satisfy replication constraints on a space
limited (or space-contended) filesystem.

We have a custom dictionary compression scheme for cell metadata that
is engaged when WAL compression is enabled in site configuration.
This is fine for that application, where we can expect the universe
of values and their lengths in the custom dictionaries to be
constrained. For arbitrary cell values it is better to use one of the
available compression codecs, which are suitable for arbitrary albeit
compressible data.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2021-05-21 11:53:36 -07:00
Rushabh Shah 7b374b7c76
HBASE-25827 Per Cell TTL tags get duplicated with increments causing tags length overflow (#3290)(#3210) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-05-21 22:58:32 +05:30
Duo Zhang d78c164429 HBASE-25897 TestRetainAssignmentOnRestart is flaky after HBASE-25032 (#3281) 
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
 2021-05-20 21:18:09 +08:00
GeorryHuang 3040be1438
HBASE-25791 UI of master-status to show a recent history of that why balancer was rejected to run (#3278) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-05-20 15:00:00 +08:00
Nick Dimiduk aab6e1d0ff
HBASE-25861 Correct the usage of Configuration#addDeprecation (#3249) 
Co-authored-by: Baiqiang Zhao <zbq.dean@gmail.com>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2021-05-18 06:53:31 -07:00
Duo Zhang 8110b18ab0 HBASE-25032 Do not assign regions to region server which has not called regionServerReport yet (#3268) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
 2021-05-18 10:43:01 +08:00
Pankaj 1a8d3c3b6b HBASE-25875 RegionServer failed to start with IllegalThreadStateException due to race condition in AuthenticationTokenSecretManager (#3250) 
* HBASE-25875 RegionServer failed to start with IllegalThreadStateException due to race condition in AuthenticationTokenSecretManager's start & retrievePassword method

Signed-off-by: stack <stack@apache.com>
(cherry picked from commit 2126ec94f0)
 2021-05-17 12:21:47 +05:30
Duo Zhang 439fd4aff2 HBASE-25872 Add documentation for LoadBalancer about synchronization (#3267) 
Signed-off-by: Xin Sun <ddupgs@gmail.com>
 2021-05-17 10:20:48 +08:00
Duo Zhang d5d972f92e HBASE-25883 The regionFinder and rackManager fields in BaseLoadBalancer should be volatile (#3262) 
Signed-off-by: meiyi <myimeiyi@gmail.com>
 2021-05-17 09:54:40 +08:00
Viraj Jasani 1bb16fea5b
HBASE-25884 Return empty records for disabled balancer in-memory queue (#3263) 
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Aman Poonia <apoonia@salesforce.com>
 2021-05-14 12:57:24 +05:30
Duo Zhang 72d7641fb5
HBASE-25852 Move all the intialization work of LoadBalancer implementation to initialize method (#3248) (#3258) 
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-05-13 14:19:03 +08:00
Michael Stack 075545731a
HBASE-25876 Add retry if we fail to read all bytes of the protobuf magic marker (#3251) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
 2021-05-12 11:20:50 -07:00
Michael Stack 61d9b46aab HBASE-25867 Extra doc around ITBLL (#3242) 
* HBASE-25867 Extra doc around ITBLL

Minor edits to a few log messages.
Explain how the '-c' option works when passed to ChaosMonkeyRunner.
Some added notes on ITBLL.
Fix whacky 'R' and 'Not r' thing in Master (shows when you run ITBLL).
In HRS, report hostname and port when it checks in (was debugging issue
where Master and HRS had different notions of its hostname).
Spare a dirty FNFException on startup if base dir not yet in place.

* Address Review by Sean

Signed-off-by: Sean Busbey <busbey@apache.org>
 2021-05-11 19:24:33 +01:00
Catalin Luca 4e507ccda0 HBASE-25859 Reference class incorrectly parses the protobuf magic marker (#3236) 
Co-authored-by: Catalin Luca <luca@adobe.com>
Signed-off-by: stack <stack@apache.org>
 2021-05-10 14:47:18 -05:00
GeorryHuang 63d49cb7ed
HBASE-25790 NamedQueue 'BalancerRejection' for recent history of balancer skipping (#3182) (#3245) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-05-09 00:09:23 +05:30
Rushabh Shah 90dc150b1b
HBASE-25860 Add metric for successful wal roll requests. (#3238) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-05-08 13:02:52 +05:30
Andrew Purtell d60aa0a865 HBASE-25774 ServerManager.getOnlineServer may miss some region servers when refreshing state in some procedure implementations 
Revert "HBASE-25032 Wait for region server to become online before adding it to online servers in Master (#2771)"

This reverts commit 3bb978894d.
 2021-05-07 17:55:20 -07:00
niuyulin 5e65da64e2 HBASE-25837 TestRollingRestart is flaky (#3220) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-05-07 20:28:37 +08:00
xiaozhang0319 bec79d877f
HBASE-25850 Fix spotbugs warnings on branch-2 (#3239) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-05-07 20:21:53 +08:00
Andrew Purtell 2938ba421b HBASE-25854 Remove redundant AM in-memory state changes in CatalogJanitor (#3234) 
In CatalogJanitor we schedule GCRegionProcedure to clean up both
filesystem and in-memory state after a split, and
GCMultipleMergedRegionsProcedure to do the same for merges. Both of these
procedures clean up in-memory state, but CatalogJanitor also does this
redundantly just after scheduling the procedures. The cleanup should be
done in only one place. Presumably we are using the procedures to do it in
a principled way. Remove the redundancy in CatalogJanitor and fix any
follow on issues, like test failures.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-05-06 09:17:29 -07:00
Duo Zhang 2d4904ceea
HBASE-25851 Make LoadBalancer not extend Configurable interface (#3233) (#3235) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-05-06 23:39:43 +08:00
Andrew Purtell b2c58331f3 HBASE-25847 More DEBUG and TRACE level logging in CatalogJanitor and HbckChore (#3230) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
 2021-05-05 17:02:12 -07:00
Nick Dimiduk 6f2842d1af HBASE-25843 move master http-related code into o.a.h.h.master.http 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-05-05 08:36:40 -07:00
Duo Zhang afa9836b87 HBASE-25834 Remove balanceTable method from LoadBalancer interface (#3217) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-05-05 16:03:51 +08:00
Nick Dimiduk 3f8d3ba6d9 HBASE-25842 move regionserver http-related code into o.a.h.h.regionserver.http 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-05-04 15:46:53 -07:00
Andrew Purtell 36994a2c5e HBASE-25835 Ignore duplicate split requests from regionserver reports (#3218) 
Processing of the RS report happens asynchronously from other activities
which can mutate region state. For example, a split procedure may already
be running. A split procedure cannot succeed if the parent region is no
longer open, so we can ignore it in that case.

Note that submitting more than one split procedure for a given region is
harmless -- the split is fenced in the procedure handling -- but it would
be noisy in the logs. Only one procedure can succeed. The other
procedure(s) would abort during initialization and report failure with
WARN level logging.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Pankaj <pankajkumar@apache.org>
 2021-05-04 10:22:57 -07:00
Andrew Purtell c8f3dd56b7 HBASE-25836 RegionStates#getAssignmentsForBalancer should only care about OPEN or OPENING regions (#3219) 
RegionStates#getAssignmentsForBalancer is used by the HMaster to
collect all regions of interest to the balancer for the next chore
iteration. We check if a table is in disabled state to exclude
regions that will not be of interest (because disabled regions are
or will be offline) or are in a state where they shouldn't be
mutated (like SPLITTING). The current checks are not actually
comprehensive.

Filter out regions not in OPEN or OPENING state when building the
set of interesting regions for the balancer to consider. Only
regions open (or opening) on the cluster are of interest to
balancing calculations for the current iteration. Regions in all
other states can be expected to not be of interest – either offline
(OFFLINE, or FAILED_*), not subject to balancer decisions now
(SPLITTING, SPLITTING_NEW, MERGING, MERGING_NEW), or will be
offline shortly (CLOSING) – until at least the next chore
iteration.

Add TRACE level logging.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-05-03 18:24:13 -07:00
Andrew Purtell 60055f5554 HBASE-25840 CatalogJanitor warns about skipping gc of regions during RIT, but does not actually skip (#3223) 
We claim in a WARN level log line to be "Playing-it-safe skipping merge/
split gc'ing of regions from hbase:meta while regions-in-transition (RIT)"
but do not actually skip because of a missing return. Remove the warning.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-05-03 18:17:41 -07:00
Duo Zhang e725ef540e HBASE-25838 Use double instead of Double in StochasticLoadBalancer (#3221) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-05-04 09:11:28 +08:00
Duo Zhang c52c091609 HBASE-25819 Fix style issues for StochasticLoadBalancer (#3207) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-04-29 11:26:58 +08:00
Nick Dimiduk 3a01941963 HBASE-25779 HRegionServer#compactSplitThread should be private 
Minor refactor. Make the `compactSplitThread` member field of `HRegionServer` private, and gate
all access through the getter method.

Signed-off-by: Yulin Niu <niuyulin@apache.org>
Signed-off-by: Pankaj Kumar <pankajkumar@apache.org>
 2021-04-28 16:54:44 -07:00
stack 04c30bda88 HBASE-25792 Filter out o.a.hadoop.thirdparty building shaded jars (#3184) 
Need to add to allowed-licenses list too....

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Duo Zhang <zhangduo@apache.org>
Reviewed-by: Nick Dimiduk <ndimiduk@apache.org>
 2021-04-27 08:41:15 -07:00
Andrew Purtell 01ce44abc4 HBASE-25756 Support alternate compression for major and minor compactions (#3142) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>

Conflicts:
	hbase-client/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java
	hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
	hbase-shell/src/main/ruby/hbase/admin.rb
 2021-04-23 16:02:41 -07:00
Duo Zhang f374357cc3 HBASE-25802 Miscellaneous style improvements for load balancer related classes (#3192) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-04-23 16:32:10 +08:00
haxiaolin fd7b2c6eec HBASE-25754 StripeCompactionPolicy should support compacting cold regions (#3152) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-04-23 15:31:55 +08:00
Duo Zhang a02ce95ff1
HBASE-25793 Move BaseLoadBalancer.Cluster to a separated file (#3185) (#3190) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-04-23 00:08:16 +08:00
Toshihiro Suzuki f4f84302fa HBASE-25766 Introduce RegionSplitRestriction that restricts the pattern of the split point 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
 2021-04-22 14:06:28 +09:00
Duo Zhang a177fb4c62 HBASE-25775 Use a special balancer to deal with maintenance mode (#3161) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2021-04-16 13:39:48 +08:00
Nick Dimiduk 32207cd478 HBASE-25770 Http InfoServers should honor gzip encoding when requested (#3159) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
 2021-04-15 13:46:53 -07:00
xiaoyu 819062bac3
HBASE-25776 Use Class.asSubclass to fix the warning in StochasticLoadBalancer.loadCustomCostFunctions (#3163) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-04-15 23:35:33 +05:30
DivyeshChandra 35520a72fb HBASE-25653 Add units and round off region size to 2 digits after decimal (#3046) 
Signed-off-by: stack <stack@duboce.net>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-04-13 09:06:05 -07:00
Duo Zhang 15659f0e85 HBASE-25767 CandidateGenerator.getRandomIterationOrder is too slow on large cluster (#3149) 
Signed-off-by: XinSun <ddupgs@gmail.com>
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-04-13 23:14:48 +08:00
Baiqiang Zhao ae3daa5c0b HBASE-25482 Improve SimpleRegionNormalizer#getAverageRegionSizeMb (#2858) 
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2021-04-12 15:19:43 -07:00
Duo Zhang 702220a135 HBASE-25759 The master services field in LocalityBasedCostFunction is never used (#3144) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-04-12 22:31:05 +08:00
Nick Dimiduk b071902b10 HBASE-25744 Change default of `hbase.normalizer.merge.min_region_size.mb` to `0` 
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
 2021-04-11 16:22:02 -07:00
Geoffrey Jacoby 89c9f28d3f HBASE-25751 - Add writable TimeToPurgeDeletes to ScanOptions (#3137) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-04-09 13:09:24 -07:00
meiyi 3faa67520c HBASE-25747 Remove unused getWriteAvailable method in OperationQuota (#3133) 
Signed-off-by: stack <stack@apache.org>
 2021-04-09 10:41:51 +08:00
Pankaj 89fa8e7333
HBASE-25717 RegionServer aborted with due to ClassCastException (#3108) 
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-04-08 16:28:41 +05:30
Baiqiang Zhao 8ff17c68e2
HBASE-25687 Backport "HBASE-25681 Add a switch for server/table query… (#3074) 
Signed-off-by: stack <stack@apache.org>
 2021-04-07 11:11:46 -07:00
Jan Hentschel 18882d6f9a
HBASE-24305 Prepare deprecations in ServerName (#1666) (#3121) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
 2021-04-05 22:10:04 +02:00
Jan Hentschel 1fa07cea80 HBASE-25199 Remove deprecated HStore#getStoreHomedir methods (#2562) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-04-03 23:44:23 +08:00
d-c-manning 554e039817 HBASE-25726 MoveCostFunction is not included in the list of cost functions for StochasticLoadBalancer (#3116) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2021-04-02 10:43:51 -07:00
stack b051ad5594 Revert "HBASE-25713 Make an hbase-wal module" 
This reverts commit e58f1b24a6.
 2021-03-31 16:15:50 -07:00
lujiefsi d90d0c2231 HBASE-25558:Adding audit log for execMasterService (#3101) 
Signed-off-by: stack <stack@apache.org>
 2021-03-31 16:14:50 -07:00
stack e58f1b24a6 HBASE-25713 Make an hbase-wal module 
First extract an hbase-coprocessor module used by hbase-client, hbase-server.
This is prerequisite to extracting an hbase-wal module.

M hbase-common/src/main/java/org/apache/hadoop/hbase/Abortable.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/DoNotRetryIOException.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/util/SortedList.java
 Move to hbase-common. Its a generic Interface. Need by

M hbase-coprocessor/src/main/java/org/apache/hadoop/hbase/Coprocessor.java
M hbase-coprocessor/src/main/java/org/apache/hadoop/hbase/CoprocessorEnvironment.java
M hbase-coprocessor/src/main/java/org/apache/hadoop/hbase/coprocessor/BaseEnvironment.java
M hbase-coprocessor/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
M hbase-coprocessor/src/main/java/org/apache/hadoop/hbase/coprocessor/CoreCoprocessor.java
M hbase-coprocessor/src/main/java/org/apache/hadoop/hbase/coprocessor/ObserverContext.java
M hbase-coprocessor/src/main/java/org/apache/hadoop/hbase/coprocessor/ObserverContextImpl.java
M hbase-coprocessor/src/main/java/org/apache/hadoop/hbase/coprocessor/ReadOnlyConfiguration.java
 Move to hbase-coprocessor.

M hbase-endpoint/src/main/java/org/apache/hadoop/hbase/client/coprocessor/BigDecimalColumnInterpreter.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/coprocessor/DoubleColumnInterpreter.java
M hbase-endpoint/src/main/java/org/apache/hadoop/hbase/client/coprocessor/LongColumnInterpreter.java
M hbase-endpoint/src/main/java/org/apache/hadoop/hbase/coprocessor/ColumnInterpreter.java
 Moved to hbase-endpoint where they are used.
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionCoprocessorHost.java
 Include region name when toString'd.
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCoprocessorHost.java
 Include WAL name when toString'd.
M hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
 Add utility used in testing here from CoprocessorHost.
 2021-03-31 16:14:45 -07:00
Toshihiro Suzuki 1d3ea38f1e
HBASE-25703 Support conditional update in MultiRowMutationEndpoint (#3107) 
Signed-off-by: Michael Stack <stack@apache.org>
 2021-03-30 23:04:29 +09:00
Josh Elser fa8b9659b1 HBASE-25692 Always try to close the WAL reader when we catch any exception (#3090) 
There are code paths in which we throw non-IOExceptions when
initializing a WAL reader. However, we only close the InputStream to the
WAL filesystem when the exception is an IOException. Close it if it is
open in all cases.

Co-authored-by: Josh Elser <jelser@cloudera.com>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2021-03-29 12:17:16 -07:00
Michael Stack 84eb0220a4
HBASE-25695 Link to the filter on hbase:meta from user tables panel on master page (#3092) 
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2021-03-27 20:35:20 -07:00
caroliney14 3bb978894d
HBASE-25032 Wait for region server to become online before adding it to online servers in Master (#2771) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
 2021-03-25 17:10:43 -07:00
Andrew Purtell 124ea5eee3 HBASE-25693 NPE getting metrics from standby masters (MetricsMasterWrapperImpl.getMergePlanCount) (#3091) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2021-03-24 19:10:58 -07:00
huaxiangsun 80529af963
HBASE-25691 Test failure: TestVerifyBucketCacheFile.testRetrieveFromFile (#3087) 
The issue is that FileInputStream is created with try-with-resources, so its close() is called right after the try sentence.
FileInputStream is a finalize class, when this object is garbage collected, its close() is called again.
To avoid this double-free resources, add guard against it.

Signed-off-by: stack <stack@apache.org>
 2021-03-24 14:38:44 -07:00
Toshihiro Suzuki 2b5c2c36d9
HBASE-25678 Support nonce operations for Increment/Append in RowMutations and CheckAndMutate (#3073) 
Signed-off-by: stack <stack@apache.org>
 2021-03-21 22:47:53 +09:00
shahrs87 ed8b72627d
HBASE-25679 Size of log queue metric is incorrect (#3071) 
Co-authored-by: Rushabh <rushabh.shah@salesforce.com>
Signed-off-by: stack <stack@apache.org>
 2021-03-19 16:14:45 -07:00
haxiaolin 566b0512a7 HBASE-25518 Support separate child regions to different region servers (#3001) 
Signed-off-by: stack <stack@apache.org>
 2021-03-18 12:39:00 -07:00
bsglz 19d094b96f HBASE-25643 The delayed FlushRegionEntry should be removed when we ne… (#3049) 
Signed-off-by: AnoopSamJohn <anoopsamjohn@apache.org>
Signed-off-by: stack <stack@apache.org>
 2021-03-18 12:13:34 -07:00
Michael Stack 75e2e7c83d HBASE-25677 Server+table counters on each scan #nextRaw invocation becomes a bottleneck when heavy load (#3061) 
Don't have every handler update regionserver metrics on each
scan#nextRaw; instead, do a batch update just before Scan
returns. Otherwise, all running handlers end up contending
on metrics update.

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 Update of regionserver metrics counters moved out to caller where
 can be done as a batch update instead of per-next.

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServer.java
 Class doc to encourage batch updating metrics.
 Remove the single update as unused anymore.

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
 Count calls to nextRaw. Update regionserver count in finally block when
 scan is done rather than per nextRaw call. Move all metrics updates to
 finally.

Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Baiqiang Zhao <ZhaoBQ>
 2021-03-18 11:40:42 -07:00
Sandeep Pal 72496272aa
HBASE-25627: HBase replication should have a metric to represent if the source is stuck getting initialized (#3018) 
Introduces a new metric that tracks number of replication sources that are stuck in initialization.

Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
(cherry picked from commit ff3821814a)
 2021-03-17 10:30:26 -07:00
bitterfox a61e9b963d HBASE-25665 Option to use hostname instead of canonical hostname for secure HBase cluster connection (#3051) 2021-03-16 21:05:23 -07:00
Baiqiang Zhao a93b0cd3ca HBASE-25597 Add row info in Exception when cell size exceeds maxCellSize (#2976) 
Signed-off-by: stack <stack@apache.org>
 2021-03-15 15:50:22 -07:00
haxiaolin 6c224b8640 HBASE-25621 Balancer should check region plan source to avoid misplace region groups (#3002) 
Signed-off-by: stack <stack@duboce.net>
 2021-03-15 14:48:35 -07:00
haxiaolin 13574de156 HBASE-25595 TestLruBlockCache.testBackgroundEvictionThread is flaky (#2974) 
Signed-off-by: stack <stack@apache.org>
 2021-03-15 14:26:06 -07:00
Michael Stack 3bf2ef4259
HBASE-25660 Print split policy in use on Region open (as well as split policy vitals) (#3044) 
Add a toString to all split policy implementations listing name and
 vitals. Use this toString in the Region open message. Ditto for flush
 policy for the Region.

 Signed-off-by: Huaxiang Sun<huaxiangsun@apache.org>
 2021-03-15 14:10:37 -07:00
haxiaolin 95e8ee5ca5 HBASE-25635 CandidateGenerator may miss some region balance actions (#3024) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-03-15 21:33:46 +08:00
Duo Zhang aafbbc2456 HBASE-25657 Fix spotbugs warnings after upgrading spotbugs to 4.x (#3041) 
Signed-off-by: meiyi <myimeiyi@gmail.com>
Signed-off-by: stack <stack@apache.org>
 2021-03-12 14:37:17 +08:00
shahrs87 bb4d8b9cf7 HBASE-25622 Result#compareResults should compare tags. (#3026) 
Signed-off-by: stack <stack@apache.org>
 2021-03-11 21:52:11 -08:00
Michael Stack 45b3280497 HBASE-25570 On largish cluster, "CleanerChore: Could not delete dir..." makes master log unreadable (#2949) 
Turn down the amount we log. If you want to see the full exception
enable TRACE-level logging.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: shahrs87
 2021-03-11 21:36:31 -08:00
Peter Somogyi af8dc2e4df HBASE-25587 [hbck2] Schedule SCP for all unknown servers (#2978) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2021-03-11 17:22:08 +01:00
meiyi b5fc5e17e2 HBASE-25636 Expose HBCK report as metrics (#3031) 
Signed-off-by: zhangduo <zhangduo@apache.org>
 2021-03-11 15:15:23 +08:00
Duo Zhang d546c8e77c HBASE-25629 Reimplement TestCurrentHourProvider to not depend on unstable TZs (#3013) 
Signed-off-by: XinSun <ddupgs@gmail.com>
 2021-03-10 23:53:59 +08:00
Wellington Ramos Chevreuil 7adc00044c HBASE-25548 Optionally allow snapshots to preserve cluster's max file… (#2923) 
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
(cherry picked from commit 373dc7788d)
 2021-03-10 10:35:58 +00:00
Anoop Sam John 29d771fab9
HBASE-25582 Support setting scan ReadType to be STREAM at cluster level (#3040) 
Signed-off-by: zhangduo <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-03-10 13:23:15 +05:30
Narges Shadab 884cb0bd2c
HBASE-25646: Possible Resource Leak in CatalogJanitor #3036 
Close the stream using a try-with-resources block.

Reviewed-by: Aman Poonia <aman.poonia.29@gmail.com>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
(cherry picked from commit 573daed419)
 2021-03-09 09:11:02 -08:00
Bharath Vissapragada 321b5c7979
HBASE-25547: Thread pools should release unused resources (#3037) 
* HBASE-25547: Thread pools should release unused resources (#2922)

Plumbs the configuration needed to enable core thread timeout on non-critical thread pools.
Currently only enabled for thread pools with op-codes RS_LOG_REPLAY_OPS, RS_PARALLEL_SEEK, MASTER_SNAPSHOT_OPERATIONS, MASTER_MERGE_OPERATIONS. Others can be added later as
needed.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
(cherry picked from commit 618236dd90)

* HBASE-25547 (addendum): Roll ExecutorType into ExecutorConfig

Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
(cherry picked from commit 4c822d7463)
 2021-03-09 09:03:16 -08:00
Duo Zhang f1dfd3b0fe HBASE-23578 [UI] Master UI shows long stack traces when table is broken (#3014) 
Co-authored-by: Shuhei Yamasaki <yamasakisua@oss.nttdata.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-03-07 21:36:24 +08:00
niuyulin 88b9e98a7d HBASE-25637 Rename method completeCompaction to refreshStoreSizeAndTotalBytes (#3023) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-03-06 18:43:47 +08:00
Narges Shadab c9d0ec903d
HBASE-25626 Possible Resource Leak in HeterogeneousRegionCountCostFunction 
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by shahrs87 <shahrs87@gmail.com>
 2021-03-05 08:56:12 +05:30
Rahul Kumar e57c73a137
HBASE-25460 : Expose drainingServers as cluster metric (#2994) (#2995) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-03-04 12:48:57 +05:30
Andrew Purtell dce639ae81 HBASE-25385 TestCurrentHourProvider fails if the latest timezone changes are not present (#3012) 
Disable this problematic test for now. May be removed after additional
discussion.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2021-03-03 18:16:20 -08:00
niuyulin 621edfe1da HBASE-25603 Add switch for compaction after bulkload (#2982) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2021-03-04 10:11:19 +08:00
Sandeep Pal 80b835927c
HBASE-25596: Fix NPE and avoid permanent unreplicated data due to EOF (#3008) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2021-03-03 11:38:46 -08:00
Akshay Sudheer 63f2c1cdac HBASE-25367 Sort broken after Change 'State time' in UI (#2964) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
(cherry picked from commit 5d9a6ed1fe)
 2021-03-03 14:00:05 +05:30
Mohammad Arshad 1cd8638508 HBASE-25371: When openRegion fails during initial verification(before… (#2785) 
Signed-off-by: stack <stack@apache.org>
 2021-03-03 08:28:21 +08:00
Andrew Purtell 0849fae730 Revert "HBASE-25596: Fix NPE and avoid permanent unreplicated data due to EOF (#2990)" 
This reverts commit d724d0576f.
 2021-03-02 12:26:23 -08:00
Balazs Meszaros f2f4b3f3ad HBASE-25586 Fix HBASE-22492 on branch-2 (SASL GapToken) (#2961) 
ServerCall.java: calling wrapWithSasl() was moved to getResponse(), so
the SASL wrapping is delayed until the reply is sent back to the client.

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
 2021-02-26 14:09:24 +01:00
XinSun 328ff8c05a HBASE-25590 Bulkload replication HFileRefs cannot be cleared in some cases where set exclude-namespace/exclude-table-cfs (#2969) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2021-02-26 09:58:05 +08:00
Sandeep Pal d724d0576f
HBASE-25596: Fix NPE and avoid permanent unreplicated data due to EOF (#2990) 
Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2021-02-25 13:36:31 -08:00
XinSun e863df383b HBASE-25598 TestFromClientSide5.testScanMetrics is flaky (#2977) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-02-24 14:19:36 +08:00
Andrew Purtell fb0d7724aa HBASE-25556 Frequent replication "Encountered a malformed edit" warnings (#2965) 
ProtobufLogReader#readNext may be called by code that attempts to advance
the reader but does not necessarily expect to succeed, for example
WALEntryStream#tryAdvanceEntry. Much of the logging in this method is
at TRACE level. Other logging at WARN level will be frequently emitted, as
often as several per minute, and this will cause false positive assessment
from operators that they are experiencing a bug. Fix the mixed intent with
respect to log levels in readNext. Log at only DEBUG level or below.

Signed-off-by: Sean Busbey <busbey@apache.org>
 2021-02-23 14:41:49 -08:00
Toshihiro Suzuki cfbae4d3a3 HBASE-25575 Should validate Puts in RowMutations 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-02-22 11:02:13 +09:00
Toshihiro Suzuki 5356edfe63 HBASE-25574 Revisit put/delete/increment/append related RegionObserver methods 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-02-22 09:51:12 +09:00
XinSun a3edcc2854 HBASE-25562 ReplicationSourceWALReader log and handle exception immediately without retrying (#2943) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: shahrs87
 2021-02-20 10:23:54 +08:00
shahrs87 6a4c9be967
HBASE-25539: Add age of oldest wal metric (#2962) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2021-02-18 20:59:07 -08:00
Sandeep Pal ec680c52e0
HBASE-25541 : Setting the path to null when we dequeue the current log (#2959) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2021-02-18 11:55:59 +05:30
pustota2009 d784e3f177
HBASE-23887 Bug fix heavyEvictionMbSizeLimit (ADDENDUM) (#2957) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-02-15 20:45:27 +05:30
Mohammad Arshad 2e31d1d46e
HBASE-25571: Compilation error in branch-2 after HBASE-25364 (#2951) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-02-13 21:41:04 +05:30
pustota2009 2d5a5db80e
HBASE-23887 AdaptiveLRU cache (#2934) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-02-13 19:58:55 +05:30
GeorryHuang a61f1082de HBASE-25364 Redo the getMidPoint() in HFileWriterImpl to get rid of the double comparison process (#2741) 
Signed-off-by: niuyulin <nyl353@163.com>
Signed-off-by: stack <stack@apache.org>
 2021-02-10 08:31:51 -08:00
XinSun b05dcac9fd HBASE-25559 Terminate threads of oldsources while RS is closing (#2938) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2021-02-09 16:40:52 +08:00
haxl e332c9498c HBASE-25507 Leak of ESTABLISHED sockets when compaction encountered "java.io.IOException: Invalid HFile block magic" (#2882) 
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: stack <stack@apache.org>
 2021-02-08 22:15:43 -08:00
Michael Stack 3c29eaed35
HBASE-25542 Add client detail to scan name so when lease expires, we … (#2930) 
* HBASE-25542 Add client detail to scan name so when lease expires, we have clue on who was scanning

When we create a scanner lease, record client ip and port (removed
unnecessary store of scannerName).

Signed-off-by: Clara Xiong <clarax98007@gmail.com>
 2021-02-08 14:33:00 -08:00