Apache
/
hbase
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
18,690 Commits 95 Branches 866 Tags 550 MiB
Commit Graph

8845 Commits

Author SHA1 Message Date
Baiqiang Zhao a1177b3e91
HBASE-25682 Add a new command to update the configuration of all RSs in a RSGroup (#3080) 
* HBASE-25682 Add a new command to update the configuration of all RSs in a RSGroup

Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
 2021-05-21 22:49:25 +05:30
Sandeep Pal 15e861169f
HBASE-25848: Add flexibility to backup replication in case replication filter throws an exception (#3283) 
* HBASE-25848: Add flexibility to backup replication in case replication filter throws an exception
 2021-05-20 13:31:44 -07:00
Duo Zhang 7c24ed4f45
HBASE-25897 TestRetainAssignmentOnRestart is flaky after HBASE-25032 (#3281) 
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
 2021-05-20 20:58:53 +08:00
GeorryHuang 5b9940907e
HBASE-25791 UI of master-status to show a recent history of that why balancer was rejected to run (#3275) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-05-19 12:05:29 +08:00
Duo Zhang 741b4b4674
HBASE-25032 Do not assign regions to region server which has not called regionServerReport yet (#3268) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
 2021-05-18 08:08:03 +08:00
Pankaj 2126ec94f0
HBASE-25875 RegionServer failed to start with IllegalThreadStateException due to race condition in AuthenticationTokenSecretManager (#3250) 
* HBASE-25875 RegionServer failed to start with IllegalThreadStateException due to race condition in AuthenticationTokenSecretManager's start & retrievePassword method

Signed-off-by: stack <stack@apache.com>
 2021-05-17 12:17:24 +05:30
Baiqiang Zhao d69d5c24b1
HBASE-25861 Correct the usage of Configuration#addDeprecation (#3249) 
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2021-05-14 09:31:06 -07:00
Viraj Jasani 0955a7a22e
HBASE-25884 Return empty records for disabled balancer in-memory queue (#3263) 
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Aman Poonia <apoonia@salesforce.com>
 2021-05-14 12:54:07 +05:30
Michael Stack 630c73fda4 HBASE-25867 Extra doc around ITBLL (#3242) 
* HBASE-25867 Extra doc around ITBLL

Minor edits to a few log messages.
Explain how the '-c' option works when passed to ChaosMonkeyRunner.
Some added notes on ITBLL.
Fix whacky 'R' and 'Not r' thing in Master (shows when you run ITBLL).
In HRS, report hostname and port when it checks in (was debugging issue
where Master and HRS had different notions of its hostname).
Spare a dirty FNFException on startup if base dir not yet in place.

* Address Review by Sean

Signed-off-by: Sean Busbey <busbey@apache.org>
 2021-05-11 19:26:57 +01:00
Duo Zhang 29bd3dd586
HBASE-25852 Move all the intialization work of LoadBalancer implementation to initialize method (#3248) 
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-05-11 22:03:33 +08:00
Catalin Luca 2b6a91a1da
HBASE-25859 Reference class incorrectly parses the protobuf magic marker (#3236) 
Co-authored-by: Catalin Luca <luca@adobe.com>
Signed-off-by: stack <stack@apache.org>
 2021-05-10 12:45:23 -07:00
Rushabh Shah 8c2332d465
HBASE-25860 Add metric for successful wal roll requests. (#3238) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-05-08 12:58:29 +05:30
Duo Zhang c2a1d31270 HBASE-25774 Addendum fix compile error 2021-05-08 13:56:57 +08:00
Andrew Purtell 02b018cf1a HBASE-25774 ServerManager.getOnlineServer may miss some region servers when refreshing state in some procedure implementations 
Revert "HBASE-25032 Wait for region server to become online before adding it to online servers in Master (#2769)"

This reverts commit 1e4639d2eb.

Conflicts:

	hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
 2021-05-07 18:21:46 -07:00
niuyulin 6cfff27465
HBASE-25837 TestRollingRestart is flaky (#3220) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-05-07 18:58:45 +08:00
Andrew Purtell 6309c090b5
HBASE-25854 Remove redundant AM in-memory state changes in CatalogJanitor (#3234) 
In CatalogJanitor we schedule GCRegionProcedure to clean up both
filesystem and in-memory state after a split, and
GCMultipleMergedRegionsProcedure to do the same for merges. Both of these
procedures clean up in-memory state, but CatalogJanitor also does this
redundantly just after scheduling the procedures. The cleanup should be
done in only one place. Presumably we are using the procedures to do it in
a principled way. Remove the redundancy in CatalogJanitor and fix any
follow on issues, like test failures.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-05-06 09:13:33 -07:00
Duo Zhang ba4cb91211
HBASE-25851 Make LoadBalancer not extend Configurable interface (#3233) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-05-06 16:11:46 +08:00
Andrew Purtell cc88cf0ecf
HBASE-25847 More DEBUG and TRACE level logging in CatalogJanitor and HbckChore (#3230) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
 2021-05-05 17:01:00 -07:00
Nick Dimiduk eb9b54304e HBASE-25843 move master http-related code into o.a.h.h.master.http 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-05-05 08:34:37 -07:00
Duo Zhang 90f986497b
HBASE-25834 Remove balanceTable method from LoadBalancer interface (#3217) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-05-05 15:48:01 +08:00
Nick Dimiduk 17193dae58 HBASE-25842 move regionserver http-related code into o.a.h.h.regionserver.http 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-05-04 15:40:17 -07:00
Andrew Purtell 432d141474
HBASE-25835 Ignore duplicate split requests from regionserver reports (#3218) 
Processing of the RS report happens asynchronously from other activities
which can mutate region state. For example, a split procedure may already
be running. A split procedure cannot succeed if the parent region is no
longer open, so we can ignore it in that case.

Note that submitting more than one split procedure for a given region is
harmless -- the split is fenced in the procedure handling -- but it would
be noisy in the logs. Only one procedure can succeed. The other
procedure(s) would abort during initialization and report failure with
WARN level logging.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Pankaj <pankajkumar@apache.org>
 2021-05-04 10:05:29 -07:00
Andrew Purtell fda324b116
HBASE-25836 RegionStates#getAssignmentsForBalancer should only care about OPEN or OPENING regions (#3219) 
RegionStates#getAssignmentsForBalancer is used by the HMaster to
collect all regions of interest to the balancer for the next chore
iteration. We check if a table is in disabled state to exclude
regions that will not be of interest (because disabled regions are
or will be offline) or are in a state where they shouldn't be
mutated (like SPLITTING). The current checks are not actually
comprehensive.

Filter out regions not in OPEN or OPENING state when building the
set of interesting regions for the balancer to consider. Only
regions open (or opening) on the cluster are of interest to
balancing calculations for the current iteration. Regions in all
other states can be expected to not be of interest – either offline
(OFFLINE, or FAILED_*), not subject to balancer decisions now
(SPLITTING, SPLITTING_NEW, MERGING, MERGING_NEW), or will be
offline shortly (CLOSING) – until at least the next chore
iteration.

Add TRACE level logging.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-05-03 18:23:07 -07:00
Andrew Purtell e44592a37d
HBASE-25840 CatalogJanitor warns about skipping gc of regions during RIT, but does not actually skip (#3223) 
We claim in a WARN level log line to be "Playing-it-safe skipping merge/
split gc'ing of regions from hbase:meta while regions-in-transition (RIT)"
but do not actually skip because of a missing return. Remove the warning.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-05-03 18:14:38 -07:00
Duo Zhang 762abe3bea
HBASE-25838 Use double instead of Double in StochasticLoadBalancer (#3221) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-05-04 09:04:47 +08:00
Duo Zhang 7640134e3e
HBASE-25774 Added more detailed logs about the restarting of region servers (#3213) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-05-03 20:33:33 +08:00
GeorryHuang 00fec24c90
HBASE-25790 NamedQueue 'BalancerRejection' for recent history of balancer skipping (#3182) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-05-02 21:30:48 +05:30
Duo Zhang 73a82bd7c6
HBASE-25825 RSGroupBasedLoadBalancer.onConfigurationChange should chain the request to internal balancer (#3209) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-04-30 22:45:33 +08:00
Duo Zhang 6c65314cdf
HBASE-25819 Fix style issues for StochasticLoadBalancer (#3207) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-04-29 11:03:55 +08:00
Nick Dimiduk b061b0c4ed HBASE-25779 HRegionServer#compactSplitThread should be private 
Minor refactor. Make the `compactSplitThread` member field of `HRegionServer` private, and gate
all access through the getter method.

Signed-off-by: Yulin Niu <niuyulin@apache.org>
Signed-off-by: Pankaj Kumar <pankajkumar@apache.org>
 2021-04-28 16:46:36 -07:00
Michael Stack 2382f68b23
HBASE-25792 Filter out o.a.hadoop.thirdparty building shaded jars (#3184) 
Need to add to allowed-licenses list too....

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Duo Zhang <zhangduo@apache.org>
Reviewed-by: Nick Dimiduk <ndimiduk@apache.org>
 2021-04-27 08:37:25 -07:00
Duo Zhang 8856f61986 HBASE-25757 Addendum remove CandidateGenerator classes under hbase-server module 2021-04-27 23:25:51 +08:00
Duo Zhang a4d954e606
HBASE-25757 Move BaseLoadBalancer to hbase-balancer module (#3191) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-04-26 12:03:25 +08:00
Duo Zhang 7f90c2201f HBASE-25723 Temporarily remove the trace support for RegionScanner.next (#3119) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-04-25 09:23:23 +08:00
Duo Zhang f6ff519dd0 HBASE-25591 Upgrade opentelemetry to 0.17.1 (#2971) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2021-04-25 09:23:23 +08:00
Duo Zhang bb8c4967f8 HBASE-25535 Set span kind to CLIENT in AbstractRpcClient (#2907) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2021-04-25 09:23:23 +08:00
Duo Zhang 2be2c63f0d HBASE-25484 Add trace support for WAL sync (#2892) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2021-04-25 09:23:23 +08:00
Duo Zhang 03e12bfa4a HBASE-25455 Add trace support for HRegion read/write operation (#2861) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2021-04-25 09:23:23 +08:00
Duo Zhang ae2c62ffaa HBASE-25481 Add host and port attribute when tracing rpc call at client side (#2857) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2021-04-25 09:23:23 +08:00
Duo Zhang 805b2ae2ad HBASE-23898 Add trace support for simple apis in async client (#2813) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2021-04-25 09:23:23 +08:00
Duo Zhang 2420286715 HBASE-25401 Add trace support for async call in rpc client (#2790) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2021-04-25 09:23:23 +08:00
Duo Zhang 302d9ea8b8 HBASE-25373 Remove HTrace completely in code base and try to make use of OpenTelemetry 
Signed-off-by: stack <stack@apache.org>
 2021-04-25 09:23:23 +08:00
Andrew Purtell 9895b2dfdf
HBASE-25756 Support alternate compression for major and minor compactions (#3142) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-04-23 15:45:26 -07:00
Duo Zhang 96fefce9c3
HBASE-25802 Miscellaneous style improvements for load balancer related classes (#3192) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-04-23 15:20:27 +08:00
haxiaolin 996862c1cc
HBASE-25754 StripeCompactionPolicy should support compacting cold regions (#3152) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-04-23 14:58:53 +08:00
Toshihiro Suzuki 5f4e2e111b
HBASE-25766 Introduce RegionSplitRestriction that restricts the pattern of the split point (#3150) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
 2021-04-22 13:53:36 +09:00
Duo Zhang 50920ee306
HBASE-25774 TestSyncReplicationStandbyKillRS#testStandbyKillRegionServer is flaky (#3189) 
Wait for the restarter thread to finish before checking the state
Add more detailed logs

Signed-off-by: meiyi <myimeiyi@gmail.com>
 2021-04-22 10:10:15 +08:00
Duo Zhang d5c5e48839
HBASE-25793 Move BaseLoadBalancer.Cluster to a separated file (#3185) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-04-22 09:59:49 +08:00
haxiaolin 0d257baf29
HBASE-25763 TestRSGroupsWithACL.setupBeforeClass is flaky (#3158) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-04-21 14:41:51 +08:00
Duo Zhang 781da1899a
HBASE-25290 Remove table on master related code in balancer implementation (#3162) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-04-20 21:31:09 +08:00
Nick Dimiduk b65890da1d Revert "HBASE-25739 TableSkewCostFunction need to use aggregated deviation (#3067)" 
This reverts commit 533c84d330.
 2021-04-16 09:35:02 -07:00
Duo Zhang bf78246b4f
HBASE-25775 Use a special balancer to deal with maintenance mode (#3161) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2021-04-16 09:50:24 +08:00
clarax 533c84d330
HBASE-25739 TableSkewCostFunction need to use aggregated deviation (#3067) 
Signed-off-by: Michael Stack <stack@apache.org>
Reviewed-by: David Manning <david.manning@salesforce.com>
 2021-04-15 13:12:07 -07:00
xiaoyu 6cf4fdde61
HBASE-25776 Use Class.asSubclass to fix the warning in StochasticLoadBalancer.loadCustomCostFunctions (#3163) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-04-15 23:34:06 +05:30
Nick Dimiduk bc52bca741
HBASE-25770 Http InfoServers should honor gzip encoding when requested (#3159) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
 2021-04-15 09:07:13 -07:00
Duo Zhang 5910e9e2d1
HBASE-25767 CandidateGenerator.getRandomIterationOrder is too slow on large cluster (#3149) 
Signed-off-by: XinSun <ddupgs@gmail.com>
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-04-13 23:00:54 +08:00
Duo Zhang de012d7d1f
HBASE-25759 The master services field in LocalityBasedCostFunction is never used (#3144) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-04-12 22:27:01 +08:00
Duo Zhang f9e928e5a7
HBASE-25184 Move RegionLocationFinder to hbase-balancer (#2543) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-04-10 21:10:53 +08:00
Nick Dimiduk 5f1f8be667 HBASE-25744 Change default of `hbase.normalizer.merge.min_region_size.mb` to `0` 
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
 2021-04-09 15:00:38 -07:00
Geoffrey Jacoby 74e533d5ab
HBASE-25751 - Add writable TimeToPurgeDeletes to ScanOptions (#3137) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-04-09 13:05:47 -07:00
meiyi ad06aa2082
HBASE-25747 Remove unused getWriteAvailable method in OperationQuota (#3133) 
Signed-off-by: stack <stack@apache.org>
 2021-04-09 10:23:34 +08:00
Pankaj 6444e94c38
HBASE-25717 RegionServer aborted with due to ClassCastException (#3108) 
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-04-08 16:27:17 +05:30
stack d9f4f41f76 HBASE-25735 Add target Region to connection exceptions 
Addendum to fix broke compile.
 2021-04-07 07:56:25 -07:00
Jan Hentschel 048ca4e43f
HBASE-25174 Remove deprecated fields in HConstants (#2558) 
Remove the deprecated fields, which can be removed in 3.0.0. Marked the
constant OLDEST_TIMESTAMP as InterfaceAudience.Private as it is only use
in classes, which are also marked as InterfaceAudience.Private.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-04-03 23:12:16 +08:00
Jan Hentschel 5a63fe65aa
HBASE-25199 Remove deprecated HStore#getStoreHomedir methods (#2562) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-04-03 23:10:20 +08:00
d-c-manning 7a31557c51
HBASE-25726 MoveCostFunction is not included in the list of cost functions for StochasticLoadBalancer (#3116) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2021-04-02 09:22:13 -07:00
lujiefsi e14ec57eee
HBASE-25558:Adding audit log for execMasterService (#3101) 
Signed-off-by: stack <stack@apache.org>
 2021-03-31 16:12:31 -07:00
Toshihiro Suzuki 46f7d9dd4b
HBASE-25703 Support conditional update in MultiRowMutationEndpoint (#3098) 
Signed-off-by: Michael Stack <stack@apache.org>
 2021-03-30 09:18:56 +09:00
Josh Elser 57a49f5ca7
HBASE-25692 Always try to close the WAL reader when we catch any exception (#3090) 
There are code paths in which we throw non-IOExceptions when
initializing a WAL reader. However, we only close the InputStream to the
WAL filesystem when the exception is an IOException. Close it if it is
open in all cases.

Co-authored-by: Josh Elser <jelser@cloudera.com>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2021-03-29 12:15:58 -07:00
Michael Stack 6a8998b29c HBASE-25695 Link to the filter on hbase:meta from user tables panel on master page (#3092) 
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2021-03-27 20:36:55 -07:00
Toshihiro Suzuki 93b1163a8b
HBASE-25702 Remove RowProcessor (#3097) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-03-28 07:38:42 +09:00
caroliney14 1e4639d2eb
HBASE-25032 Wait for region server to become online before adding it to online servers in Master (#2769) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
 2021-03-25 10:39:07 -07:00
Andrew Purtell f6bb4bb93e
HBASE-25693 NPE getting metrics from standby masters (MetricsMasterWrapperImpl.getMergePlanCount) (#3091) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2021-03-24 19:09:10 -07:00
huaxiangsun 1e3fe3ceac
HBASE-25691 Test failure: TestVerifyBucketCacheFile.testRetrieveFromFile (#3081) 
The issue is that FileInputStream is created with try-with-resources, so its close() is called right after the try sentence.
FileInputStream is a finalize class, when this object is garbage collected, its close() is called again.
To avoid this double-free resources, add guard against it.

Signed-off-by: stack <stack@apache.org>
 2021-03-24 09:01:17 -07:00
XinSun 3358091b7e
HBASE-25683 Simplify UTs using DummyServer (#3069) 
Co-authored-by: sunxin <sunxin@apache.com>
Signed-off-by: stack <stack@apache.org>
 2021-03-22 08:54:09 -07:00
Duo Zhang ba3610d097
HBASE-19577 Use log4j2 instead of log4j for logging (#1708) 
Signed-off-by: stack <stack@apache.org>
 2021-03-20 09:21:25 +08:00
Baiqiang Zhao a3938c8725
HBASE-25681 Add a switch for server/table queryMeter (#3070) 
Signed-off-by: stack <stack@apache.org>
 2021-03-19 16:23:41 -07:00
shahrs87 fea4bd12e2 HBASE-25679 Size of log queue metric is incorrect (#3071) 
Co-authored-by: Rushabh <rushabh.shah@salesforce.com>
Signed-off-by: stack <stack@apache.org>
 2021-03-19 16:16:48 -07:00
Toshihiro Suzuki f4059907e2
HBASE-25678 Support nonce operations for Increment/Append in RowMutations and CheckAndMutate (#3064) 
Signed-off-by: stack <stack@apache.org>
 2021-03-19 21:25:40 +09:00
haxiaolin 585aca1f05
HBASE-25518 Support separate child regions to different region servers (#3001) 
Signed-off-by: stack <stack@apache.org>
 2021-03-18 12:38:17 -07:00
bsglz d93035a131
HBASE-25643 The delayed FlushRegionEntry should be removed when we ne… (#3049) 
Signed-off-by: AnoopSamJohn <anoopsamjohn@apache.org>
Signed-off-by: stack <stack@apache.org>
 2021-03-18 12:13:06 -07:00
Michael Stack 7ac1c8bbf8 HBASE-25677 Server+table counters on each scan #nextRaw invocation becomes a bottleneck when heavy load (#3061) 
Don't have every handler update regionserver metrics on each
scan#nextRaw; instead, do a batch update just before Scan
returns. Otherwise, all running handlers end up contending
on metrics update.

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 Update of regionserver metrics counters moved out to caller where
 can be done as a batch update instead of per-next.

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServer.java
 Class doc to encourage batch updating metrics.
 Remove the single update as unused anymore.

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
 Count calls to nextRaw. Update regionserver count in finally block when
 scan is done rather than per nextRaw call. Move all metrics updates to
 finally.

Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Baiqiang Zhao <ZhaoBQ>
 2021-03-18 11:46:37 -07:00
Sandeep Pal ff3821814a
HBASE-25627: HBase replication should have a metric to represent if the source is stuck getting initialized (#3018) 
Introduces a new metric that tracks number of replication sources that are stuck in initialization.

Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2021-03-17 09:10:44 -07:00
DivyeshChandra bcf503e6c2
HBASE-25653 Add units and round off region size to 2 digits after decimal (#3046) 
Signed-off-by: stack <stack@duboce.net>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
 2021-03-16 21:32:12 -07:00
bitterfox ebb0adf500
HBASE-25665 Option to use hostname instead of canonical hostname for secure HBase cluster connection (#3051) 2021-03-16 21:04:25 -07:00
Baiqiang Zhao db2e6d8c63
HBASE-25597 Add row info in Exception when cell size exceeds maxCellSize (#2976) 
Signed-off-by: stack <stack@apache.org>
 2021-03-15 15:49:33 -07:00
haxiaolin 0ef892b68a
HBASE-25621 Balancer should check region plan source to avoid misplace region groups (#3002) 
Signed-off-by: stack <stack@duboce.net>
 2021-03-15 14:47:27 -07:00
haxiaolin 625bea3ecc
HBASE-25595 TestLruBlockCache.testBackgroundEvictionThread is flaky (#2974) 
Signed-off-by: stack <stack@apache.org>
 2021-03-15 14:25:38 -07:00
Michael Stack 630f47e4ec HBASE-25660 Print split policy in use on Region open (as well as split policy vitals) (#3044) 
Add a toString to all split policy implementations listing name and
 vitals. Use this toString in the Region open message. Ditto for flush
 policy for the Region.

 Signed-off-by: Huaxiang Sun<huaxiangsun@apache.org>
 2021-03-15 14:12:31 -07:00
haxiaolin aeec8ca64b
HBASE-25635 CandidateGenerator may miss some region balance actions (#3024) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-03-15 21:28:22 +08:00
Duo Zhang 876fec1648
HBASE-25657 Fix spotbugs warnings after upgrading spotbugs to 4.x (#3041) 
Signed-off-by: meiyi <myimeiyi@gmail.com>
Signed-off-by: stack <stack@apache.org>
 2021-03-12 14:34:10 +08:00
shahrs87 7386fb6e1f
HBASE-25622 Result#compareResults should compare tags. (#3026) 
Signed-off-by: stack <stack@apache.org>
 2021-03-11 21:51:07 -08:00
Michael Stack 1a69a52653
HBASE-25570 On largish cluster, "CleanerChore: Could not delete dir..." makes master log unreadable (#2949) 
Turn down the amount we log. If you want to see the full exception
enable TRACE-level logging.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: shahrs87
 2021-03-11 21:35:24 -08:00
Peter Somogyi 0cc1ae48ed
HBASE-25587 [hbck2] Schedule SCP for all unknown servers (#2978) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2021-03-11 17:20:36 +01:00
meiyi 0e6c2c419c
HBASE-25636 Expose HBCK report as metrics (#3031) 
Signed-off-by: zhangduo <zhangduo@apache.org>
 2021-03-11 15:12:30 +08:00
Duo Zhang d79019ba3b
HBASE-25629 Reimplement TestCurrentHourProvider to not depend on unstable TZs (#3013) 
Signed-off-by: XinSun <ddupgs@gmail.com>
 2021-03-10 23:45:38 +08:00
Wellington Ramos Chevreuil 373dc7788d
HBASE-25548 Optionally allow snapshots to preserve cluster's max file… (#2923) 
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
 2021-03-10 10:20:10 +00:00
Anoop Sam John d818eff93a
HBASE-25582 Support setting scan ReadType to be STREAM at cluster level (#3035) 
Signed-off-by: zhangduo <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-03-10 10:18:12 +05:30
Narges Shadab 573daed419
HBASE-25646: Possible Resource Leak in CatalogJanitor #3036 
Close the stream using a try-with-resources block.

Reviewed-by: Aman Poonia <aman.poonia.29@gmail.com>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2021-03-09 09:07:02 -08:00
niuyulin 109bd24065
HBASE-25630 Set switch compaction after bulkload default as false (#3022) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-03-09 12:14:53 +08:00
Bharath Vissapragada c1dacfd577
HBASE-25547 (addendum): Roll ExecutorType into ExecutorConfig (#2996) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-03-08 15:00:18 -08:00
Duo Zhang 9b0485fb98
HBASE-23578 [UI] Master UI shows long stack traces when table is broken (#3014) 
Co-authored-by: Shuhei Yamasaki <yamasakisua@oss.nttdata.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-03-07 00:14:21 +08:00
niuyulin dd4a11e421
HBASE-25637 Rename method completeCompaction to refreshStoreSizeAndTotalBytes (#3023) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-03-06 18:41:07 +08:00
Rahul Kumar 830d2895b2
HBASE-25460 : Expose drainingServers as cluster metric (#2995) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-03-04 12:47:57 +05:30
Andrew Purtell f93c9c6f15
HBASE-25385 TestCurrentHourProvider fails if the latest timezone changes are not present (#3012) 
Disable this problematic test for now. May be removed after additional
discussion.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2021-03-03 18:12:15 -08:00
niuyulin e80b901e47
HBASE-25603 Add switch for compaction after bulkload (#2982) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2021-03-04 10:09:08 +08:00
Akshay Sudheer 5d9a6ed1fe
HBASE-25367 Sort broken after Change 'State time' in UI (#2964) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
 2021-03-03 13:57:42 +05:30
Narges Shadab e099ef349b
HBASE-25626 Possible Resource Leak in HeterogeneousRegionCountCostFunction 
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by shahrs87 <shahrs87@gmail.com>
 2021-03-03 11:45:56 +05:30
Akshay Sudheer 157200ef83
HBASE-25402 Sorting order by start key or end key is not considering empty start key/end key (#2955) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
 2021-03-03 09:38:55 +05:30
Balazs Meszaros a984358d9f
HBASE-25586 Fix HBASE-22492 on branch-2 (SASL GapToken) (#2961) 
ServerCall.java: calling wrapWithSasl() was moved to getResponse(), so
the SASL wrapping is delayed until the reply is sent back to the client.

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
 2021-02-26 14:05:59 +01:00
XinSun 8d0de96976
HBASE-25590 Bulkload replication HFileRefs cannot be cleared in some cases where set exclude-namespace/exclude-table-cfs (#2969) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2021-02-26 09:50:23 +08:00
Sandeep Pal 3f1c486ddb
HBASE-25596: Fix NPE and avoid permanent unreplicated data due to EOF (#2987) 
Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2021-02-25 13:36:11 -08:00
XinSun 51a3d45f9d
HBASE-25598 TestFromClientSide5.testScanMetrics is flaky (#2977) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-02-24 14:15:51 +08:00
Andrew Purtell 8f03c4429f
HBASE-25556 Frequent replication "Encountered a malformed edit" warnings (#2965) 
ProtobufLogReader#readNext may be called by code that attempts to advance
the reader but does not necessarily expect to succeed, for example
WALEntryStream#tryAdvanceEntry. Much of the logging in this method is
at TRACE level. Other logging at WARN level will be frequently emitted, as
often as several per minute, and this will cause false positive assessment
from operators that they are experiencing a bug. Fix the mixed intent with
respect to log levels in readNext. Log at only DEBUG level or below.

Signed-off-by: Sean Busbey <busbey@apache.org>
 2021-02-23 14:38:45 -08:00
Toshihiro Suzuki 5fa15cfde3
HBASE-25575 Should validate Puts in RowMutations (#2954) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-02-22 10:46:16 +09:00
Toshihiro Suzuki d8b86627ea
HBASE-25574 Revisit put/delete/increment/append related RegionObserver methods (#2953) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-02-22 09:37:30 +09:00
XinSun ed90a14995
HBASE-25562 ReplicationSourceWALReader log and handle exception immediately without retrying (#2943) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: shahrs87
 2021-02-20 10:20:54 +08:00
shahrs87 88057d8ab6
HBASE-25539: Add age of oldest wal metric (#2945) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2021-02-18 20:58:24 -08:00
Sandeep Pal 1beda0d598
HBASE-25541 : Setting the path to null when we dequeue the current log (#2959) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2021-02-18 11:53:52 +05:30
Bharath Vissapragada 618236dd90
HBASE-25547: Thread pools should release unused resources (#2922) 
Plumbs the configuration needed to enable core thread timeout on non-critical thread pools.
Currently only enabled for thread pools with op-codes RS_LOG_REPLAY_OPS, RS_PARALLEL_SEEK, MASTER_SNAPSHOT_OPERATIONS, MASTER_MERGE_OPERATIONS. Others can be added later as
needed.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2021-02-16 11:12:00 -08:00
pustota2009 b6649a8784
HBASE-23887 Bug fix heavyEvictionMbSizeLimit (ADDENDUM) (#2957) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-02-15 20:43:33 +05:30
pustota2009 b2f85e5fd2
HBASE-23887 AdaptiveLRU cache (#2934) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-02-13 19:57:28 +05:30
GeorryHuang 3e743df6d0
HBASE-25364 Redo the getMidPoint() in HFileWriterImpl to get rid of the double comparison process (#2741) 
Signed-off-by: niuyulin <nyl353@163.com>
Signed-off-by: stack <stack@apache.org>
 2021-02-10 08:31:07 -08:00
Baiqiang Zhao 25e3633e62
HBASE-25534 Honor TableDescriptor settings earlier in normalization (#2917) 
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2021-02-09 16:45:26 -08:00
XinSun 4a3ff98943
HBASE-25559 Terminate threads of oldsources while RS is closing (#2938) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2021-02-09 16:32:46 +08:00
haxl 0353909bc2
HBASE-25507 Leak of ESTABLISHED sockets when compaction encountered "java.io.IOException: Invalid HFile block magic" (#2882) 
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: stack <stack@apache.org>
 2021-02-08 22:14:00 -08:00
Michael Stack ae063f953e HBASE-25542 Add client detail to scan name so when lease expires, we … (#2930) 
* HBASE-25542 Add client detail to scan name so when lease expires, we have clue on who was scanning

When we create a scanner lease, record client ip and port (removed
unnecessary store of scannerName).

Signed-off-by: Clara Xiong <clarax98007@gmail.com>
 2021-02-08 14:40:31 -08:00
GeorryHuang ce9c9b453f
HBASE-25550 More readable Competition Time (#2925) 2021-02-08 13:12:24 -08:00
XinSun 355bccbd0b
HBASE-25560 Remove unused parameter named peerId in the constructor method of CatalogReplicationSourcePeer (#2939) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-02-08 12:29:05 -08:00
XinSun d6aff6cbae
HBASE-25553 It is better for ReplicationTracker.getListOfRegionServers to return ServerName instead of String (#2928) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-02-07 17:13:47 +08:00
Baiqiang Zhao 304379605a
HBASE-25554 NPE when init RegionMover (#2927) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-02-05 14:01:53 +05:30
YutSean 7905749733
HBASE-25543 When configuration hadoop.security.authorization is set to false, the system will still try to authorize an RPC and raise AccessDeniedException (#2919) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
 2021-02-05 15:37:34 +08:00
stack 3a690931f5 HBASE-25546 PerfTestCompactionPolicies is failing because of NPE (addendum2); add class comment 2021-02-04 11:27:06 -08:00
robin7roy 1e8ff16eae
HBASE-25546 PerfTestCompactionPolicies is failing because of NPE (addendum) (#2926) 
Added HBaseClassTestRule for PerfTestCompactionPolicies
 2021-02-04 07:25:40 -08:00
robin7roy d6d67d1e9f
HBASE-25546 PerfTestCompactionPolicies is failing because of NPE (#2921) 
Adds mock for getRegionInfo which was missing earlier.

Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
 2021-02-03 10:34:34 +05:30
Pankaj a04ea7ea44
HBASE-24900 Make retain assignment configurable during SCP (#2313) 
Retain assignment will be useful in non-cloud scenario where RegionServer and Datanode are deployed in same machine and will avoid remote read.

Signed-off-by: Guanghao Zhang <zghao@apache.org> 
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
 2021-02-01 10:37:57 +05:30
shahrs87 d234b4dec2
[HBASE-25536] Remove 0 length wal file from logQueue if it belongs to old sources (#2908) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Geoffrey Jacoby <gjacoby@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-01-29 17:47:30 +05:30
Bharath Vissapragada de51a40b53
HBASE-25528: Dedicated merge dispatch threadpool on master (#2904) 
Adds "hbase.master.executor.merge.dispatch.threads" and defaults to 2.

Also adds additional logging that includes the number of split plans
and merge plans computed for each normalizer run.

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-01-28 09:33:35 -08:00
Bo Cui 4aff481318
HBASE-25506 ServerManager#startChore affects MTTR of HMaster (#2889) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-01-28 22:55:05 +08:00
Aman Poonia 4a66f68f9c
HBASE-25523 Region normalizer chore thread is getting killed (#2903) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-01-26 11:55:12 +05:30
Baiqiang Zhao 0b0cdcac53
HBASE-25522 Remove deprecated methods in ReplicationPeerConfig (#2898) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-01-24 21:07:03 +05:30
huaxiangsun 5c13c68494
HBASE-25368 Filter out more invalid encoded name in isEncodedRegionNa… (#2868) 
HBASE-25368 Filter out more invalid encoded name in isEncodedRegionName(byte[] regionName)

Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-01-20 09:04:50 -08:00
Duo Zhang a37e727990
HBASE-25509 ChoreService.cancelChore will not call ScheduledChore.cle… (#2890) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-01-20 16:10:36 +08:00
Baiqiang Zhao 871eb09b3d
HBASE-25482 Improve SimpleRegionNormalizer#getAverageRegionSizeMb (#2858) 
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2021-01-19 11:09:50 -08:00
Baiqiang Zhao bc4f5c2709
HBASE-25513 When the table is turned on normalize, the first region may not be merged even the size is 0 (#2887) 
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2021-01-19 07:53:51 -08:00
Anjan Das d5debe15b5
HBASE-25475 : Unset zk based wal splitting explicitly in tests (ADDENDUM) (#2891) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-01-17 16:31:07 +05:30
Christine Feng 643548f5f5
HBASE-25478 - Implement retries when enabling tables in TestRegionReplicaReplicationEndpoint (#2866) 
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-01-16 13:33:13 +05:30
Anjan Das 1474ebcc69
HBASE-25475: Improve UT added as part of HBASE-25445 in TestSplitWALManager (#2855) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2021-01-15 12:18:12 +05:30
Baiqiang Zhao 9b670a489b
HBASE-25431 MAX_FILESIZE and MEMSTORE_FLUSHSIZE should not be set negative number (#2803) 
Signed-off-by: stack <stack@apache.org>
 2021-01-14 22:00:50 -08:00
caroliney14 dfefff7e59
HBASE-25329 Dump ritsOverThreshold in logs (#2761) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-01-13 12:10:12 +05:30