Commit Graph

8920 Commits

Author SHA1 Message Date
Duo Zhang 7640134e3e
HBASE-25774 Added more detailed logs about the restarting of region servers (#3213)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-05-03 20:33:33 +08:00
GeorryHuang 00fec24c90
HBASE-25790 NamedQueue 'BalancerRejection' for recent history of balancer skipping (#3182)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-05-02 21:30:48 +05:30
Duo Zhang 73a82bd7c6
HBASE-25825 RSGroupBasedLoadBalancer.onConfigurationChange should chain the request to internal balancer (#3209)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-04-30 22:45:33 +08:00
Duo Zhang 6c65314cdf
HBASE-25819 Fix style issues for StochasticLoadBalancer (#3207)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-04-29 11:03:55 +08:00
Nick Dimiduk b061b0c4ed HBASE-25779 HRegionServer#compactSplitThread should be private
Minor refactor. Make the `compactSplitThread` member field of `HRegionServer` private, and gate
all access through the getter method.

Signed-off-by: Yulin Niu <niuyulin@apache.org>
Signed-off-by: Pankaj Kumar <pankajkumar@apache.org>
2021-04-28 16:46:36 -07:00
Michael Stack 2382f68b23
HBASE-25792 Filter out o.a.hadoop.thirdparty building shaded jars (#3184)
Need to add to allowed-licenses list too....

Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Reviewed-by: Duo Zhang <zhangduo@apache.org>
Reviewed-by: Nick Dimiduk <ndimiduk@apache.org>
2021-04-27 08:37:25 -07:00
Duo Zhang 8856f61986 HBASE-25757 Addendum remove CandidateGenerator classes under hbase-server module 2021-04-27 23:25:51 +08:00
Duo Zhang a4d954e606
HBASE-25757 Move BaseLoadBalancer to hbase-balancer module (#3191)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-04-26 12:03:25 +08:00
Duo Zhang 7f90c2201f HBASE-25723 Temporarily remove the trace support for RegionScanner.next (#3119)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-04-25 09:23:23 +08:00
Duo Zhang f6ff519dd0 HBASE-25591 Upgrade opentelemetry to 0.17.1 (#2971)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2021-04-25 09:23:23 +08:00
Duo Zhang bb8c4967f8 HBASE-25535 Set span kind to CLIENT in AbstractRpcClient (#2907)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2021-04-25 09:23:23 +08:00
Duo Zhang 2be2c63f0d HBASE-25484 Add trace support for WAL sync (#2892)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2021-04-25 09:23:23 +08:00
Duo Zhang 03e12bfa4a HBASE-25455 Add trace support for HRegion read/write operation (#2861)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2021-04-25 09:23:23 +08:00
Duo Zhang ae2c62ffaa HBASE-25481 Add host and port attribute when tracing rpc call at client side (#2857)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2021-04-25 09:23:23 +08:00
Duo Zhang 805b2ae2ad HBASE-23898 Add trace support for simple apis in async client (#2813)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2021-04-25 09:23:23 +08:00
Duo Zhang 2420286715 HBASE-25401 Add trace support for async call in rpc client (#2790)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2021-04-25 09:23:23 +08:00
Duo Zhang 302d9ea8b8 HBASE-25373 Remove HTrace completely in code base and try to make use of OpenTelemetry
Signed-off-by: stack <stack@apache.org>
2021-04-25 09:23:23 +08:00
Andrew Purtell 9895b2dfdf
HBASE-25756 Support alternate compression for major and minor compactions (#3142)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-04-23 15:45:26 -07:00
Duo Zhang 96fefce9c3
HBASE-25802 Miscellaneous style improvements for load balancer related classes (#3192)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-04-23 15:20:27 +08:00
haxiaolin 996862c1cc
HBASE-25754 StripeCompactionPolicy should support compacting cold regions (#3152)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-04-23 14:58:53 +08:00
Toshihiro Suzuki 5f4e2e111b
HBASE-25766 Introduce RegionSplitRestriction that restricts the pattern of the split point (#3150)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2021-04-22 13:53:36 +09:00
Duo Zhang 50920ee306
HBASE-25774 TestSyncReplicationStandbyKillRS#testStandbyKillRegionServer is flaky (#3189)
Wait for the restarter thread to finish before checking the state
Add more detailed logs

Signed-off-by: meiyi <myimeiyi@gmail.com>
2021-04-22 10:10:15 +08:00
Duo Zhang d5c5e48839
HBASE-25793 Move BaseLoadBalancer.Cluster to a separated file (#3185)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-04-22 09:59:49 +08:00
haxiaolin 0d257baf29
HBASE-25763 TestRSGroupsWithACL.setupBeforeClass is flaky (#3158)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-04-21 14:41:51 +08:00
Duo Zhang 781da1899a
HBASE-25290 Remove table on master related code in balancer implementation (#3162)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-04-20 21:31:09 +08:00
Nick Dimiduk b65890da1d Revert "HBASE-25739 TableSkewCostFunction need to use aggregated deviation (#3067)"
This reverts commit 533c84d330.
2021-04-16 09:35:02 -07:00
Duo Zhang bf78246b4f
HBASE-25775 Use a special balancer to deal with maintenance mode (#3161)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2021-04-16 09:50:24 +08:00
clarax 533c84d330
HBASE-25739 TableSkewCostFunction need to use aggregated deviation (#3067)
Signed-off-by: Michael Stack <stack@apache.org>
Reviewed-by: David Manning <david.manning@salesforce.com>
2021-04-15 13:12:07 -07:00
xiaoyu 6cf4fdde61
HBASE-25776 Use Class.asSubclass to fix the warning in StochasticLoadBalancer.loadCustomCostFunctions (#3163)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-04-15 23:34:06 +05:30
Nick Dimiduk bc52bca741
HBASE-25770 Http InfoServers should honor gzip encoding when requested (#3159)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2021-04-15 09:07:13 -07:00
Duo Zhang 5910e9e2d1
HBASE-25767 CandidateGenerator.getRandomIterationOrder is too slow on large cluster (#3149)
Signed-off-by: XinSun <ddupgs@gmail.com>
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-04-13 23:00:54 +08:00
Duo Zhang de012d7d1f
HBASE-25759 The master services field in LocalityBasedCostFunction is never used (#3144)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-04-12 22:27:01 +08:00
Duo Zhang f9e928e5a7
HBASE-25184 Move RegionLocationFinder to hbase-balancer (#2543)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-04-10 21:10:53 +08:00
Nick Dimiduk 5f1f8be667 HBASE-25744 Change default of `hbase.normalizer.merge.min_region_size.mb` to `0`
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
2021-04-09 15:00:38 -07:00
Geoffrey Jacoby 74e533d5ab
HBASE-25751 - Add writable TimeToPurgeDeletes to ScanOptions (#3137)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-04-09 13:05:47 -07:00
meiyi ad06aa2082
HBASE-25747 Remove unused getWriteAvailable method in OperationQuota (#3133)
Signed-off-by: stack <stack@apache.org>
2021-04-09 10:23:34 +08:00
Pankaj 6444e94c38
HBASE-25717 RegionServer aborted with due to ClassCastException (#3108)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-04-08 16:27:17 +05:30
stack d9f4f41f76 HBASE-25735 Add target Region to connection exceptions
Addendum to fix broke compile.
2021-04-07 07:56:25 -07:00
Jan Hentschel 048ca4e43f
HBASE-25174 Remove deprecated fields in HConstants (#2558)
Remove the deprecated fields, which can be removed in 3.0.0. Marked the
constant OLDEST_TIMESTAMP as InterfaceAudience.Private as it is only use
in classes, which are also marked as InterfaceAudience.Private.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-04-03 23:12:16 +08:00
Jan Hentschel 5a63fe65aa
HBASE-25199 Remove deprecated HStore#getStoreHomedir methods (#2562)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-04-03 23:10:20 +08:00
d-c-manning 7a31557c51
HBASE-25726 MoveCostFunction is not included in the list of cost functions for StochasticLoadBalancer (#3116)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-04-02 09:22:13 -07:00
lujiefsi e14ec57eee
HBASE-25558:Adding audit log for execMasterService (#3101)
Signed-off-by: stack <stack@apache.org>
2021-03-31 16:12:31 -07:00
Toshihiro Suzuki 46f7d9dd4b
HBASE-25703 Support conditional update in MultiRowMutationEndpoint (#3098)
Signed-off-by: Michael Stack <stack@apache.org>
2021-03-30 09:18:56 +09:00
Josh Elser 57a49f5ca7
HBASE-25692 Always try to close the WAL reader when we catch any exception (#3090)
There are code paths in which we throw non-IOExceptions when
initializing a WAL reader. However, we only close the InputStream to the
WAL filesystem when the exception is an IOException. Close it if it is
open in all cases.

Co-authored-by: Josh Elser <jelser@cloudera.com>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-03-29 12:15:58 -07:00
Michael Stack 6a8998b29c HBASE-25695 Link to the filter on hbase:meta from user tables panel on master page (#3092)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2021-03-27 20:36:55 -07:00
Toshihiro Suzuki 93b1163a8b
HBASE-25702 Remove RowProcessor (#3097)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-03-28 07:38:42 +09:00
caroliney14 1e4639d2eb
HBASE-25032 Wait for region server to become online before adding it to online servers in Master (#2769)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2021-03-25 10:39:07 -07:00
Andrew Purtell f6bb4bb93e
HBASE-25693 NPE getting metrics from standby masters (MetricsMasterWrapperImpl.getMergePlanCount) (#3091)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2021-03-24 19:09:10 -07:00
huaxiangsun 1e3fe3ceac
HBASE-25691 Test failure: TestVerifyBucketCacheFile.testRetrieveFromFile (#3081)
The issue is that FileInputStream is created with try-with-resources, so its close() is called right after the try sentence.
FileInputStream is a finalize class, when this object is garbage collected, its close() is called again.
To avoid this double-free resources, add guard against it.

Signed-off-by: stack <stack@apache.org>
2021-03-24 09:01:17 -07:00
XinSun 3358091b7e
HBASE-25683 Simplify UTs using DummyServer (#3069)
Co-authored-by: sunxin <sunxin@apache.com>
Signed-off-by: stack <stack@apache.org>
2021-03-22 08:54:09 -07:00
Duo Zhang ba3610d097
HBASE-19577 Use log4j2 instead of log4j for logging (#1708)
Signed-off-by: stack <stack@apache.org>
2021-03-20 09:21:25 +08:00
Baiqiang Zhao a3938c8725
HBASE-25681 Add a switch for server/table queryMeter (#3070)
Signed-off-by: stack <stack@apache.org>
2021-03-19 16:23:41 -07:00
shahrs87 fea4bd12e2 HBASE-25679 Size of log queue metric is incorrect (#3071)
Co-authored-by: Rushabh <rushabh.shah@salesforce.com>
Signed-off-by: stack <stack@apache.org>
2021-03-19 16:16:48 -07:00
Toshihiro Suzuki f4059907e2
HBASE-25678 Support nonce operations for Increment/Append in RowMutations and CheckAndMutate (#3064)
Signed-off-by: stack <stack@apache.org>
2021-03-19 21:25:40 +09:00
haxiaolin 585aca1f05
HBASE-25518 Support separate child regions to different region servers (#3001)
Signed-off-by: stack <stack@apache.org>
2021-03-18 12:38:17 -07:00
bsglz d93035a131
HBASE-25643 The delayed FlushRegionEntry should be removed when we ne… (#3049)
Signed-off-by: AnoopSamJohn <anoopsamjohn@apache.org>
Signed-off-by: stack <stack@apache.org>
2021-03-18 12:13:06 -07:00
Michael Stack 7ac1c8bbf8 HBASE-25677 Server+table counters on each scan #nextRaw invocation becomes a bottleneck when heavy load (#3061)
Don't have every handler update regionserver metrics on each
scan#nextRaw; instead, do a batch update just before Scan
returns. Otherwise, all running handlers end up contending
on metrics update.

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 Update of regionserver metrics counters moved out to caller where
 can be done as a batch update instead of per-next.

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServer.java
 Class doc to encourage batch updating metrics.
 Remove the single update as unused anymore.

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
 Count calls to nextRaw. Update regionserver count in finally block when
 scan is done rather than per nextRaw call. Move all metrics updates to
 finally.

Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Baiqiang Zhao <ZhaoBQ>
2021-03-18 11:46:37 -07:00
Sandeep Pal ff3821814a
HBASE-25627: HBase replication should have a metric to represent if the source is stuck getting initialized (#3018)
Introduces a new metric that tracks number of replication sources that are stuck in initialization.

Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2021-03-17 09:10:44 -07:00
DivyeshChandra bcf503e6c2
HBASE-25653 Add units and round off region size to 2 digits after decimal (#3046)
Signed-off-by: stack <stack@duboce.net>
Reviewed-by: Viraj Jasani <vjasani@apache.org>
2021-03-16 21:32:12 -07:00
bitterfox ebb0adf500
HBASE-25665 Option to use hostname instead of canonical hostname for secure HBase cluster connection (#3051) 2021-03-16 21:04:25 -07:00
Baiqiang Zhao db2e6d8c63
HBASE-25597 Add row info in Exception when cell size exceeds maxCellSize (#2976)
Signed-off-by: stack <stack@apache.org>
2021-03-15 15:49:33 -07:00
haxiaolin 0ef892b68a
HBASE-25621 Balancer should check region plan source to avoid misplace region groups (#3002)
Signed-off-by: stack <stack@duboce.net>
2021-03-15 14:47:27 -07:00
haxiaolin 625bea3ecc
HBASE-25595 TestLruBlockCache.testBackgroundEvictionThread is flaky (#2974)
Signed-off-by: stack <stack@apache.org>
2021-03-15 14:25:38 -07:00
Michael Stack 630f47e4ec HBASE-25660 Print split policy in use on Region open (as well as split policy vitals) (#3044)
Add a toString to all split policy implementations listing name and
 vitals. Use this toString in the Region open message. Ditto for flush
 policy for the Region.

 Signed-off-by: Huaxiang Sun<huaxiangsun@apache.org>
2021-03-15 14:12:31 -07:00
haxiaolin aeec8ca64b
HBASE-25635 CandidateGenerator may miss some region balance actions (#3024)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-03-15 21:28:22 +08:00
Duo Zhang 876fec1648
HBASE-25657 Fix spotbugs warnings after upgrading spotbugs to 4.x (#3041)
Signed-off-by: meiyi <myimeiyi@gmail.com>
Signed-off-by: stack <stack@apache.org>
2021-03-12 14:34:10 +08:00
shahrs87 7386fb6e1f
HBASE-25622 Result#compareResults should compare tags. (#3026)
Signed-off-by: stack <stack@apache.org>
2021-03-11 21:51:07 -08:00
Michael Stack 1a69a52653
HBASE-25570 On largish cluster, "CleanerChore: Could not delete dir..." makes master log unreadable (#2949)
Turn down the amount we log. If you want to see the full exception
enable TRACE-level logging.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: shahrs87
2021-03-11 21:35:24 -08:00
Peter Somogyi 0cc1ae48ed
HBASE-25587 [hbck2] Schedule SCP for all unknown servers (#2978)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2021-03-11 17:20:36 +01:00
meiyi 0e6c2c419c
HBASE-25636 Expose HBCK report as metrics (#3031)
Signed-off-by: zhangduo <zhangduo@apache.org>
2021-03-11 15:12:30 +08:00
Duo Zhang d79019ba3b
HBASE-25629 Reimplement TestCurrentHourProvider to not depend on unstable TZs (#3013)
Signed-off-by: XinSun <ddupgs@gmail.com>
2021-03-10 23:45:38 +08:00
Wellington Ramos Chevreuil 373dc7788d
HBASE-25548 Optionally allow snapshots to preserve cluster's max file… (#2923)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2021-03-10 10:20:10 +00:00
Anoop Sam John d818eff93a
HBASE-25582 Support setting scan ReadType to be STREAM at cluster level (#3035)
Signed-off-by: zhangduo <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-03-10 10:18:12 +05:30
Narges Shadab 573daed419
HBASE-25646: Possible Resource Leak in CatalogJanitor #3036
Close the stream using a try-with-resources block.

Reviewed-by: Aman Poonia <aman.poonia.29@gmail.com>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2021-03-09 09:07:02 -08:00
niuyulin 109bd24065
HBASE-25630 Set switch compaction after bulkload default as false (#3022)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-03-09 12:14:53 +08:00
Bharath Vissapragada c1dacfd577
HBASE-25547 (addendum): Roll ExecutorType into ExecutorConfig (#2996)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-03-08 15:00:18 -08:00
Duo Zhang 9b0485fb98
HBASE-23578 [UI] Master UI shows long stack traces when table is broken (#3014)
Co-authored-by: Shuhei Yamasaki <yamasakisua@oss.nttdata.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-03-07 00:14:21 +08:00
niuyulin dd4a11e421
HBASE-25637 Rename method completeCompaction to refreshStoreSizeAndTotalBytes (#3023)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-03-06 18:41:07 +08:00
Rahul Kumar 830d2895b2
HBASE-25460 : Expose drainingServers as cluster metric (#2995)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-03-04 12:47:57 +05:30
Andrew Purtell f93c9c6f15
HBASE-25385 TestCurrentHourProvider fails if the latest timezone changes are not present (#3012)
Disable this problematic test for now. May be removed after additional
discussion.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-03-03 18:12:15 -08:00
niuyulin e80b901e47
HBASE-25603 Add switch for compaction after bulkload (#2982)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2021-03-04 10:09:08 +08:00
Akshay Sudheer 5d9a6ed1fe
HBASE-25367 Sort broken after Change 'State time' in UI (#2964)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
2021-03-03 13:57:42 +05:30
Narges Shadab e099ef349b
HBASE-25626 Possible Resource Leak in HeterogeneousRegionCountCostFunction
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by shahrs87 <shahrs87@gmail.com>
2021-03-03 11:45:56 +05:30
Akshay Sudheer 157200ef83
HBASE-25402 Sorting order by start key or end key is not considering empty start key/end key (#2955)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
2021-03-03 09:38:55 +05:30
Balazs Meszaros a984358d9f
HBASE-25586 Fix HBASE-22492 on branch-2 (SASL GapToken) (#2961)
ServerCall.java: calling wrapWithSasl() was moved to getResponse(), so
the SASL wrapping is delayed until the reply is sent back to the client.

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2021-02-26 14:05:59 +01:00
XinSun 8d0de96976
HBASE-25590 Bulkload replication HFileRefs cannot be cleared in some cases where set exclude-namespace/exclude-table-cfs (#2969)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2021-02-26 09:50:23 +08:00
Sandeep Pal 3f1c486ddb
HBASE-25596: Fix NPE and avoid permanent unreplicated data due to EOF (#2987)
Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-02-25 13:36:11 -08:00
XinSun 51a3d45f9d
HBASE-25598 TestFromClientSide5.testScanMetrics is flaky (#2977)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-02-24 14:15:51 +08:00
Andrew Purtell 8f03c4429f
HBASE-25556 Frequent replication "Encountered a malformed edit" warnings (#2965)
ProtobufLogReader#readNext may be called by code that attempts to advance
the reader but does not necessarily expect to succeed, for example
WALEntryStream#tryAdvanceEntry. Much of the logging in this method is
at TRACE level. Other logging at WARN level will be frequently emitted, as
often as several per minute, and this will cause false positive assessment
from operators that they are experiencing a bug. Fix the mixed intent with
respect to log levels in readNext. Log at only DEBUG level or below.

Signed-off-by: Sean Busbey <busbey@apache.org>
2021-02-23 14:38:45 -08:00
Toshihiro Suzuki 5fa15cfde3
HBASE-25575 Should validate Puts in RowMutations (#2954)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-02-22 10:46:16 +09:00
Toshihiro Suzuki d8b86627ea
HBASE-25574 Revisit put/delete/increment/append related RegionObserver methods (#2953)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-02-22 09:37:30 +09:00
XinSun ed90a14995
HBASE-25562 ReplicationSourceWALReader log and handle exception immediately without retrying (#2943)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: shahrs87
2021-02-20 10:20:54 +08:00
shahrs87 88057d8ab6
HBASE-25539: Add age of oldest wal metric (#2945)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2021-02-18 20:58:24 -08:00
Sandeep Pal 1beda0d598
HBASE-25541 : Setting the path to null when we dequeue the current log (#2959)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2021-02-18 11:53:52 +05:30
Bharath Vissapragada 618236dd90
HBASE-25547: Thread pools should release unused resources (#2922)
Plumbs the configuration needed to enable core thread timeout on non-critical thread pools.
Currently only enabled for thread pools with op-codes RS_LOG_REPLAY_OPS, RS_PARALLEL_SEEK, MASTER_SNAPSHOT_OPERATIONS, MASTER_MERGE_OPERATIONS. Others can be added later as
needed.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2021-02-16 11:12:00 -08:00
pustota2009 b6649a8784
HBASE-23887 Bug fix heavyEvictionMbSizeLimit (ADDENDUM) (#2957)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-02-15 20:43:33 +05:30
pustota2009 b2f85e5fd2
HBASE-23887 AdaptiveLRU cache (#2934)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-02-13 19:57:28 +05:30
GeorryHuang 3e743df6d0
HBASE-25364 Redo the getMidPoint() in HFileWriterImpl to get rid of the double comparison process (#2741)
Signed-off-by: niuyulin <nyl353@163.com>
Signed-off-by: stack <stack@apache.org>
2021-02-10 08:31:07 -08:00
Baiqiang Zhao 25e3633e62
HBASE-25534 Honor TableDescriptor settings earlier in normalization (#2917)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2021-02-09 16:45:26 -08:00
XinSun 4a3ff98943
HBASE-25559 Terminate threads of oldsources while RS is closing (#2938)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2021-02-09 16:32:46 +08:00
haxl 0353909bc2
HBASE-25507 Leak of ESTABLISHED sockets when compaction encountered "java.io.IOException: Invalid HFile block magic" (#2882)
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: stack <stack@apache.org>
2021-02-08 22:14:00 -08:00
Michael Stack ae063f953e HBASE-25542 Add client detail to scan name so when lease expires, we … (#2930)
* HBASE-25542 Add client detail to scan name so when lease expires, we have clue on who was scanning

When we create a scanner lease, record client ip and port (removed
unnecessary store of scannerName).

Signed-off-by: Clara Xiong <clarax98007@gmail.com>
2021-02-08 14:40:31 -08:00
GeorryHuang ce9c9b453f
HBASE-25550 More readable Competition Time (#2925) 2021-02-08 13:12:24 -08:00
XinSun 355bccbd0b
HBASE-25560 Remove unused parameter named peerId in the constructor method of CatalogReplicationSourcePeer (#2939)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-02-08 12:29:05 -08:00
XinSun d6aff6cbae
HBASE-25553 It is better for ReplicationTracker.getListOfRegionServers to return ServerName instead of String (#2928)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-02-07 17:13:47 +08:00
Baiqiang Zhao 304379605a
HBASE-25554 NPE when init RegionMover (#2927)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-02-05 14:01:53 +05:30
YutSean 7905749733
HBASE-25543 When configuration hadoop.security.authorization is set to false, the system will still try to authorize an RPC and raise AccessDeniedException (#2919)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
2021-02-05 15:37:34 +08:00
stack 3a690931f5 HBASE-25546 PerfTestCompactionPolicies is failing because of NPE (addendum2); add class comment 2021-02-04 11:27:06 -08:00
robin7roy 1e8ff16eae
HBASE-25546 PerfTestCompactionPolicies is failing because of NPE (addendum) (#2926)
Added HBaseClassTestRule for PerfTestCompactionPolicies
2021-02-04 07:25:40 -08:00
robin7roy d6d67d1e9f
HBASE-25546 PerfTestCompactionPolicies is failing because of NPE (#2921)
Adds mock for getRegionInfo which was missing earlier.

Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
2021-02-03 10:34:34 +05:30
Pankaj a04ea7ea44
HBASE-24900 Make retain assignment configurable during SCP (#2313)
Retain assignment will be useful in non-cloud scenario where RegionServer and Datanode are deployed in same machine and will avoid remote read.

Signed-off-by: Guanghao Zhang <zghao@apache.org> 
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2021-02-01 10:37:57 +05:30
shahrs87 d234b4dec2
[HBASE-25536] Remove 0 length wal file from logQueue if it belongs to old sources (#2908)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Geoffrey Jacoby <gjacoby@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-01-29 17:47:30 +05:30
Bharath Vissapragada de51a40b53
HBASE-25528: Dedicated merge dispatch threadpool on master (#2904)
Adds "hbase.master.executor.merge.dispatch.threads" and defaults to 2.

Also adds additional logging that includes the number of split plans
and merge plans computed for each normalizer run.

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-01-28 09:33:35 -08:00
Bo Cui 4aff481318
HBASE-25506 ServerManager#startChore affects MTTR of HMaster (#2889)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-01-28 22:55:05 +08:00
Aman Poonia 4a66f68f9c
HBASE-25523 Region normalizer chore thread is getting killed (#2903)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-01-26 11:55:12 +05:30
Baiqiang Zhao 0b0cdcac53
HBASE-25522 Remove deprecated methods in ReplicationPeerConfig (#2898)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-01-24 21:07:03 +05:30
huaxiangsun 5c13c68494
HBASE-25368 Filter out more invalid encoded name in isEncodedRegionNa… (#2868)
HBASE-25368 Filter out more invalid encoded name in isEncodedRegionName(byte[] regionName)

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-01-20 09:04:50 -08:00
Duo Zhang a37e727990
HBASE-25509 ChoreService.cancelChore will not call ScheduledChore.cle… (#2890)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-01-20 16:10:36 +08:00
Baiqiang Zhao 871eb09b3d
HBASE-25482 Improve SimpleRegionNormalizer#getAverageRegionSizeMb (#2858)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2021-01-19 11:09:50 -08:00
Baiqiang Zhao bc4f5c2709
HBASE-25513 When the table is turned on normalize, the first region may not be merged even the size is 0 (#2887)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2021-01-19 07:53:51 -08:00
Anjan Das d5debe15b5
HBASE-25475 : Unset zk based wal splitting explicitly in tests (ADDENDUM) (#2891)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-01-17 16:31:07 +05:30
Christine Feng 643548f5f5
HBASE-25478 - Implement retries when enabling tables in TestRegionReplicaReplicationEndpoint (#2866)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-01-16 13:33:13 +05:30
Anjan Das 1474ebcc69
HBASE-25475: Improve UT added as part of HBASE-25445 in TestSplitWALManager (#2855)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2021-01-15 12:18:12 +05:30
Baiqiang Zhao 9b670a489b
HBASE-25431 MAX_FILESIZE and MEMSTORE_FLUSHSIZE should not be set negative number (#2803)
Signed-off-by: stack <stack@apache.org>
2021-01-14 22:00:50 -08:00
caroliney14 dfefff7e59
HBASE-25329 Dump ritsOverThreshold in logs (#2761)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-01-13 12:10:12 +05:30
Viraj Jasani a3913a39c3
HBASE-25211 : Rack awareness in RegionMover (#2795)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-01-12 15:31:26 -08:00
Tak Lon (Stephen) Wu 686b72c44e
HBASE-25249 Adding StoreContext (#2800)
Adding StoreContext which contains the metadata about the HStore. This
metadata can be used across the HFileWriter/Readers and other HStore
consumers without the need of passing around the complete store and
exposing its internals.

Co-authored-by: Abhishek Khanna <akkhanna@amazon.com>

Signed-off-by: stack <stack@apache.org>
Signed-off-by: Zach York <zyork@apache.org>
2021-01-08 20:16:45 -08:00
xijiawen 5c233e9785
HBASE-25459 WAL can't be cleaned in some scenes (#2848)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2021-01-08 16:00:15 +00:00
Bo Cui a348204d4a
HBASE-25483 set the loadMeta log level to debug (#2859)
Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
2021-01-08 19:27:23 +05:30
xijiawen 7d70141b90
HBASE-25434 SlowDelete & SlowPut metric value should use updateDelete & updatePut (#2837)
Signed-off-by: Zheng Wang <wangzheng@apache.org>
2021-01-08 18:58:30 +08:00
Duo Zhang 45ef88dc8d
HBASE-25471 Move RegionScannerImpl out of HRegion (#2853)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2021-01-08 17:50:08 +08:00
Anjan Das 4b6215297d
HBASE-25445: Use WAL FS instead of master FS in SplitWALManager (#2844)
Signed-off-by: Pankaj <pankajkumar@apache.org>
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-01-07 15:31:50 +05:30
mokai a414361ed9
HBASE-24755 [LOG][RSGroup]Error message is confusing while adding a offline RS to rsgroup (#2846)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2021-01-06 19:57:58 +05:30
Duo Zhang bedb45d4ac
HBASE-25458 HRegion methods cleanup (#2838)
Signed-off-by: meiyi <myimeiyi@gmail.com>
2021-01-06 15:13:10 +08:00
Wellington Ramos Chevreuil 85842634e5
HBASE-24813 ReplicationSource should clear buffer usage on Replicatio… (#2546)
Signed-off-by: Ankit Singhal <ankit@apache.org>
2021-01-05 10:21:26 +00:00
Bo Cui 600be60a4b
HBASE-25447 remoteProc is suspended due to OOM ERROR (#2824)
Some OMME can not cause the JVM to exit, like "java.lang.OutOfMemoryError: Direct buffer memory", "java.lang.OutOfMemoryError: unable to create new native thread", as they dont call vmError#next_OnError_command. So abort HMaster when uncaught exception occurs in TimeoutExecutor, the new active Hmaster will resume the suspended procedure.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.com>
Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
2021-01-04 21:04:38 +05:30
申胜利 eef076828a
Shenshengli hbase-25450 The parameter "hbase.bucketcache.size" is misdescribed (#2821)
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: stack <stack@apache.org>
2021-01-03 09:27:08 -08:00
GeorryHuang 8b9a074bf7
HBASE-24751 Display Task completion time and/or processing duration on Web UI (#2815)
Signed-off-by: stack <stack@apache.org>
2021-01-02 11:22:36 -08:00
lujiefsi 126d01dae3
HBASE-25456 : add security check for setRegionStateInMeta (#2833)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-01-01 12:17:34 +05:30
leyangyueshan 043da5f5ee
HBASE-25435 Slow metric value can be configured (#2823)
* HBASE-25435 Slow metric value can be configured

* fix  HBASE-25435

* hbase-25435 add blank

* fix hbase-25435-2

Co-authored-by: stevenxi <stevenxi@tencent.com>
2020-12-31 22:37:31 +08:00
Duo Zhang fbf00f9c28
HBASE-25451 Upgrade commons-io to 2.8.0 (#2825)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-12-31 16:57:27 +08:00
lujiefsi 29a9a16de4
HBASE-25441 : add security check for some APIs in RSRpcServices (#2810)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-31 00:22:26 +05:30
Duo Zhang 55a4eca9e6
HBASE-25452 Use MatcherAssert.assertThat instead of org.junit.Assert.assertThat (#2826)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-12-30 09:47:56 +08:00
Pankaj c96fbf0407
HBASE-25379 Make retry pause time configurable for regionserver short operation RPC (reportRegionStateTransition/reportProcedureDone) (#2757)
* HBASE-25379 Make retry pause time configurable for regionserver short operation RPC (reportRegionStateTransition/reportProcedureDone)
* HBASE-25379 RemoteProcedureResultReporter also should retry after the configured pause time
* Addressed the review comments

Signed-off-by: Yulin Niu <niuyulin@apache.org>
2020-12-29 22:25:36 +05:30
lujiefsi d963342f8a
HBASE-25432:add security checks for setTableStateInMeta and fixMeta (#2809)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-28 10:57:30 -08:00
ramkrish86 140c7f6ea0
HBASE-24850 CellComparator perf improvement (#2802)
* Using ContiguousCellFormat as a marker alone

* Commit the new file

* Fix the comparator logic that was an oversight

* Fix the sequenceId check order

* Adding few more static methods that helps in scan flow like query
matcher where we have more cols

* Remove ContiguousCellFormat and ensure compare() can be inlined

* applying negation as per review comment

* Fix checkstyle comments

* fix review comments

* Address review comments

* Fix the checkstyle issues

* Fix javadoc

Signed-off-by: stack <stack@apache.org>
Signed-off-by: AnoopSamJohn <anoopsamjohn@apache.org>
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
2020-12-28 13:02:06 +05:30
Huang Zhuoyue 0f868da05d HBASE-25443 Improve the experience of using the Master webpage by change the loading process of snapshot list to asynchronous 2020-12-25 10:09:32 +08:00
Mohammad Arshad dcb38f47db
HBASE-25371: When openRegion fails during initial verification(before… (#2785)
Signed-off-by: stack <stack@apache.org>
2020-12-21 08:11:22 -08:00
scotthunt 904b555edc HBASE-25084 Add "regexstringnocase" to ParseFilter for case-insensitivity (#2784)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-12-21 08:44:33 +00:00
Wellington Chevreuil d13010fa26 Revert "[PATCH] Add "regexstringnocase" to ParseFilter for case-insensitivity (#2784)"
Will re-commit with the proper jira ID

This reverts commit 51cee0016f.
2020-12-21 08:41:52 +00:00
scotthunt 51cee0016f
[PATCH] Add "regexstringnocase" to ParseFilter for case-insensitivity (#2784)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-12-21 08:39:13 +00:00
Duo Zhang 772350b26e
HBASE-25419 Remove deprecated methods in RpcServer implementation (#2791)
Signed-off-by: XinSun <ddupgs@gmail.com>
Signed-off-by: stack <stack@apache.com>
2020-12-20 12:00:07 +08:00
Duo Zhang 1540b89cee
HBASE-25420 Some minor improvements in rpc implementation (#2792)
Signed-off-by: XinSun <ddupgs@gmail.com>
Signed-off-by: stack <stack@apache.com>
2020-12-20 11:26:36 +08:00
Akshay Sudheer 7b1e9cd0d2
HBASE-25404 Procedures table Id under master web UI gets word break to single character (#2783)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-12-19 12:37:21 -08:00
Duo Zhang 33441f540a Revert "HBASE-25368 Filter out more invalid encoded name in isEncodedRegionName(byte[] regionName) (#2753)"
This reverts commit c327680125.
2020-12-19 20:28:30 +08:00
Sandeep Pal f600856a3b
HBASE-25383: Ability to update and remove peer base config
Closes #2778

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Geoffrey Jacoby <gjacoby@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-18 14:19:17 +05:30
stack 888a4fb86a HBASE-25400 [Flakey Tests] branch-2 TestRegionMoveAndAbandon 2020-12-16 10:39:11 -08:00
shahrs87 74d68180e6
HBASE-25246 Backup/Restore hbase cell tags
Closes #2745

Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-16 18:52:36 +05:30
huaxiangsun c327680125
HBASE-25368 Filter out more invalid encoded name in isEncodedRegionName(byte[] regionName) (#2753)
Signed-off-by: stack <stack@apache.com>
2020-12-15 21:52:54 -08:00
huaxiangsun 1c217da2ff
HBASE-25293 Followup jira to address the client handling issue when chaning from meta replica to non-meta-replica at the server side. (#2768)
Signed-off-by: stack <stack@apache.org>
2020-12-15 21:45:39 -08:00
Baiqiang Zhao 1bb9b78787
HBASE-25365 The log in move_servers_rsgroup is incorrect (#2742)
Signed-off-by: stack <stack@apache.or>
2020-12-15 20:25:59 -08:00
Pankaj 9bdac6cd17
HBASE-25378 Legacy comparator in Hfile trailer will fail to load (#2756)
Signed-off-by: stack <stack@apache.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-15 23:56:02 +05:30
Michael Stack a4d42d1fcc HBASE-25389 [Flakey Tests] branch-2 TestMetaShutdownHandler (#2773)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-12-15 09:51:56 -08:00
Bo Cui d50816fe44
HBASE-23340 hmaster /hbase/replication/rs session expired (hbase repl… (#2739)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-12-12 21:10:33 +08:00
Huaxiang Sun e04792959a Revert "HBASE-25293 Followup jira to address the client handling issue when chaning from meta replica to non-meta-replica at the server side."
This reverts commit c1aa3b24e9.
2020-12-11 10:22:41 -08:00
haxl 635c911532
HBASE-25287 Forgetting to unbuffer streams results in many CLOSE_WAIT sockets when loading files (#2699)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-12-10 22:15:39 +08:00
haxl a9b8c10f80
HBASE-25334 TestRSGroupsFallback.testFallback is flaky (#2728)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-12-10 22:07:06 +08:00
Huaxiang Sun c1aa3b24e9 HBASE-25293 Followup jira to address the client handling issue when chaning from meta replica to non-meta-replica at the server side. 2020-12-09 09:12:01 -08:00
haxiaolin 56dd3eba81 HBASE-25363 Improve performance of HFileLinkCleaner by using ReadWriteLock instead of synchronize 2020-12-08 21:09:31 +08:00
Pankaj fb6e498b32
HBASE-25277 postScannerFilterRow impacts Scan performance a lot in HBase 2.x (#2675)
* HBASE-25277 postScannerFilterRow impacts Scan performance a lot in HBase 2.x

1. Added a check for Object class in RegionCoprocessorHost to avoid wrong initialization of hasCustomPostScannerFilterRow
2. Removed dummy implementation of postScannerFilterRow from AccessController, VisibilityController & ConstraintProcessor (which are not required currently)

Signed-off-by Ramkrishna S Vasudevan <ramkrishna@apache.org>
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-12-07 23:00:48 +05:30
stack 7d0a687e57 HBASE-25361 [Flakey Tests] branch-2 TestMetaRegionLocationCache.testStandByMetaLocations (#2736)
Addendum; Reset counter so we actually wait in the new loop added by the
above.
2020-12-05 14:01:56 -08:00
Michael Stack 6791b29771 HBASE-25361 [Flakey Tests] branch-2 TestMetaRegionLocationCache.testStandByMetaLocations (#2736)
Add a bit of a wait before testing if online replicas match the zk count. It might take a
while for all replicas to come online.

Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
2020-12-05 08:56:44 -08:00
Michael Stack 946fa81715 HBASE-25349 [Flakey Tests] branch-2 TestRefreshRecoveredReplication.testReplicationRefreshSource:141 Waiting timed out after [60,000] msec (#2731)
Start the check for recovered queue presence earlier.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-12-02 09:58:12 -08:00
lujiefsi 8938b7a678
HBASE-25332:fix One pontential NPE
Closes #2715

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-01 19:41:38 +05:30
Duo Zhang 86bb037eb0 Revert "HBASE-25334 TestRSGroupsFallback.testFallback is flaky (#2712)"
This reverts commit f71eb27be1.
2020-12-01 16:22:09 +08:00
haxl f71eb27be1
HBASE-25334 TestRSGroupsFallback.testFallback is flaky (#2712)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-12-01 15:24:02 +08:00
Michael Stack 7da90a1e23 HBASE-25345 [Flakey Tests] branch-2 TestReadReplicas#testVerifySecondaryAbilityToReadWithOnFiles (#2727)
Check TEST_SKIP_REPORTING_TRANSITION and if true, skip trying to talk to
update master on state transition -- i.e. reportFileArchivalForQuotas --
as we allow for reportRegionStateTransition (For tests only)

Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-11-30 21:04:51 -08:00
GeorryHuang 264412d30c
HBASE-25339 Method parameter and member variable are duplicated in checkSplittable() of SplitTableRegionProcedure
Closes #2717

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-30 19:49:46 +05:30
Viraj Jasani a8a35da73e
Revert "HBASE-25339 Method parameter and member variable are duplicated in checkSplittable() of SplitTableRegionProcedure"
This reverts commit c76145c505.
2020-11-30 19:48:58 +05:30
GeorryHuang c76145c505
HBASE-25339 Method parameter and member variable are duplicated in checkSplittable() of SplitTableRegionProcedure
Closes #2717

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-30 17:59:45 +05:30
Duo Zhang fc62214e77
HBASE-25341 Fix ErrorProne error which causes nightly to fail (#2718)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-30 14:14:44 +08:00
Akshay Sudheer bfa5cb7d48
HBASE-25321 The sort icons not shown after Upgrade JQuery to 3.5.1 (#2694)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-29 12:08:57 +08:00
Baiqiang Zhao e2c1b10cb7
HBASE-25330 RSGroupInfoManagerImpl#moveServers return is not set of servers moved (#2708)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-29 11:32:51 +08:00
Andrew Purtell 679dd7b8f9
HBASE-25292 Improve InetSocketAddress usage discipline (#2669)
Network identities should be bound late. Remote addresses should be
resolved at the last possible moment, just before connect(). Network
identity mappings can change, so our code should not inappropriately
cache them. Otherwise we might miss a change and fail to operate normally.

Revert "HBASE-14544 Allow HConnectionImpl to not refresh the dns on errors"
Removes hbase.resolve.hostnames.on.failure and related code. We always
resolve hostnames, as late as possible.

Preserve InetSocketAddress caching per RPC connection. Avoids potential
lookups per Call.

Replace InetSocketAddress with Address where used as a map key. If we want
to key by hostname and/or resolved address we should be explicit about it.
Using Address chooses mapping by hostname and port only.

Add metrics for potential nameservice resolution attempts, whenever an
InetSocketAddress is instantiated for connect; and metrics for failed
resolution, whenever InetSocketAddress#isUnresolved on the new instance
is true.

* Use ServerName directly to build a stub key

* Resolve and cache ISA on a RpcChannel as late as possible, at first call

* Remove now invalid unit test TestCIBadHostname

We resolve DNS at the latest possible time, at first call, and do not
resolve hostnames for creating stubs at all, so this unit test cannot
work now.

Reviewed-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-28 21:01:22 +08:00
Lokesh Khurana e6165412e0
HBASE-25237 : 'hbase master stop' shuts down the cluster, not the master only
Closes #2705

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-27 16:02:30 +05:30
Qi Yu dc48071e01
HBASE-25324 Remove unnecessary array to list conversion in SplitLogManager (#2703)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.com>
2020-11-27 08:20:24 +08:00
Mate Szalay-Beko 451a4b06b1 HBASE-25263 Various improvements of column family encryption
This PR is a follow-up of HBASE-25181 (#2539), where several issues were
discussed on the PR:

1. Currently we use PBKDF2WithHmacSHA1 key generation algorithm to generate a
secret key for HFile / WalFile encryption, when the user is defining a string
encryption key in the hbase shell. This algorithm is not secure enough and
not allowed in certain environments (e.g. on FIPS compliant clusters). We are
changing it to PBKDF2WithHmacSHA384. It will not break backward-compatibility,
as even the tables created by the shell using the new algorithm will be able
to load (e.g. during bulkload / replication) the HFiles serialized with the
key generated by an old algorithm, as the HFiles themselves already contain
the key necessary for their decryption.

Smaller issues fixed by this commit:

2. Improve the documentation e.g. with the changes introduced by HBASE-25181
and also by some points discussed on the Jira ticket of HBASE-25263.

3. In EncryptionUtil.createEncryptionContext the various encryption config
checks should throw IllegalStateExceptions instead of RuntimeExceptions.

4. Test cases in TestEncryptionTest.java should be broken down into smaller
tests.

5. TestEncryptionDisabled.java should use ExpectedException JUnit rule to
validate exceptions.

closes #2676

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-11-26 16:44:34 +01:00
Toshihiro Suzuki b142f5dcd2
HBASE-25242 Add Increment/Append support to RowMutations (#2630)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-11-26 13:31:43 +09:00
Andrew Purtell 1726160839
HBASE-24640 Purge use of VisibleForTesting (#2695)
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-11-25 14:10:12 -08:00
Qi Yu 403756ade5
HBASE-25282 Remove processingServers in DeadServer as we can get this… (#2657)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-25 15:16:27 +08:00
Qi Yu 984d578dd4
HBASE-25323 Fix potential NPE when the zookeeper path of RegionServerTracker does not exist when start (#2702)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-25 15:13:09 +08:00
Qi Yu 31d880b5dd
HBASE-25325 Remove unused class ClusterSchemaException (#2704)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-25 15:09:40 +08:00
niuyulin 4affae5938
HBASE-25213 Should request Compaction when bulkLoadHFiles is done (addendum) (#2701)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-25 07:57:18 +08:00
tianhang 390abb520c
HBASE-25299 Deprecate Scan#setRowPrefixFilter because of an unexpected behavior
Closes #2674

Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-24 19:39:25 +05:30
ramkrish86 f221d11227
HBASE-25050 - We initialize Filesystems more than once. (#2419)
* HBASE-25050 - We initialize Filesystems more than once.

* Ensuring that calling the FS#get() will only ensure FS init.

* Fix for testfailures. We should pass the entire path and no the scheme
alone

* Cases where we don't have a scheme for the URI

* Address review comments

* Add some comments on why FS#get(URI, conf) is getting used

* Adding the comment as per Sean's review

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2020-11-24 15:31:04 +05:30
XinSun 1cd8f3cf94
HBASE-24654 Allow unset table's rsgroup (#1994)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-23 17:22:37 +08:00
Bo Cui b0f20399e7
HBASE-25311 UI throws NPE (#2688)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Wellington Chevreuil <wellington.chevreuil@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-23 12:07:42 +08:00
niuyulin a307d706c8
HBASE-25281 Bulkload split hfile too many times due to unreasonable split point (#2667)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Wellington Chevreuil <wellington.chevreuil@gmail.com>
2020-11-23 10:07:19 +08:00
stack eca904e0fb HBASE-25291 Document how to enable the meta replica load balance mode for the client and clean up around hbase:meta read replicas
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSelector.java
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSelectorFactory.java
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSimpleSelector.java
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaMode.java
 Make the leading license comment opener align on all these new
 files.... one star rather than two.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java
 Add some comment better situating the catalogreplicationsource
 specialization.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
 Add comment how this differs from replicationsource factory and why no
 need to keep offset up in zk.

src/main/asciidoc/_chapters/architecture.adoc
 Clean up the read replica of META doc. Introduce the new read replica
 liveness on META and the new 'LoadBalance' feature (with warnings).
 General edit.
2020-11-20 00:04:48 -08:00
Huaxiang Sun 9ecf6ff6a2 HBASE-25126 Add load balance logic in hbase-client to distribute read load over meta replica regions (addendum) 2020-11-20 00:04:48 -08:00
Huaxiang Sun 690b4d8973 HBASE-25126 Add load balance logic in hbase-client to distribute read load over meta replica regions
It adds load balance support for meta lookup in AsyncTableRegionLocator.
The existing meta replica mode is renamed as "HedgedRead", client sends scan request to the primary meta replica region first,
if response is not back within a configured amount of time, it will send scan requests to all meta replica regions and
take the first response. On top of the existing mode, a new mode "LoadBalance" is introduced. In this mode, client first
choose a meta replica region to send scan request. If the response is stale, it may send the request to another meta replica region or
the primary region. In this mode, meta scan requests are load balanced across all replica regions with the primary mode as
the ultimate source of truth.

Two new config knobs are added:

1. hbase.locator.meta.replicas.mode
   Valid options are "None", "HedgedRead" and "LoadBalance", they are case insensitive. The default mode is "None".

2. hbase.locator.meta.replicas.mode.loadbalance.selector
   The load balance alogrithm to select a meta replica to send the requests.
   Only org.apache.hadoop.hbase.client.CatalogReplicaLoadBalanceReplicaSimpleSelector.class
   is supported for now, which is the default as well. The algorithm works as follows:
      a. Clients select a randome meta replica region to send the requests if there is no entry for the location in the stale
         location cache.
      b. If the location from one meta replica region is stale, a stale entry will be created in the statle location cache
         for the region.
      c. Clients select the primary meta region if the location is in the stale location cache.
      d. The stale location cache entries time out in 3 seconds.

If there is no "hbase.locator.meta.replicas.mode" configured, it will check the config knob "hbase.meta.replicas.use".
If "hbase.meta.replicas.use" is configured, the mode will be set to "HedgedRead".
2020-11-20 00:04:48 -08:00
stack 40843bb567 HBASE-25151 warmupRegion frustrates registering WALs on the catalog replicationsource
warmupRegion called by Master on Region move will instatiate
the meta WALProvider as part of its action making it so
it is already created by the time we go to open the
hbsae:meta Region. Accommodate meta walProvider
being already up.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
 Pass regionInfo. Needed internally.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
 Add handling if meta wal provider already instantiated when
 addCatalogReplicationSource runs.

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java
 Add exercising moving meta around between servers. Test replication
 keeps working.
2020-11-20 00:04:48 -08:00
Michael Stack 89cb0c5370 HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable w… (#2451)
* HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable when hbase:meta assigned to RS

Fill in gap left by HBASE-11183 'Timeline Consistent region replicas - Phase 2 design'.
HBASE-11183 left off implementing 'async WAL Replication' on the hbase:meta
Table; hbase:meta Table could only do Phase 1 Region Replicas reading
the primary Regions' hfiles. Here we add 'async WAL Replication' to
hbase:meta so Replicas can be more current with the primary's changes.

Adds a 'special' ReplicationSource that reads hbase:meta WAL files and replicates
all edits to the configured in-cluster endpoint (Defaults to the
RegionReadReplicaEndpoint.class -- set hbase.region.replica.catalog.replication to
target a different endpoint implementation).

Set hbase.region.replica.replication.catalog.enabled to enable async WAL
Replication for hbase:meta region replicas. Its off by default.

The CatalogReplicationSource for async WAL Replication of hbase:meta does
NOT need to keep up WAL offset or a queue of WALs-to-replicate in the
replication queue store as is done in other ReplicationSource implementations;
the CatalogReplicationSource is for Region Replicas only. General
Replication does not replicate hbase:meta. hbase:meta Region Replicas reset
on crash of the primary replica so there is no need to 'recover'
replication that was running on the crashed server.

Because it so different in operation, the CatalogReplicationSource is bolted
on to the side of the ReplicationSourceManager. It is lazily
instantiated to match the lazy instantiation of the hbase:meta
WALProvider, created and started on the open of the first Region of an
hbase:meta table. Thereafter it stays up till the process dies, even if
all hbase:meta Regions have moved off the server, in case a hbase:meta
Region is moved back (Doing this latter simplifies the implementation)

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
  Read configuration to see if we need to wait on setting a Region read-enabled
  (if so, replicas will only flip to enable read after confirming a
  flush of the primary so they for sure are a replica of a known point)

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
 If configured, on open of hbase:meta, ask the ReplicationSourceManager
 to add a ReplicationSource (if it hasn't already).

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java
 Edit log message.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
 If configured, on close of hbase:meta, update ReplicationSourceManager
 that a source Region has closed.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java
 javadoc and make constructor private.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java
 Add logPositionAndCleanOldLogs w/ default of the old behavior so
 CatalogReplicationSource can bypass updating store with WAL position,
 etc.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
 Add creation and start of an CatalogReplicationSource.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java
 Go via ReplicationSource when calling logPostionAndCleanOldLogs so new RS can intercept.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java
 Javadoc.

hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java
 Add utility for reading configurations for hbase:meta region replicas.

hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java
 Javadoc.

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
 Use define.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java
 Specical version of ReplicationSource for Region Replicas on hbase:meta.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSourcePeer.java
 Needs a special peer too (peers are baked into replication though we don't use 'peers' here)

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALProvider.java
 Tests.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-11-20 00:04:48 -08:00
stack 7964d2ec6d HBASE-25068 Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space
Pass WALFactory to Replication instead of WALProvider. WALFactory has all
WALProviders in it, not just the user-space WALProvider. Do this so
ReplicationService has access to all WALProviders in the Server (To be
exploited by the follow-on patch in HBASE-25055)
2020-11-20 00:04:48 -08:00
niuyulin 55399a0320
HBASE-25213 Should request Compaction after bulkLoadHFiles is done (#2587)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-20 14:18:42 +08:00
Baiqiang Zhao 30ef3aa457
HBASE-25306 The log in SimpleLoadBalancer#onConfigurationChange is wrong
Closes #2679

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-19 22:20:03 +05:30
Baiqiang Zhao 8c1e4763b3
HBASE-25298 hbase.rsgroup.fallback.enable should support dynamic configuration (#2668)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-19 09:20:01 +08:00
Mate Szalay-Beko 6a529d3269 HBASE-25261 Upgrade Bootstrap to 3.4.1
HBase UI is currently using in bootstrap 3.3.7. This version is vulnerable to 4
medium CVEs (CVE-2018-14040, CVE-2018-14041, CVE-2018-14042, and CVE-2019-8331).
Details on all the bootstrap versions and vulnerabilities is
here: https://snyk.io/vuln/npm:bootstrap

Upgrading to bootstrap 4 would be nice, but potentially more work to do. We
should at least upgrade to the latest bootstrap 3, which is 3.4.1 currently.

closes #2661

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-11-17 12:44:20 +01:00
gkanade 300b0a650f
HBASE-25026 Create a metric to track full region scans RPCs
Add new metric rpcFullScanRequestCount to track number of requests that are full region scans. Can be used to notify user to check if this is truly intended.
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by Ramkrishna S Vasudevan <ramkrishna@apache.org>
2020-11-17 08:48:59 +05:30
stack c07f27e025 Revert "HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConnectionRegistry (#2652)"
This reverts commit 6210dafc47.

Applied to master when should have been applied to branch. Revert.
2020-11-16 08:59:12 -08:00
Michael Stack 6210dafc47
HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConnectionRegistry (#2652)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-11-16 08:58:05 -08:00
stack 1c85c14994 Revert "HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConnectionRegistry"
This reverts commit adbf81c010a5fbfebfcc4aae68e35dbd12da1601.

Premature. Still some comments to address.
2020-11-16 08:45:20 -08:00
stack 0aff1759a8 HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConnectionRegistry
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-11-16 08:45:20 -08:00
Hossein Zolfi f68f3dd799
HBASE-25273 fix typo in StripeStoreFileManager java doc (#2653)
Co-authored-by: Hossein Zolfi <h.zolfi@inside.sahab.ir>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-11-14 18:47:57 +01:00
Duo Zhang 09aaa68be4 HBASE-25255 Addendum wait for meta loaded instead of master initialized for system table creation 2020-11-13 15:19:09 +08:00
Duo Zhang f89faf3ac8
HBASE-25255 Master fails to initialize when creating rs group table (#2638)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-13 14:28:31 +08:00
WenFeiYi aaeeaa582e
HBASE-25253 Deprecated master carrys regions related methods and configs (#2635)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-12 22:13:24 +08:00
niuyulin 0b6d6fd773
HBASE-25276 Need to throw the original exception in HRegion#openHRegion (#2648)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-12 17:40:24 +08:00
ramkrish86 57d9cae480
HBASE-25187 Improve SizeCachedKV variants initialization (#2582)
* HBASE-25187 Improve SizeCachedKV variants initialization

* HBASE-25187 Improve SizeCachedKV variants initialization

* The BBKeyValue also can be optimized

* Change for SizeCachedKeyValue

* Addressing revew comments

* Fixing checkstyle and spot bugs comments

* Spot bug fix for hashCode

* Minor updates make the rowLen as short and some consturctor formatting

* Change two more places where there was a cast
2020-11-11 17:39:39 +05:30
Mate Szalay-Beko 6a5c928539
HBASE-25181 Add options for disabling column family encryption and choosing hash algorithm for wrapped encryption keys.
Prior to this patch hbase always used the MD5 hash algorithm to store a hash for encryption keys.
This hash is needed to verify the secret key of the subject. (e.g. making
sure that the same secrey key is used during encrypted HFile read and write).
The MD5 algorithm is considered weak, and can not be used in some
(e.g. FIPS compliant) clusters.

In this patch we:
- add a config parameter to globally enable/disable column family encryption (def enabled)
- introduce a backward compatible way of specifying the hash algorithm.
  This enable us to use newer and more secure hash algorithms like SHA-384
  or SHA-512 (which are FIPS compliant).
- add a config parameter to fail if an hfile is encountered that uses a
  different hash algorithm than the one currently configured to ease validation after
  migrating key hash algorithms (def disabled)

Closes #2539

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Esteban Gutierrez <esteban@apache.org>
2020-11-09 13:53:50 -06:00
Duo Zhang 58c974888f
HBASE-25257 Remove MirroringTableStateManager (#2634)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-09 21:36:44 +08:00
gvprathyusha6 5c7432f4a9
HBASE-24667 Rename configs that support atypical DNS set ups to put them in hbase.unsafe
Closes #2542

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-09 12:27:34 +05:30
Duo Zhang c36ee4fac0
HBASE-25254 Rewrite TestMultiLogThreshold to remove the LogDelegate in RSRpcServices (#2631)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-08 21:47:18 +08:00
Duo Zhang 671129df56
HBASE-25252 Move HMaster inner classes out (#2628)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-07 20:05:04 +08:00
Michael Stack 23e656712b HBASE-25238 Upgrading HBase from 2.2.0 to 2.3.x fails because of “Message missing required fields: state” (#2625)
Make protobuf fields add post-2.0.0 release marked 'required' instead
be 'optional' so migrations from 2.0.x to 2.1+ or 2.2+ succeeds.

Signed-off-by: Viraj Jasani vjasani@apache.org
2020-11-05 08:49:47 -08:00
WenFeiYi 0356e8efd1
HBASE-25240 gson format of RpcServer.logResponse is abnormal
Closes #2623

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-05 19:58:05 +05:30
niuyulin 0e71d6192a
HBASE-25053 WAL replay should ignore 0-length files (#2437)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-04 08:21:31 -08:00
Duo Zhang 49774c7e18
HBASE-25216 The client zk syncer should deal with meta replica count change (#2614)
Signed-off-by: Yu Li <liyu@apache.org>
2020-11-04 17:54:18 +08:00
Duo Zhang f37cd05c32
HBASE-25235 Cleanup the deprecated methods in TimeRange (#2616)
Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-11-04 11:02:28 +08:00
Andrew Purtell c98e993b23
HBASE-25212 Optionally abort requests in progress after deciding a region should close (#2574)
If hbase.regionserver.close.wait.abort is set to true, interrupt RPC
handler threads holding the region close lock.

Until requests in progress can be aborted, wait on the region close lock for
a configurable interval (specified by hbase.regionserver.close.wait.time.ms,
default 60000 (1 minute)). If we have failed to acquire the close lock after
this interval elapses, if allowed (also specified by
hbase.regionserver.close.wait.abort), abort the regionserver.

We will attempt to interrupt any running handlers every
hbase.regionserver.close.wait.interval.ms (default 10000 (10 seconds)) until
either the close lock is acquired or we reach the maximum wait time.

Define a subset of region operations as interruptible. Track threads holding
the close lock transiting those operations. Set the thread interrupt status
of tracked threads when trying to close the region. Use the thread interrupt
status where safe to break out of request processing.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-03 15:20:27 -08:00
Nick Dimiduk d790bdedde
HBASE-25167 Normalizer support for hot config reloading (#2523)
Wire up the `ConfigurationObserver` chain for
`RegionNormalizerManager`. The following configuration keys support
hot-reloading:
 * hbase.normalizer.throughput.max_bytes_per_sec
 * hbase.normalizer.split.enabled
 * hbase.normalizer.merge.enabled
 * hbase.normalizer.min.region.count
 * hbase.normalizer.merge.min_region_age.days
 * hbase.normalizer.merge.min_region_size.mb

Note that support for `hbase.normalizer.period` is not provided
here. Support would need to be implemented generally for the `Chore`
subsystem.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Aman Poonia <aman.poonia.29@gmail.com>
2020-10-30 10:41:56 -07:00
Pankaj e3beccf1fc
HBASE-24977 Meta table shouldn't be modified as read only (#2537)
Signed-off-by: stack <stack@apache.org>
2020-10-29 16:34:23 -07:00
Nick Dimiduk 12d0397015 HBASE-24419 Normalizer merge plans should consider more than 2 regions when possible
The core change here is to the loop in
`SimpleRegionNormalizer#computeMergeNormalizationPlans`. It's a nested
loop that walks the table's region chain once, looking for contiguous
sequences of regions that meet the criteria for merge. The outer loop
tracks the starting point of the next sequence, the inner loop looks
for the end of that sequence. A single sequence becomes an instance of
`MergeNormalizationPlan`.

Signed-off-by: Huaxiang Sun <huaxiangsun@apache.org>
2020-10-29 13:37:23 -07:00
GeorryHuang 35b344c337 HBASE-25090 CompactionConfiguration logs unrealistic store file sizes (#2595)
Signed-off-by: stack <stack@apache.org>
2020-10-29 13:10:35 -07:00
Peter Somogyi bb4a9d335f
HBASE-25224 Maximize sleep for checking meta and namespace regions availability (#2593)
Signed-off-by: Michael Stack <stack@apache.org>
2020-10-28 18:08:05 +01:00
Minji Kim 735689d0f7
HBASE-25223 Use try-with-resources statement (#2592)
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-10-28 09:17:31 -07:00
niuyulin 17f9aded2c
HBASE-25201 YouAreDeadException should be moved to hbase-server module (#2581)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-28 14:58:28 +08:00
niuyulin 485da758bb
HBASE-25176 MasterStoppedException should be moved to hbase-client module (#2538)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-28 14:52:27 +08:00
WenFeiYi 4bbc772ffc
HBASE-25173 Remove owner related methods in TableDescriptor/TableDescriptorBuilder (#2541)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-27 22:01:57 +08:00
Duo Zhang 164cc5a3dc
HBASE-25215 TestClientOperationTimeout.testScanTimeout is flaky (#2583)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-10-26 10:03:43 +08:00
Reid Chan e5d4e2fc81
HBASE-25189 [Metrics] Add checkAndPut and checkAndDelete latency metrics at table level (#2549)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-10-25 17:46:14 +08:00
Toshihiro Suzuki 881c92b892
HBASE-25206 Data loss can happen if a cloned table loses original split region(delete table) (#2569)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-24 22:49:14 +08:00
Sandeep Pal 5e3ffb1db6
HBASE-25193: Add support for row prefix and type in the WAL Pretty Printer
Closes #2556

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-10-23 12:30:35 +05:30
Duo Zhang 74df1e1aba
HBASE-25207 Revisit the implementation and usage of RegionStates.include (#2571)
Remove the RegionStates.include method as its name is ambiguous.
Add more comments to describe the logic on why we filter region like
this.

Signed-off-by: Toshihiro Suzuki <brfrn169@gmail.com>
2020-10-22 21:50:43 +08:00
Jan Hentschel 2cd5f914af
HBASE-25197 Remove SingletonCoprocessorService interface (#2560)
Remove the SingletonCoprocessorService interface targeted for removal in
3.0.0.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-10-21 19:04:17 +02:00
Jan Hentschel 1b58af1841
HBASE-25198 Remove deprecated RpcSchedulerFactory#create (#2561)
Remove the deprecated RpcSchedulerFactory#create(Configuration,
PriorityFunction) method from the interface and in all implementing
classes.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-21 21:40:45 +08:00
Ankit Singhal 7d00e43dce
HBASE-25166 MobFileCompactionChore is closing the master's shared cluster connection (#2514) 2020-10-19 22:22:33 -07:00
shahrs87 e6c78f4037
HBASE-25179 : Fix Assert format in HFilePerformanceEvaluation class
Closes #2551

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-10-16 12:08:10 +05:30
Nick Dimiduk 70d2f37ee3
HBASE-24628 Region normalizer now respects a rate limit (HMaster chore shutdown NPE ADDENDUM) (#2540)
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-15 22:02:53 +08:00
Duo Zhang 38ac7e26ae
HBASE-25186 TestMasterRegionOnTwoFileSystems is failing after HBASE-25065 (#2544)
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
2020-10-15 10:22:14 +08:00