Commit Graph

8484 Commits

Author SHA1 Message Date
huaxiangsun 91205bf227
HBASE-25691 Test failure: TestVerifyBucketCacheFile.testRetrieveFromFile (#3088)
The issue is that FileInputStream is created with try-with-resources, so its close() is called right after the try sentence.
FileInputStream is a finalize class, when this object is garbage collected, its close() is called again.
To avoid this double-free resources, add guard against it.

Signed-off-by: stack <stack@apache.org>
2021-03-24 14:39:35 -07:00
Toshihiro Suzuki 4cf28c43f5 HBASE-25678 Support nonce operations for Increment/Append in RowMutations and CheckAndMutate
Signed-off-by: stack <stack@apache.org>
2021-03-21 23:39:45 +09:00
shahrs87 27e9c281da HBASE-25679 Size of log queue metric is incorrect (#3071)
Co-authored-by: Rushabh <rushabh.shah@salesforce.com>
Signed-off-by: stack <stack@apache.org>
2021-03-19 16:16:13 -07:00
haxiaolin df6e5dd0b1 HBASE-25518 Support separate child regions to different region servers (#3001)
Signed-off-by: stack <stack@apache.org>
2021-03-18 12:39:15 -07:00
Michael Stack 3331e8307a HBASE-25677 Server+table counters on each scan #nextRaw invocation becomes a bottleneck when heavy load (#3061)
Don't have every handler update regionserver metrics on each
scan#nextRaw; instead, do a batch update just before Scan
returns. Otherwise, all running handlers end up contending
on metrics update.

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 Update of regionserver metrics counters moved out to caller where
 can be done as a batch update instead of per-next.

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServer.java
 Class doc to encourage batch updating metrics.
 Remove the single update as unused anymore.

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
 Count calls to nextRaw. Update regionserver count in finally block when
 scan is done rather than per nextRaw call. Move all metrics updates to
 finally.

Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Baiqiang Zhao <ZhaoBQ>
2021-03-18 11:34:53 -07:00
Sandeep Pal c485a6bb79
HBASE-25627: HBase replication should have a metric to represent if the source is stuck getting initialized (#3018)
Introduces a new metric that tracks number of replication sources that are stuck in initialization.

Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
(cherry picked from commit ff3821814a)
2021-03-17 10:36:24 -07:00
stack 7de67b30fc Revert "HBASE-25665 Option to use hostname instead of canonical hostname for secure HBase cluster connection (#3051)"
This reverts commit 90c147f97c.

Reverting because new feature and only want bug fixes on branches.
2021-03-16 21:08:24 -07:00
bitterfox 90c147f97c HBASE-25665 Option to use hostname instead of canonical hostname for secure HBase cluster connection (#3051) 2021-03-16 21:05:42 -07:00
Baiqiang Zhao aa6c013a01 HBASE-25597 Add row info in Exception when cell size exceeds maxCellSize (#2976)
Signed-off-by: stack <stack@apache.org>
2021-03-15 15:50:45 -07:00
haxiaolin df2daf9e1e HBASE-25621 Balancer should check region plan source to avoid misplace region groups (#3002)
Signed-off-by: stack <stack@duboce.net>
2021-03-15 14:48:47 -07:00
haxiaolin 3c9a7e01ed HBASE-25595 TestLruBlockCache.testBackgroundEvictionThread is flaky (#2974)
Signed-off-by: stack <stack@apache.org>
2021-03-15 14:26:31 -07:00
Michael Stack 6414ef91d6 HBASE-25660 Print split policy in use on Region open (as well as split policy vitals) (#3044)
Add a toString to all split policy implementations listing name and
 vitals. Use this toString in the Region open message. Ditto for flush
 policy for the Region.

 Signed-off-by: Huaxiang Sun<huaxiangsun@apache.org>
2021-03-15 14:11:54 -07:00
haxiaolin 2ecf9232bf HBASE-25635 CandidateGenerator may miss some region balance actions (#3024)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-03-15 21:42:02 +08:00
Duo Zhang f9d80f18cb HBASE-25657 Fix spotbugs warnings after upgrading spotbugs to 4.x (#3041)
Signed-off-by: meiyi <myimeiyi@gmail.com>
Signed-off-by: stack <stack@apache.org>
2021-03-12 14:37:29 +08:00
shahrs87 5c149d1ce6 HBASE-25622 Result#compareResults should compare tags. (#3026)
Signed-off-by: stack <stack@apache.org>
2021-03-11 21:52:45 -08:00
Michael Stack 8658650823 HBASE-25570 On largish cluster, "CleanerChore: Could not delete dir..." makes master log unreadable (#2949)
Turn down the amount we log. If you want to see the full exception
enable TRACE-level logging.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: shahrs87
2021-03-11 21:37:16 -08:00
Peter Somogyi 723f2eea76 HBASE-25587 [hbck2] Schedule SCP for all unknown servers (#2978)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2021-03-11 17:28:56 +01:00
meiyi 35d44c6ce6 HBASE-25636 Expose HBCK report as metrics (#3031)
Signed-off-by: zhangduo <zhangduo@apache.org>
2021-03-11 15:32:41 +08:00
Duo Zhang ee7beca604 HBASE-25629 Reimplement TestCurrentHourProvider to not depend on unstable TZs (#3013)
Signed-off-by: XinSun <ddupgs@gmail.com>
2021-03-10 23:54:10 +08:00
Wellington Ramos Chevreuil 75a4b6143e HBASE-25548 Optionally allow snapshots to preserve cluster's max file… (#2923)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
(cherry picked from commit 373dc7788d)
2021-03-10 10:37:46 +00:00
Anoop Sam John e793303d7f HBASE-25582 Support setting scan ReadType to be STREAM at cluster level (#3040)
Signed-off-by: zhangduo <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-03-10 13:24:31 +05:30
Narges Shadab 4a0b3cc724
HBASE-25646: Possible Resource Leak in CatalogJanitor #3036
Close the stream using a try-with-resources block.

Reviewed-by: Aman Poonia <aman.poonia.29@gmail.com>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
(cherry picked from commit 573daed419)
2021-03-09 09:14:44 -08:00
Andrew Purtell bda2b8af8b Preparing development version 2.4.3-SNAPSHOT
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-03-08 15:59:42 -08:00
Andrew Purtell 3e98c51c51 Preparing hbase release 2.4.2RC0; tagging and updates to CHANGES.md and RELEASENOTES.md
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-03-08 15:59:36 -08:00
Duo Zhang 7f4ad3cfb2 HBASE-23578 [UI] Master UI shows long stack traces when table is broken (#3014)
Co-authored-by: Shuhei Yamasaki <yamasakisua@oss.nttdata.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-03-07 21:36:55 +08:00
Narges Shadab 13a86924d7 HBASE-25626 Possible Resource Leak in HeterogeneousRegionCountCostFunction
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by shahrs87 <shahrs87@gmail.com>
2021-03-05 08:58:05 +05:30
Rahul Kumar eb640a6f0f
HBASE-25460 : Expose drainingServers as cluster metric (#2994) (#2995)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-03-04 12:51:47 +05:30
Andrew Purtell ddbb3e4b69 HBASE-25385 TestCurrentHourProvider fails if the latest timezone changes are not present (#3012)
Disable this problematic test for now. May be removed after additional
discussion.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-03-03 18:16:24 -08:00
Sandeep Pal f535223861 HBASE-25596: Fix NPE and avoid permanent unreplicated data due to EOF (#3008)
Signed-off-by: Andrew Purtell <apurtell@apache.org>

Conflicts:
	hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
2021-03-03 11:40:51 -08:00
Akshay Sudheer 5b92554906 HBASE-25367 Sort broken after Change 'State time' in UI (#2964)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
(cherry picked from commit 5d9a6ed1fe)
2021-03-03 15:12:47 +05:30
Mohammad Arshad a561c71f00 HBASE-25371: When openRegion fails during initial verification(before… (#2785)
Signed-off-by: stack <stack@apache.org>
2021-03-03 08:28:31 +08:00
Andrew Purtell 65999a9027 Revert "HBASE-25596: Fix NPE and avoid permanent unreplicated data due to EOF (#2990)"
This reverts commit 9590080a21.
2021-03-02 12:23:17 -08:00
Balazs Meszaros 1c49ffd432 HBASE-25586 Fix HBASE-22492 on branch-2 (SASL GapToken) (#2961)
ServerCall.java: calling wrapWithSasl() was moved to getResponse(), so
the SASL wrapping is delayed until the reply is sent back to the client.

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2021-02-26 14:11:33 +01:00
XinSun db4f2bfa25 HBASE-25590 Bulkload replication HFileRefs cannot be cleared in some cases where set exclude-namespace/exclude-table-cfs (#2969)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2021-02-26 09:59:52 +08:00
Sandeep Pal 9590080a21 HBASE-25596: Fix NPE and avoid permanent unreplicated data due to EOF (#2990)
Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>

Conflicts:
	hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
2021-02-25 13:38:08 -08:00
XinSun 9ebddeeda5 HBASE-25598 TestFromClientSide5.testScanMetrics is flaky (#2977)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-02-24 14:45:19 +08:00
Andrew Purtell 4240dca4ff HBASE-25556 Frequent replication "Encountered a malformed edit" warnings (#2965)
ProtobufLogReader#readNext may be called by code that attempts to advance
the reader but does not necessarily expect to succeed, for example
WALEntryStream#tryAdvanceEntry. Much of the logging in this method is
at TRACE level. Other logging at WARN level will be frequently emitted, as
often as several per minute, and this will cause false positive assessment
from operators that they are experiencing a bug. Fix the mixed intent with
respect to log levels in readNext. Log at only DEBUG level or below.

Signed-off-by: Sean Busbey <busbey@apache.org>
2021-02-23 14:41:31 -08:00
Toshihiro Suzuki 9cfeec0deb HBASE-25575 Should validate Puts in RowMutations
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-02-22 11:04:09 +09:00
Toshihiro Suzuki b78366cf50 HBASE-25574 Revisit put/delete/increment/append related RegionObserver methods
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-02-22 10:16:42 +09:00
XinSun c1a9e8973a HBASE-25562 ReplicationSourceWALReader log and handle exception immediately without retrying (#2943)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: shahrs87
2021-02-20 10:28:18 +08:00
shahrs87 627ac016cc
HBASE-25539: Add age of oldest wal metric (#2962)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
(cherry picked from commit 6a4c9be967)
2021-02-18 21:26:39 -08:00
Sandeep Pal f637a60992
HBASE-25541 : Setting the path to null when we dequeue the current log (#2959)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2021-02-18 11:57:01 +05:30
pustota2009 83cc05a14e
HBASE-23887 Bug fix heavyEvictionMbSizeLimit (ADDENDUM) (#2957)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-02-15 20:46:14 +05:30
pustota2009 79272e481b
HBASE-23887 AdaptiveLRU cache (#2934)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-02-13 20:12:16 +05:30
Baiqiang Zhao 56937056ea HBASE-25534 Honor TableDescriptor settings earlier in normalization (#2917)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2021-02-09 16:57:59 -08:00
XinSun e7414f98fd HBASE-25559 Terminate threads of oldsources while RS is closing (#2938)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2021-02-09 16:57:29 +08:00
haxl 6849279023 HBASE-25507 Leak of ESTABLISHED sockets when compaction encountered "java.io.IOException: Invalid HFile block magic" (#2882)
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: stack <stack@apache.org>
2021-02-08 22:16:18 -08:00
Michael Stack de9bbd6968 HBASE-25542 Add client detail to scan name so when lease expires, we … (#2930)
* HBASE-25542 Add client detail to scan name so when lease expires, we have clue on who was scanning

When we create a scanner lease, record client ip and port (removed
unnecessary store of scannerName).

Signed-off-by: Clara Xiong <clarax98007@gmail.com>
2021-02-08 14:39:30 -08:00
Baiqiang Zhao 2989f8edb0 HBASE-25431 MAX_FILESIZE and MEMSTORE_FLUSHSIZE should not be set negative number (#2803)
Signed-off-by: stack <stack@apache.org>
2021-02-08 14:39:25 -08:00
XinSun 87e516da6c HBASE-25553 It is better for ReplicationTracker.getListOfRegionServers to return ServerName instead of String (#2928)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-02-07 19:06:46 +08:00
YutSean 829790e11a
HBASE-25543 When configuration hadoop.security.authorization is set to false, the system will still try to authorize an RPC and raise AccessDeniedException (#2929) (#2919)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
2021-02-05 22:08:32 +05:30
Baiqiang Zhao 1b6bf2bcd0
HBASE-25554 NPE when init RegionMover (#2927)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-02-05 14:05:23 +05:30
Pankaj 7a8e6a7106 HBASE-24900 Make retain assignment configurable during SCP (#2313)
Retain assignment will be useful in non-cloud scenario where RegionServer and Datanode are deployed in same machine and will avoid remote read.

Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
(cherry picked from commit a04ea7ea44)
2021-02-01 10:55:23 +05:30
shahrs87 8055a64b80
HBASE-25536 Remove 0 length wal file from logQueue if it belongs to old sources (#2908)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Geoffrey Jacoby <gjacoby@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-01-29 17:56:17 +05:30
Bharath Vissapragada 083b632c4b
HBASE-25528: Dedicated merge dispatch threadpool on master
Adds "hbase.master.executor.merge.dispatch.threads" and defaults to 2.

Also adds additional logging that includes the number of split plans
and merge plans computed for each normalizer run.

(cherry picked from commit 36b4698cea)
2021-01-28 16:15:27 -08:00
Aman Poonia 8a330d4591
HBASE-25523 Region normalizer chore thread is getting killed (#2903)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-01-26 11:57:17 +05:30
Duo Zhang 219273768e HBASE-25509 fix typo when resolving conflicts 2021-01-20 23:37:02 +08:00
Duo Zhang 559a1f0a28 HBASE-25509 ChoreService.cancelChore will not call ScheduledChore.cle… (#2890)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-01-20 23:25:30 +08:00
Christine Feng bc827f8b5b
HBASE-25478 : Implement retries when enabling tables in TestRegionReplicaReplicationEndpoint (#2896)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-01-20 11:34:02 +05:30
Baiqiang Zhao 47fbff98e3 HBASE-25482 Improve SimpleRegionNormalizer#getAverageRegionSizeMb (#2858)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2021-01-19 14:49:01 -08:00
Baiqiang Zhao cc5b884a8d HBASE-25513 When the table is turned on normalize, the first region may not be merged even the size is 0 (#2887)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2021-01-19 11:05:13 -08:00
Anjan Das 7aca7814d9
HBASE-25475 : Unset zk based wal splitting explicitly in tests (ADDENDUM) (#2891)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-01-17 16:35:05 +05:30
Andrew Purtell bab2d41390 Preparing development version 2.4.2-SNAPSHOT
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-01-15 10:51:49 -08:00
Akshay Sudheer caea47ae1d HBASE-25404 Procedures table Id under master web UI gets word break to single character (#2783)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-01-15 10:32:10 -08:00
Andrew Purtell 4c3e174297 Revert "Preparing development version 2.4.2-SNAPSHOT"
This reverts commit e0aa015542.
2021-01-15 10:19:09 -08:00
Anjan Das dea4aa97ea
HBASE-25475: Improve UT added as part of HBASE-25445 in TestSplitWALManager (#2855)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2021-01-15 12:36:38 +05:30
Andrew Purtell e0aa015542 Preparing development version 2.4.2-SNAPSHOT
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-01-14 12:55:38 -08:00
Andrew Purtell 67a87017b4 Revert "Preparing development version 2.4.2-SNAPSHOT"
This reverts commit 00fec5a2f7.
2021-01-14 12:54:05 -08:00
Andrew Purtell 00fec5a2f7 Preparing development version 2.4.2-SNAPSHOT
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-01-14 11:43:50 -08:00
Andrew Purtell 3f495149e9 Revert "Preparing development version 2.4.2-SNAPSHOT"
This reverts commit 9f17bd2f1b.
2021-01-14 11:00:42 -08:00
Tak Lon (Stephen) Wu 83a6ca54bd Backport "HBASE-25249 Adding StoreContext" to branch-2 (#2869)
Co-authored-by: Abhishek Khanna <akkhanna@amazon.com>

Signed-off-by: Zach York <zyork@apache.org>
2021-01-14 10:56:26 -08:00
Andrew Purtell 9f17bd2f1b Preparing development version 2.4.2-SNAPSHOT
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-01-13 18:03:20 -08:00
Andrew Purtell 422a8cc32d Revert "Preparing development version 2.4.2-SNAPSHOT"
This reverts commit 29619049b2.
2021-01-13 17:44:09 -08:00
Andrew Purtell 29619049b2 Preparing development version 2.4.2-SNAPSHOT
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-01-13 11:39:43 -08:00
Andrew Purtell 645b564310 HBASE-25504 [branch-2.4] Restore method removed by HBASE-25277 to LP(CONFIG) coprocessors
Restore removed method overrides for AccessController and VisibilityController,
which are both LimitedPrivate(CONFIG) interfaces and per compatibility
guidelines cannot have methods removed in patch releases.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-01-13 11:04:19 -08:00
Andrew Purtell 6125d4ef84 Revert "[branch-2.4] Restore AccessController method removed by HBASE-25277"
This reverts commit d3b1689f60.
2021-01-13 11:04:19 -08:00
caroliney14 a35a8ad5f3
HBASE-25329 Dump ritsOverThreshold in logs (#2761)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-01-14 00:10:06 +05:30
Andrew Purtell d3b1689f60 [branch-2.4] Restore AccessController method removed by HBASE-25277
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-01-13 10:24:09 -08:00
Andrew Purtell 85075940c0 Revert "Preparing development version 2.4.2-SNAPSHOT"
This reverts commit 51c55fcc2f.
2021-01-13 10:24:09 -08:00
Andrew Purtell 51c55fcc2f Preparing development version 2.4.2-SNAPSHOT
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-01-12 17:06:38 -08:00
Andrew Purtell 35044cc40c Preparing hbase release 2.4.1RC0; tagging and updates to CHANGES.md and RELEASENOTES.md
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-01-12 17:06:33 -08:00
Viraj Jasani e8279b26fa HBASE-25211 : Rack awareness in RegionMover (#2795)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2021-01-12 15:37:53 -08:00
Wellington Ramos Chevreuil cc12edbe17 HBASE-24813 ReplicationSource should clear buffer usage on Replicatio… (#2546) (#2849)
Signed-off-by: Ankit Singhal <ankit@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2021-01-12 09:45:51 +00:00
xijiawen aa27488268 HBASE-25459 WAL can't be cleaned in some scenes (#2848)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
(cherry picked from commit 6a401d89f8)
2021-01-11 10:06:09 +00:00
Bo Cui 8b48c60347 HBASE-25483 set the loadMeta log level to debug (#2859)
Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
(cherry picked from commit a348204d4a)
2021-01-08 19:32:48 +05:30
xijiawen 99f2b71cc9 HBASE-25434 SlowDelete & SlowPut metric value should use updateDelete & updatePut (#2837)
Signed-off-by: Zheng Wang <wangzheng@apache.org>
(cherry picked from commit 7d70141b90)
2021-01-08 19:02:59 +08:00
Anjan Das e39bae4ddb
HBASE-25445: Use WAL FS instead of master FS in SplitWALManager (#2844)
Signed-off-by: Pankaj <pankajkumar@apache.org>
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-01-07 15:38:52 +05:30
Bo Cui 82a63abc8a HBASE-25447 remoteProc is suspended due to OOM ERROR (#2824)
Some OMME can not cause the JVM to exit, like "java.lang.OutOfMemoryError: Direct buffer memory", "java.lang.OutOfMemoryError: unable to create new native thread", as they dont call vmError#next_OnError_command. So abort HMaster when uncaught exception occurs in TimeoutExecutor, the new active Hmaster will resume the suspended procedure.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.com>
Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
(cherry picked from commit 600be60a4b)
(cherry picked from commit ae77f81e7e)
2021-01-06 23:46:24 +05:30
leyangyueshan 35648278fc HBASE-25435 Slow metric value can be configured
Signed-off-by: jediwang <jediwang@tencent.com>
2021-01-04 13:06:55 +08:00
lujiefsi f5a7fffd8b
HBASE-25456 : add security check for setRegionStateInMeta (#2835) (#2833)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-01-01 12:19:25 +05:30
lujiefsi a14f08876b
HBASE-25441 : add security check for some APIs in RSRpcServices (#2830) (#2810)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-31 00:25:49 +05:30
Duo Zhang 90db5e4f99 HBASE-25452 Use MatcherAssert.assertThat instead of org.junit.Assert.assertThat (#2826)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-12-30 11:06:58 +08:00
lujiefsi c3d755e318
HBASE-25432: branch-2.4 add security checks for setTableStateInMeta and fixMeta (#2817)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-28 10:58:31 -08:00
ramkrish86 e7307c91d0 HBASE-24850 CellComparator perf improvement (#2776)
* Using ContiguousCellFormat as a marker alone

* Commit the new file

* Fix the comparator logic that was an oversight

* Fix the sequenceId check order

* Adding few more static methods that helps in scan flow like query
matcher where we have more cols

* Remove ContiguousCellFormat and ensure compare() can be inlined

* applying negation as per review comment

* Fix checkstyle comments

* fix review comments

* Address review comments

Signed-off-by: stack <stack@apache.org>
Signed-off-by: AnoopSamJohn <anoopsamjohn@apache.org>
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
2020-12-28 13:12:38 +05:30
Duo Zhang dede90dc54 HBASE-25420 Some minor improvements in rpc implementation (#2792)
Signed-off-by: XinSun <ddupgs@gmail.com>
Signed-off-by: stack <stack@apache.com>
2020-12-20 11:30:35 +08:00
huaxiangsun baf3122ad8
HBASE-25356 HBaseAdmin#getRegion() needs to filter out non-regionName and non-encodedRegionName (#2759) (#2789)
Signed-off-by: stack <stack@apache.org>
2020-12-18 13:16:03 -08:00
huaxiangsun 1932a58a62
HBASE-25293 Followup jira to address the client handling issue when chaning from meta replica to non-meta-replica at the server side. (#2768) (#2787)
Signed-off-by: stack <stack@apache.org>
2020-12-18 13:15:25 -08:00
Sandeep Pal 4512163c2f
HBASE-25383: Ability to update and remove peer base config
Closes #2778

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Geoffrey Jacoby <gjacoby@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-18 14:21:54 +05:30
shahrs87 47efdbb871
HBASE-25246 Backup/Restore hbase cell tags (#2766)
Closes #2745

Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-17 12:55:41 +05:30
stack 5eefe62e29 HBASE-25400 [Flakey Tests] branch-2 TestRegionMoveAndAbandon 2020-12-16 10:36:18 -08:00
Michael Stack bdc864fb69 HBASE-25389 [Flakey Tests] branch-2 TestMetaShutdownHandler (#2773)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-12-15 09:51:19 -08:00
Pankaj ec8eb653f6 HBASE-25277 postScannerFilterRow impacts Scan performance a lot in HBase 2.x (#2675)
* HBASE-25277 postScannerFilterRow impacts Scan performance a lot in HBase 2.x

1. Added a check for Object class in RegionCoprocessorHost to avoid wrong initialization of hasCustomPostScannerFilterRow
2. Removed dummy implementation of postScannerFilterRow from AccessController, VisibilityController & ConstraintProcessor (which are not required currently)

Signed-off-by Ramkrishna S Vasudevan <ramkrishna@apache.org>
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit fb6e498b32)
2020-12-11 00:04:41 +05:30
haxl f526ce6814 HBASE-25287 Forgetting to unbuffer streams results in many CLOSE_WAIT sockets when loading files (#2699)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-12-10 22:21:54 +08:00
stack bac9764f26 HBASE-25353 [Flakey Tests] branch-2 TestShutdownBackupMaster 2020-12-05 14:23:18 -08:00
stack b1a5567740 HBASE-25361 [Flakey Tests] branch-2 TestMetaRegionLocationCache.testStandByMetaLocations (#2736)
Addendum; Reset counter so we actually wait in the new loop added by the
above.
2020-12-05 14:01:12 -08:00
Michael Stack a0f4c28430 HBASE-25361 [Flakey Tests] branch-2 TestMetaRegionLocationCache.testStandByMetaLocations (#2736)
Add a bit of a wait before testing if online replicas match the zk count. It might take a
while for all replicas to come online.

Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
2020-12-05 08:57:11 -08:00
Andrew Purtell 3ff115ebe4 Preparing development version 2.4.1-SNAPSHOT 2020-12-03 09:52:11 -08:00
Andrew Purtell f7bc7be1eb Revert "Preparing development version 2.4.1-SNAPSHOT"
This reverts commit 4127a3940a.
2020-12-03 09:37:38 -08:00
Andrew Purtell 4127a3940a Preparing development version 2.4.1-SNAPSHOT 2020-12-02 14:04:53 -08:00
Andrew Purtell 108c2ecb52 Preparing hbase release 2.4.0RC0; tagging and updates to CHANGES.md and RELEASENOTES.md 2020-12-02 14:04:49 -08:00
Michael Stack 9cc2f76248
HBASE-25349 [Flakey Tests] branch-2 TestRefreshRecoveredReplication.testReplicationRefreshSource:141 Waiting timed out after [60,000] msec (#2731)
Start the check for recovered queue presence earlier.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-12-02 09:55:24 -08:00
Michael Stack f0a4aefcdf
HBASE-25345 [Flakey Tests] branch-2 TestReadReplicas#testVerifySecondaryAbilityToReadWithOnFiles (#2727)
Check TEST_SKIP_REPORTING_TRANSITION and if true, skip trying to talk to
update master on state transition -- i.e. reportFileArchivalForQuotas --
as we allow for reportRegionStateTransition (For tests only)

Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-11-30 21:02:45 -08:00
GeorryHuang 433235e98c
HBASE-25339 Method parameter and member variable are duplicated in checkSplittable() of SplitTableRegionProcedure
Closes #2717

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-30 19:50:54 +05:30
Viraj Jasani 904b8a814b
Revert "HBASE-25339 Method parameter and member variable are duplicated in checkSplittable() of SplitTableRegionProcedure"
This reverts commit 861c5509ad.
2020-11-30 19:50:29 +05:30
GeorryHuang 861c5509ad
HBASE-25339 Method parameter and member variable are duplicated in checkSplittable() of SplitTableRegionProcedure
Closes #2717

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-30 18:01:05 +05:30
Duo Zhang d6d0a20247 HBASE-25341 Fix ErrorProne error which causes nightly to fail (#2718)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-30 14:48:41 +08:00
Akshay Sudheer 85bf542a44 HBASE-25321 The sort icons not shown after Upgrade JQuery to 3.5.1 (#2694)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-29 12:13:16 +08:00
Lokesh Khurana 8472c68720
HBASE-25237 : 'hbase master stop' shuts down the cluster, not the master only
Closes #2705

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-27 16:05:07 +05:30
Qi Yu e81f49c918 HBASE-25324 Remove unnecessary array to list conversion in SplitLogManager (#2703)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.com>
2020-11-27 08:21:13 +08:00
Toshihiro Suzuki 3775464981
HBASE-25242 Add Increment/Append support to RowMutations (#2711)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-11-27 03:53:19 +09:00
Mate Szalay-Beko 1dbfe96d69 HBASE-25263 Various improvements of column family encryption
This PR is a follow-up of HBASE-25181 (#2539), where several issues were
discussed on the PR:

1. Currently we use PBKDF2WithHmacSHA1 key generation algorithm to generate a
secret key for HFile / WalFile encryption, when the user is defining a string
encryption key in the hbase shell. This algorithm is not secure enough and
not allowed in certain environments (e.g. on FIPS compliant clusters). We are
changing it to PBKDF2WithHmacSHA384. It will not break backward-compatibility,
as even the tables created by the shell using the new algorithm will be able
to load (e.g. during bulkload / replication) the HFiles serialized with the
key generated by an old algorithm, as the HFiles themselves already contain
the key necessary for their decryption.

Smaller issues fixed by this commit:

2. Improve the documentation e.g. with the changes introduced by HBASE-25181
and also by some points discussed on the Jira ticket of HBASE-25263.

3. In EncryptionUtil.createEncryptionContext the various encryption config
checks should throw IllegalStateExceptions instead of RuntimeExceptions.

4. Test cases in TestEncryptionTest.java should be broken down into smaller
tests.

5. TestEncryptionDisabled.java should use ExpectedException JUnit rule to
validate exceptions.

closes #2676

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-11-26 16:57:12 +01:00
Andrew Purtell 1b6399a480
HBASE-24640 [branch-2] Purge use of VisibleForTesting (#2696)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-11-25 14:15:32 -08:00
Qi Yu 3dd425abfa HBASE-25323 Fix potential NPE when the zookeeper path of RegionServerTracker does not exist when start (#2702)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-25 15:28:32 +08:00
Qi Yu 6f07efb971 HBASE-25325 Remove unused class ClusterSchemaException (#2704)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-25 15:28:32 +08:00
niuyulin 8e3727ea06
HBASE-25281 Bulkload split hfile too many times due to unreasonable split point (#2692)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-25 15:27:09 +08:00
niuyulin d523d758af HBASE-25213 Should request Compaction when bulkLoadHFiles is done (addendum) (#2701)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-25 08:00:28 +08:00
ramkrish86 988be9321d HBASE-25050 - We initialize Filesystems more than once. (#2419)
* HBASE-25050 - We initialize Filesystems more than once.

* Ensuring that calling the FS#get() will only ensure FS init.

* Fix for testfailures. We should pass the entire path and no the scheme
alone

* Cases where we don't have a scheme for the URI

* Address review comments

* Add some comments on why FS#get(URI, conf) is getting used

* Adding the comment as per Sean's review

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2020-11-24 15:31:55 +05:30
ramkrish86 a8e02c97bc HBASE-25187 Improve SizeCachedKV variants initialization (#2582)
* HBASE-25187 Improve SizeCachedKV variants initialization

* HBASE-25187 Improve SizeCachedKV variants initialization

* The BBKeyValue also can be optimized

* Change for SizeCachedKeyValue

* Addressing revew comments

* Fixing checkstyle and spot bugs comments

* Spot bug fix for hashCode

* Minor updates make the rowLen as short and some consturctor formatting

* Change two more places where there was a cast
2020-11-24 15:10:44 +05:30
Bo Cui 0a1decdf18 HBASE-25311 UI throws NPE (#2688)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Wellington Chevreuil <wellington.chevreuil@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-23 12:08:14 +08:00
niuyulin 8976781302
HBASE-25213 Should request Compaction after bulkLoadHFiles is done (#2684)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-23 08:33:41 +08:00
stack 73086f25e6 HBASE-25291 Document how to enable the meta replica load balance mode for the client and clean up around hbase:meta read replicas
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSelector.java
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSelectorFactory.java
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSimpleSelector.java
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaMode.java
 Make the leading license comment opener align on all these new
 files.... one star rather than two.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java
 Add some comment better situating the catalogreplicationsource
 specialization.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
 Add comment how this differs from replicationsource factory and why no
 need to keep offset up in zk.

src/main/asciidoc/_chapters/architecture.adoc
 Clean up the read replica of META doc. Introduce the new read replica
 liveness on META and the new 'LoadBalance' feature (with warnings).
 General edit.
2020-11-20 15:49:00 -08:00
Huaxiang Sun 920cf9faf4 HBASE-25126 Add load balance logic in hbase-client to distribute read load over meta replica regions
It adds load balance support for meta lookup in AsyncTableRegionLocator.
The existing meta replica mode is renamed as "HedgedRead", client sends scan request to the primary meta replica region first,
if response is not back within a configured amount of time, it will send scan requests to all meta replica regions and
take the first response. On top of the existing mode, a new mode "LoadBalance" is introduced. In this mode, client first
choose a meta replica region to send scan request. If the response is stale, it may send the request to another meta replica region or
the primary region. In this mode, meta scan requests are load balanced across all replica regions with the primary mode as
the ultimate source of truth.

Two new config knobs are added:

1. hbase.locator.meta.replicas.mode
   Valid options are "None", "HedgedRead" and "LoadBalance", they are case insensitive. The default mode is "None".

2. hbase.locator.meta.replicas.mode.loadbalance.selector
   The load balance alogrithm to select a meta replica to send the requests.
   Only org.apache.hadoop.hbase.client.CatalogReplicaLoadBalanceReplicaSimpleSelector.class
   is supported for now, which is the default as well. The algorithm works as follows:
      a. Clients select a randome meta replica region to send the requests if there is no entry for the location in the stale
         location cache.
      b. If the location from one meta replica region is stale, a stale entry will be created in the statle location cache
         for the region.
      c. Clients select the primary meta region if the location is in the stale location cache.
      d. The stale location cache entries time out in 3 seconds.

If there is no "hbase.locator.meta.replicas.mode" configured, it will check the config knob "hbase.meta.replicas.use".
If "hbase.meta.replicas.use" is configured, the mode will be set to "HedgedRead".

For branch-2 support, it introduces support for ReversedScan over a specific non-default replica region, this is mainly
for load balance meta scan among all its replica regions.

Remove ConnectionImplementation#setUseMetaReplicas()
2020-11-20 10:45:24 -08:00
Michael Stack 47ad207664 HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConnectionRegistry (#2652)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-11-20 00:31:43 -08:00
stack 1f6779b2e0 HBASE-25151 warmupRegion frustrates registering WALs on the catalog replicationsource
warmupRegion called by Master on Region move will instatiate
the meta WALProvider as part of its action making it so
it is already created by the time we go to open the
hbsae:meta Region. Accommodate meta walProvider
being already up.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
 Pass regionInfo. Needed internally.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
 Add handling if meta wal provider already instantiated when
 addCatalogReplicationSource runs.

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java
 Add exercising moving meta around between servers. Test replication
 keeps working.
2020-11-20 00:31:43 -08:00
Michael Stack de1843ac94 HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable w… (#2451)
* HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable when hbase:meta assigned to RS

Fill in gap left by HBASE-11183 'Timeline Consistent region replicas - Phase 2 design'.
HBASE-11183 left off implementing 'async WAL Replication' on the hbase:meta
Table; hbase:meta Table could only do Phase 1 Region Replicas reading
the primary Regions' hfiles. Here we add 'async WAL Replication' to
hbase:meta so Replicas can be more current with the primary's changes.

Adds a 'special' ReplicationSource that reads hbase:meta WAL files and replicates
all edits to the configured in-cluster endpoint (Defaults to the
RegionReadReplicaEndpoint.class -- set hbase.region.replica.catalog.replication to
target a different endpoint implementation).

Set hbase.region.replica.replication.catalog.enabled to enable async WAL
Replication for hbase:meta region replicas. Its off by default.

The CatalogReplicationSource for async WAL Replication of hbase:meta does
NOT need to keep up WAL offset or a queue of WALs-to-replicate in the
replication queue store as is done in other ReplicationSource implementations;
the CatalogReplicationSource is for Region Replicas only. General
Replication does not replicate hbase:meta. hbase:meta Region Replicas reset
on crash of the primary replica so there is no need to 'recover'
replication that was running on the crashed server.

Because it so different in operation, the CatalogReplicationSource is bolted
on to the side of the ReplicationSourceManager. It is lazily
instantiated to match the lazy instantiation of the hbase:meta
WALProvider, created and started on the open of the first Region of an
hbase:meta table. Thereafter it stays up till the process dies, even if
all hbase:meta Regions have moved off the server, in case a hbase:meta
Region is moved back (Doing this latter simplifies the implementation)

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
  Read configuration to see if we need to wait on setting a Region read-enabled
  (if so, replicas will only flip to enable read after confirming a
  flush of the primary so they for sure are a replica of a known point)

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
 If configured, on open of hbase:meta, ask the ReplicationSourceManager
 to add a ReplicationSource (if it hasn't already).

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java
 Edit log message.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
 If configured, on close of hbase:meta, update ReplicationSourceManager
 that a source Region has closed.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java
 javadoc and make constructor private.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java
 Add logPositionAndCleanOldLogs w/ default of the old behavior so
 CatalogReplicationSource can bypass updating store with WAL position,
 etc.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
 Add creation and start of an CatalogReplicationSource.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java
 Go via ReplicationSource when calling logPostionAndCleanOldLogs so new RS can intercept.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java
 Javadoc.

hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java
 Add utility for reading configurations for hbase:meta region replicas.

hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java
 Javadoc.

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
 Use define.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java
 Specical version of ReplicationSource for Region Replicas on hbase:meta.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSourcePeer.java
 Needs a special peer too (peers are baked into replication though we don't use 'peers' here)

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALProvider.java
 Tests.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-11-20 00:31:43 -08:00
stack 2b1e8b306f HBASE-25068 Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space
Pass WALFactory to Replication instead of WALProvider. WALFactory has all
WALProviders in it, not just the user-space WALProvider. Do this so
ReplicationService has access to all WALProviders in the Server (To be
exploited by the follow-on patch in HBASE-25055)
2020-11-20 00:31:43 -08:00
Baiqiang Zhao bace137c36
HBASE-25306 The log in SimpleLoadBalancer#onConfigurationChange is wrong
Closes #2679

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-19 22:22:22 +05:30
gkanade 024349bd5d
HBASE-25026 Create a metric to track full region scans RPCs
Add new metric rpcFullScanRequestCount to track number of requests that are full region scans. Can be used to notify user to check if this is truly intended.

Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by Ramkrishna S Vasudevan <ramkrishna@apache.org>
2020-11-19 09:55:33 +05:30
Mate Szalay-Beko e8ecf2e016 HBASE-25261 Upgrade Bootstrap to 3.4.1
HBase UI is currently using in bootstrap 3.3.7. This version is vulnerable to 4
medium CVEs (CVE-2018-14040, CVE-2018-14041, CVE-2018-14042, and CVE-2019-8331).
Details on all the bootstrap versions and vulnerabilities is
here: https://snyk.io/vuln/npm:bootstrap

Upgrading to bootstrap 4 would be nice, but potentially more work to do. We
should at least upgrade to the latest bootstrap 3, which is 3.4.1 currently.

closes #2661

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-11-17 17:15:57 +01:00
Duo Zhang f50ede32d9 HBASE-25255 Master fails to initialize when creating rs group table (#2638)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-13 15:36:16 +08:00
WenFeiYi ea1375d6be HBASE-25253 Deprecated master carrys regions related methods and configs (#2635)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-12 22:23:30 +08:00
Duo Zhang b19df076a4
HBASE-25272 Support scan on a specific replica (#2645)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.org>
2020-11-12 21:25:53 +08:00
niuyulin c48f6ebd09 HBASE-25276 Need to throw the original exception in HRegion#openHRegion (#2648)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-12 17:41:39 +08:00
Mate Szalay-Beko 9fe62c3ec6
HBASE-25181 Add options for disabling column family encryption and choosing hash algorithm for wrapped encryption keys.
Prior to this patch hbase always used the MD5 hash algorithm to store a hash for encryption keys.
This hash is needed to verify the secret key of the subject. (e.g. making
sure that the same secrey key is used during encrypted HFile read and write).
The MD5 algorithm is considered weak, and can not be used in some
(e.g. FIPS compliant) clusters.

In this patch we:
- add a config parameter to globally enable/disable column family encryption (def enabled)
- introduce a backward compatible way of specifying the hash algorithm.
  This enable us to use newer and more secure hash algorithms like SHA-384
  or SHA-512 (which are FIPS compliant).
- add a config parameter to fail if an hfile is encountered that uses a
  different hash algorithm than the one currently configured to ease validation after
  migrating key hash algorithms (def disabled)

Closes #2539

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Esteban Gutierrez <esteban@apache.org>
(cherry picked from commit 6a5c928539)
2020-11-09 14:15:22 -06:00
gvprathyusha6 21b32d9a6f
HBASE-24667 Rename configs that support atypical DNS set ups to put them in hbase.unsafe
Closes #2542

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-11-09 12:41:04 +05:30
Duo Zhang 7a07dfe059 HBASE-25254 Rewrite TestMultiLogThreshold to remove the LogDelegate in RSRpcServices (#2631)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-08 22:16:03 +08:00
Duo Zhang 38512e6f38 HBASE-25252 Move HMaster inner classes out (#2628)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-07 20:14:39 +08:00
Michael Stack df0009d879
HBASE-25238 Upgrading HBase from 2.2.0 to 2.3.x fails because of “Message missing required fields: state” (#2625)
Make protobuf fields add post-2.0.0 release marked 'required' instead
be 'optional' so migrations from 2.0.x to 2.1+ or 2.2+ succeeds.
2020-11-05 08:36:55 -08:00
WenFeiYi de74312d74
HBASE-25240 gson format of RpcServer.logResponse is abnormal
Closes #2623

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-05 19:59:34 +05:30
niuyulin 3d8152b635 HBASE-25053 WAL replay should ignore 0-length files (#2437)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-04 08:22:14 -08:00
Duo Zhang 04293ad6dd HBASE-25216 The client zk syncer should deal with meta replica count change (#2614)
Signed-off-by: Yu Li <liyu@apache.org>
2020-11-04 18:25:08 +08:00
Andrew Purtell 1cc39ffd59
HBASE-25212 [branch-2] Optionally abort requests in progress after deciding a region should close (#2575)
If hbase.regionserver.close.wait.abort is set to true, interrupt RPC
handler threads holding the region close lock.

Until requests in progress can be aborted, wait on the region close lock for
a configurable interval (specified by hbase.regionserver.close.wait.time.ms,
default 60000 (1 minute)). If we have failed to acquire the close lock after
this interval elapses, if allowed (also specified by
hbase.regionserver.close.wait.abort), abort the regionserver.

We will attempt to interrupt any running handlers every
hbase.regionserver.close.wait.interval.ms (default 10000 (10 seconds)) until
either the close lock is acquired or we reach the maximum wait time.

Define a subset of region operations as interruptible. Track threads holding
the close lock transiting those operations. Set the thread interrupt status
of tracked threads when trying to close the region. Use the thread interrupt
status where safe to break out of request processing.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-03 15:21:32 -08:00
bsglz 3a45fc17e5
HBASE-24967 The table.jsp cost long time to load if the table include closed regions (#2613) 2020-11-01 21:11:15 +08:00
Nick Dimiduk 1c7d472537 HBASE-25167 Normalizer support for hot config reloading (#2523)
Wire up the `ConfigurationObserver` chain for
`RegionNormalizerManager`. The following configuration keys support
hot-reloading:
 * hbase.normalizer.throughput.max_bytes_per_sec
 * hbase.normalizer.split.enabled
 * hbase.normalizer.merge.enabled
 * hbase.normalizer.min.region.count
 * hbase.normalizer.merge.min_region_age.days
 * hbase.normalizer.merge.min_region_size.mb

Note that support for `hbase.normalizer.period` is not provided
here. Support would need to be implemented generally for the `Chore`
subsystem.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Aman Poonia <aman.poonia.29@gmail.com>
2020-10-30 15:32:36 -07:00
Nick Dimiduk b84e2f598b HBASE-24419 Normalizer merge plans should consider more than 2 regions when possible
The core change here is to the loop in
`SimpleRegionNormalizer#computeMergeNormalizationPlans`. It's a nested
loop that walks the table's region chain once, looking for contiguous
sequences of regions that meet the criteria for merge. The outer loop
tracks the starting point of the next sequence, the inner loop looks
for the end of that sequence. A single sequence becomes an instance of
`MergeNormalizationPlan`.

Signed-off-by: Huaxiang Sun <huaxiangsun@apache.org>
2020-10-30 10:43:43 -07:00
Pankaj 40608c33ef HBASE-24977 Meta table shouldn't be modified as read only (#2537)
Signed-off-by: stack <stack@apache.org>
2020-10-29 16:35:07 -07:00
GeorryHuang 69f282e21a
HBASE-25090 CompactionConfiguration logs unrealistic store file sizes (#2595)
Signed-off-by: stack <stack@apache.org>
2020-10-29 13:09:18 -07:00
Peter Somogyi b155c62f38 HBASE-25224 Maximize sleep for checking meta and namespace regions availability (#2593)
Signed-off-by: Michael Stack <stack@apache.org>
2020-10-28 18:09:16 +01:00
Minji Kim 970aaf38bd HBASE-25223 Use try-with-resources statement (#2592)
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-10-28 09:20:35 -07:00
niuyulin bfe92ea53b HBASE-25201 YouAreDeadException should be moved to hbase-server module (#2581)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-28 15:02:30 +08:00
niuyulin 1bf68514ea HBASE-25176 MasterStoppedException should be moved to hbase-client module (#2538)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-28 15:02:20 +08:00
Reid Chan 70631e901f HBASE-25189 [Metrics] Add checkAndPut and checkAndDelete latency metrics at table level (#2549)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-10-25 17:47:58 +08:00
Toshihiro Suzuki f7e6143f49 HBASE-25206 Data loss can happen if a cloned table loses original split region(delete table) (#2569)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-24 23:04:33 +08:00
Sandeep Pal 3862dfc930
HBASE-25193: Add support for row prefix and type in the WAL Pretty Printer
Closes #2556

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-10-23 12:47:31 +05:30
Duo Zhang fdea847130 HBASE-25207 Revisit the implementation and usage of RegionStates.include (#2571)
Remove the RegionStates.include method as its name is ambiguous.
Add more comments to describe the logic on why we filter region like
this.

Signed-off-by: Toshihiro Suzuki <brfrn169@gmail.com>
2020-10-22 22:01:17 +08:00
Duo Zhang 83b189fb46
HBASE-25186 Addendum remove extra sleep to fix TestMasterRegionWALCleaner (#2577)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-10-22 10:54:57 +08:00
Duo Zhang 9d61afae6e
HBASE-25186 Addendum fix compile error (#2572)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-10-21 16:47:56 +08:00
shahrs87 06bb12ed83
HBASE-25179 : Fix Assert format in HFilePerformanceEvaluation class
Closes #2551

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-10-16 12:09:10 +05:30
Nick Dimiduk fc4957f456 HBASE-24628 Region normalizer now respects a rate limit (HMaster chore shutdown NPE ADDENDUM) (#2540)
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-15 22:06:40 +08:00
Duo Zhang 2f566dc234 HBASE-25186 TestMasterRegionOnTwoFileSystems is failing after HBASE-25065 (#2544)
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
2020-10-15 10:45:09 +08:00
XinSun 4e6c52c319 HBASE-25117 ReplicationSourceShipper thread can not be finished (#2521)
Signed-off-by: Wellington Chevreuil <wellington.chevreuil@gmail.com>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>

(cherry picked from commit 78b7244091)
2020-10-14 18:11:10 +01:00
Wellington Ramos Chevreuil 5d65bd45fa
HBASE-24877 Add option to avoid aborting RS process upon uncaught exceptions happen on replication source (#2399)
Signed-off-by: stack <stack@apache.org>
2020-10-14 10:54:23 +01:00
bsglz dd6974fda8
HBASE-24875 Remove the force param for unassign since it dose not take effect any more (#2254)
Modified compared to main branch to deprecate obviated MasterObserver interface methods instead of remove them.

Signed-off-by: Sean Busbey <busbey@apache.org>
(cherry picked from commit c5ca191921)

 Conflicts:
	hbase-client/src/main/java/org/apache/hadoop/hbase/client/AdminOverAsyncAdmin.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdmin.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestWithDisabledAuthorization.java
2020-10-13 16:29:03 -05:00
niuyulin b0d140f40a
HBASE-25093 the RSGroupBasedLoadBalancer#retainAssignment throws NPE (#2534)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-13 21:25:09 +08:00
Duo Zhang 37c2ffdc2b HBASE-25164 Make ModifyTableProcedure support changing meta replica count (#2513)
Signed-off-by: Michael Stack <stack@apache.org>
2020-10-13 10:13:48 +08:00
ramkrish86 ed7aa8e369
HBASE-25065 - WAL archival to be done by a separate thread (#2531)
* HBASE-25065 - WAL archival to be done by a separate thread

* Fix checkstyle comments

* Fix compile issue

* Fix checkstyle and make the failing test more reliable

* Remove unused import
2020-10-12 21:57:25 +05:30
Nick Dimiduk 2253800ee1 HBASE-24628 Region normalizer now respects a rate limit
Implement a rate limiter for the normalizer. Implemented in terms of
MB/sec of affacted region size (the same metrics used to make
normalization decisions). Uses Guava `RateLimiter` to perform the
resource accounting. `RateLimiter` works by blocking (uninterruptible
😖) the calling thread. Thus, the whole construction of the normalizer
subsystem needed refactoring. See the provided `package-info.java` for
an overview of this new structure.

Introduces a new configuration,
`hbase.normalizer.throughput.max_bytes_per_sec`, for specifying a
limit on the throughput of actions executed by the normalizer. Note
that while this configuration value is in bytes, the minimum honored
valued `1_000_000`. Supports values configured using the
human-readable suffixes honored by `Configuration.getLongBytes`

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
Signed-off-by: Michael Stack <stack@apache.org>
2020-10-09 09:47:45 -07:00
Michael Stack 45e10d43fe HBASE-22976 [HBCK2] Add RecoveredEditsPlayer (#2504)
Make it so WALPlayer can replay recovered.edits files.

hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/WALInputFormat.java
 Allow for WAL files that do NOT have a startime in their name.
 Use the 'generic' WAL-filename parser instead of the one that
 used be local here. Implement support for 'startTime' filter.
 Previous was just not implemented.

hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java
 Checkstyle.

hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractFSWALProvider.java
 Use the new general WAL name timestamp parser.

hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WAL.java
 Utility for parsing timestamp from WAL filename.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRecoveredEdits.java
 Export attributes about the local recovered.edits test file
 so other tests can play with it.

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-10-09 08:54:53 -07:00
Nick Dimiduk 627002d279 HBASE-25156 TestMasterFailover.testSimpleMasterFailover is flaky (#2507)
Change the test to wait for evidence that the active master has seen
that the backup master killed by the test has gone away. This is done
before proceeding to validate that the dead backup is correctly
omitted from the ClusterStatus report.

Also, minor fixup to several assertions, using `assertEquals` instead
of `assertTrue(...equals(...))` and correcting expected vs. actual
ordering of assertion arguments.

Signed-off-by: Michael Stack <stack@apache.org>
2020-10-08 14:27:05 -07:00
Michael Stack d07d181ea4 HBASE-25165 Change 'State time' in UI so sorts (#2508)
Display startcode in iso8601.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-10-08 09:11:27 -07:00
Duo Zhang 703b0613bd HBASE-25124 Support changing region replica count without disabling table (#2497)
Signed-off-by: stack <stack@apache.org>
2020-10-08 22:00:05 +08:00
Toshihiro Suzuki 481574072c HBASE-25160 Refactor AccessController and VisibilityController (#2506)
Signed-off-by: stack <stack@apache.org>
2020-10-08 17:07:07 +09:00
Duo Zhang 68fe79a6cf Revert "HBASE-24813 ReplicationSource should clear buffer usage on Replicatio… (#2453)"
This reverts commit dedb630da0.
2020-10-06 21:15:33 +08:00
Andor Molnár 269ea09499 HBASE-25146 Add extra logging at info level to HFileCorruptionChecker in order to report progress (#2503)
Signed-off-by: Wellington Chevreuil <wellington.chevreuil@gmail.com>
Signed-off-by: Mate Szalay-Beko <szalay.beko.mate@gmail.com>
Signed-off-by: Balazs Meszaros <meszibalu@apache.org>
2020-10-06 09:40:34 +02:00
Duo Zhang 4df536e31d HBASE-25121 Refactor MetaTableAccessor.addRegionsToMeta and its usage places (#2476)
Signed-off-by: stack <stack@apache.org>
2020-10-05 22:05:01 +08:00
Viraj Jasani 283bb52f7e
HBASE-25147 : Serialize regionNames in ReopenTableRegionsProcedure
Closes #2494

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-05 15:50:57 +05:30
niuyulin 0ab78378a9
HBASE-25115 HFilePrettyPrinter can't seek to the row which is the first row of a hfile
Closes #2473

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-10-04 16:08:47 +05:30
Duo Zhang fe937242c1 HBASE-25080 Should not use AssignmentManager to test whether a table is enabled or not (#2436)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-10-03 21:16:47 +08:00
Michael Stack 8bfd7cc408 HBASE-25091 Move LogComparator from ReplicationSource to AbstractFSWALProvider#.WALsStartTimeComparator (#2449)
Give the comparator a more descriptive name, a better location,
and make it work even when passed hbase:meta WAL files.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-10-01 10:12:16 -07:00
ramkrish86 11cbca1a5e
HBASE-25135 Convert the internal seperator while emitting the memstore read metrics to # (#2486) (#2489)
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2020-10-01 18:18:31 +05:30
Wellington Ramos Chevreuil dedb630da0
HBASE-24813 ReplicationSource should clear buffer usage on Replicatio… (#2453)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-10-01 08:03:46 +01:00
Duo Zhang f973a8087f HBASE-25103 Remove ZNodePaths.metaReplicaZNodes (#2464)
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-09-29 15:51:06 +08:00
Toshihiro Suzuki 4941fc02ea HBASE-25096 WAL size in RegionServer UI is wrong (#2456)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-28 14:49:04 +09:00
Mohammad Arshad bd2dd21e48 HBASE-25077: hbck.jsp page loading fails, logs NPE in master log. (#2433)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
2020-09-27 14:06:29 +08:00
Joseph295 6b93373a0a
HBASE-25088 CatalogFamilyFormat/MetaTableAccessor.parseRegionInfoFromRegionName incorrectly setEndKey to regionId (#2462)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-09-27 11:05:16 +08:00
meiyi 0e6775c919 HBASE-25097 Wrong RIT page number in Master UI (#2458)
Signed-off-by: Viraj Jasani<virajjasani@apache.org>
2020-09-25 18:41:36 +08:00
Bharath Vissapragada 505ceacb4b
HBASE-25082: Per table WAL metrics: appendCount and appendSize (#2440)
Signed-off-by: Geoffrey Jacoby <gjacoby@apache.org>
Signed-off-by: Ankit Jain <jain.ankit@salesforce.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 56c7505f8f)
2020-09-23 21:07:44 -07:00
Andrew Purtell b2f2c79d8f HBASE-25079 Upgrade Bootstrap to 3.3.7 (#2442)
Signed-off-by: Viraj Jasani<virajjasani@apache.org>
2020-09-23 14:57:42 -07:00
stack 7a1fda667e Revert "HBASE-25068 Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space"
This reverts commit b5a242f42a.
2020-09-22 20:51:13 -07:00
stack b5a242f42a HBASE-25068 Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space
Pass WALFactory to Replication instead of WALProvider. WALFactory has all
WALProviders in it, not just the user-space WALProvider. Do this so
ReplicationService has access to all WALProviders in the Server (To be
exploited by the follow-on patch in HBASE-25055)

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-09-22 12:07:00 -07:00