Apache
/
hbase
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
18,220 Commits 95 Branches 866 Tags 550 MiB
Commit Graph

8737 Commits

Author SHA1 Message Date
Nick Dimiduk d4f2b66a43
HBASE-26474 Implement connection-level attributes (#4014) 
Add support for `db.system`, `db.connection_string`, `db.user`.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.org>
Co-authored-by: Josh Elser <josh.elser@gmail.com>
 2022-01-18 12:29:35 -08:00
Duo Zhang f1dd865c35 HBASE-26654 ModifyTableDescriptorProcedure shoud load TableDescriptor while executing (#4034) 
Signed-off-by: GeorryHuang <huangzhuoyue@apache.org>
 2022-01-16 16:44:08 +08:00
xianjingfeng cf4a8266eb HBASE-26671 Misspellings of hbck usage (#4033) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2022-01-16 16:34:24 +08:00
Yutong Xiao 933fcdbe7c
Backport HBASE-26551 Add FastPath feature to HBase RWQueueRpcExecutor to branch-2 (#4027) 
Signed-off-by: Reid Chan <reidchan@apache.org>
 2022-01-15 17:12:07 +08:00
Nick Dimiduk 2a42e8ed7a HBASE-26616 Refactor code related to ZooKeeper authentication (#3973) 
This refactor reduces the size and scope of the `ZKUtil` class. The core of this refactor is
moving the `login*` methods from `ZKUtil` into their own class, `ZKAuthentication`. The class
`JaasConfiguration` is also moved along with them.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2022-01-10 10:11:10 -08:00
Xiaolin Ha d9fae5cb67 HBASE-26397 Display the excluded datanodes on regionserver UI (#3990) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2022-01-10 13:02:31 +08:00
Duo Zhang 2f0b5ae59b HBASE-26643 LoadBalancer should not return empty map (#4005) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2022-01-08 00:39:45 +08:00
Duo Zhang 6e1f5b7fec HBASE-26586 Should not rely on the global config when setting SFT implementation for a table while upgrading (#4006) 
Signed-off-by: GeorryHuang <huangzhuoyue@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
 2022-01-07 23:47:38 +08:00
Duo Zhang 39c42c7dc2 HBASE-26639 The implementation of TestMergesSplitsAddToTracker is problematic (#4010) 
Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
 2022-01-07 23:14:57 +08:00
Xiaolin Ha 7db277737a
HBASE-26347 Support detect and exclude slow DNs in fan-out of WAL (#3800) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2022-01-06 11:45:38 +08:00
Wellington Ramos Chevreuil 755b3b4cb8 HBASE-26585 Add SFT configuration to META table descriptor when creating META (#3998) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>

(cherry picked from commit baeb51ff8a)
 2022-01-05 13:08:31 +00:00
Xiaolin Ha b2e65b7ce3
HBASE-26578 CompactedHFilesDischarger should be shutdown when aborting regionserver (#4001) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2022-01-05 15:26:49 +08:00
chenglei fb1c88adac HBASE-26567 Remove IndexType from ChunkCreator (#3947) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2022-01-05 00:08:16 +08:00
BukrosSzabolcs 53285d8e2a HBASE-26286: Add support for specifying store file tracker when restoring or cloning snapshot 
Closes #3851

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
 2022-01-04 22:58:03 +08:00
Wellington Ramos Chevreuil 48e28917e0 HBASE-26454 CreateTableProcedure still relies on temp dir and renames… (#3845) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2022-01-04 22:58:03 +08:00
BukrosSzabolcs 68252e17cb HBASE-26271 Cleanup the broken store files under data directory (#3786) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
 2022-01-04 22:58:03 +08:00
GeorryHuang c4325ff088 HBASE-26263 [Rolling Upgrading] Persist the StoreFileTracker configurations to TableDescriptor for existing tables (#3700) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Reviewed-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
 2022-01-04 22:58:03 +08:00
Wellington Ramos Chevreuil 80b04229a2 HBASE-26328 Clone snapshot doesn't load reference files into FILE SFT impl (#3749) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2022-01-04 22:58:03 +08:00
Duo Zhang ca07fb869f HBASE-26386 Refactor StoreFileTracker implementations to expose the set method (#3774) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2022-01-04 22:58:03 +08:00
Wellington Ramos Chevreuil 030a2a83ca HBASE-26326 CreateTableProcedure fails when FileBasedStoreFileTracker… (#3721) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
 2022-01-04 22:58:03 +08:00
Duo Zhang b812a277ab HBASE-26280 Use store file tracker when snapshoting (#3685) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Reviewed-by: Josh Elser <elserj@apache.org>
 2022-01-04 22:58:03 +08:00
Duo Zhang dcaea11a52 HBASE-26264 Add more checks to prevent misconfiguration on store file tracker (#3681) 
Signed-off-by: Josh Elser <elserj@apache.org>
 2022-01-04 22:58:03 +08:00
Duo Zhang 49c40998ef HBASE-26248 Should find a suitable way to let users specify the store file tracker implementation (#3665) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2022-01-04 22:58:03 +08:00
Wellington Ramos Chevreuil 26f263b792 HBASE-26246 Persist the StoreFileTracker configurations to TableDescriptor when creating table (#3666) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2022-01-04 22:58:03 +08:00
Duo Zhang f298f9cacd HBASE-26224 Introduce a MigrationStoreFileTracker to support migrating from different store file tracker implementations (#3656) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2022-01-04 22:58:03 +08:00
Wellington Ramos Chevreuil e89afdfa08 HBASE-26079 Use StoreFileTracker when splitting and merging (#3617) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2022-01-04 22:58:03 +08:00
Duo Zhang 826e54ce78 HBASE-25988 Store the store file list by a file (#3578) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2022-01-04 22:58:03 +08:00
Duo Zhang 04e1980dea HBASE-26064 Introduce a StoreFileTracker to abstract the store file tracking logic 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2022-01-04 22:58:03 +08:00
Duo Zhang fff59ac083
HBASE-26642 Increase the timeout for TestStochasticLoadBalancerRegionReplicaLargeCluster (#3995) 
Signed-off-by: Yulin Niu <niuyulin@apache.org
 2022-01-03 23:13:05 +08:00
Duo Zhang 49e64ffa9b HBASE-26641 Split TestMasterFailoverWithProcedures (#3994) 
Signed-off-by: GeorryHuang <huangzhuoyue@apache.org>
 2022-01-03 00:41:19 +08:00
chenglei a67fb0a85c HBASE-26494 Using RefCnt to fix the flawed MemStoreLABImpl (#3983) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2022-01-01 20:57:25 +08:00
Duo Zhang e53712ae99
HBASE-26523 Upgrade hbase-thirdparty dependency to 4.0.1 (#3988) 
Signed-off-by: GeorryHuang <huangzhuoyue@apache.org>
 2021-12-31 12:10:08 +08:00
meiyi 5b08db92c7 HBASE-26625 ExportSnapshot tool failed to copy data files for tables with merge region (#3981) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-12-30 11:08:36 +08:00
Richard Marscher 9890ac9639 HBASE-26623 Report CallDroppedException in exception metrics (#3980) 
`CallDroppedException` can be thrown with `CallRunner.drop()` by queue implementations that decide to drop calls to groom the RPC call backlog. The LifoCoDel queue does this I believe and with Pluggable queue it's possible for 3rd party queue implementations to be using `drop()` for similar reasons. It would be nice for the server to be tracking these exceptions in metrics since otherwise you might have to do some extra lifting on the client side.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Reviewed-by: Bryan Beaudreault <bbeaudreault@hubspot.com>
 2021-12-30 00:05:04 +08:00
Yutong Xiao c6a287930b
HBASE-26564 Retire the method visitLogEntryBeforeWrite without RegionInfo in WALActionListner (#3985) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-12-28 21:42:00 +08:00
meiyi 7fcf8e84a5
HBASE-26615 Snapshot referenced data files are deleted when delete a table with merge regions (#3976) 
Signed-off-by: GeorryHuang <huangzhuoyue@apache.org>
 2021-12-23 18:51:28 +08:00
Duo Zhang c6ac99684c HBASE-26609 Round the size to MB or KB at the end of calculation in HRegionServer.createRegionLoad (#3967) 
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
 2021-12-23 11:58:29 +08:00
Samir Khan 55ddcbaaf5 HBASE-26598 Fix excessive connections in MajorCompactor (#3961) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-12-23 11:55:21 +08:00
zhengzhuobinzzb 68d8573c09 HBASE-26580 The message of StoreTooBusy is confused (#3949) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Reviewed-by: Bryan Beaudreault <bbeaudreault@hubspot.com>
 2021-12-18 23:33:58 +08:00
zhengzhuobinzzb 12a286ca66 HBASE-26579 Set storage policy of recovered edits when hbase.wal.storage.type is configured (#3948) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
 2021-12-18 23:27:42 +08:00
chenglei 2d5d5b2011 HBASE-26488 Memory leak when MemStore retry flushing (#3899) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-12-18 12:22:42 +08:00
Norbert Kalmar 471e398df0 HBASE-26340 TableSplit returns false size under 1MB (#3872) 
Signed-off-by: Peter Somogyi <psomogyi@apache.org
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-12-18 12:21:45 +08:00
Nick Dimiduk 9d6823236d HBASE-26542 Apply a `package` to test protobuf files (addendum) 
RpcServer identifies the services it hosts by unqualified service name. Thus, use `getName()`
instead of `getFullName()`. See also HBASE-26589.

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
 2021-12-16 16:39:40 -08:00
Josh Elser cb3155459f HBASE-26550 Make sure the master is running normally before accepting a balance command (#3928) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-12-14 23:32:24 +08:00
Bryan Beaudreault ef48a6975d
HBASE-26537: Make HBASE-15676 backwards compatible, using a flag on the proto (#3931) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2021-12-13 13:31:14 -08:00
bsglz 26e37bcbfe
HBASE-26027 The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone caused by ArrayStoreException (#3925) 
* HBASE-26027 The calling of HTable.batch blocked at AsyncRequestFutureImpl.waitUntilDone caused by ArrayStoreException
 2021-12-10 14:34:17 +08:00
Andrew Purtell b1bc5f3a5c Renumber to 2.6.0-SNAPSHOT after branching branch-2.5 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2021-12-08 16:54:32 -08:00
Guanghao Zhang 140b5d8d26 HBASE-24870 Ignore TestAsyncTableRSCrashPublish (#2248) 
Signed-off-by: meiyi <myimeiyi@gmail.com>
 2021-12-06 19:07:07 -08:00
LiangJun He ec52519f7c HBASE-26462 Should persist restoreAcl flag in the procedure state for CloneSnapshotProcedure and RestoreSnapshotProcedure (#3921) 
Signed-off-by: Yu Li <liyu@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-12-06 23:16:15 +08:00
RamanChodzka 074456e190 HBASE-26533 KeyValueScanner might not be properly closed when using InternalScan.checkOnlyMemStore() (#3917) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-12-05 22:35:07 +08:00
Tomu Tsuruhara 19d991d291 HBASE-26517 Add auth method information to AccessChecker audit log (#3897) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-12-04 23:14:42 +08:00
Bryan Beaudreault d504504875
HBASE-26304 Reflect out of band locality improvements in metrics and balancer (#3895) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-12-04 12:23:09 +08:00
Nick Dimiduk d1c68a2d5c HBASE-26471 Move tracing semantic attributes to their own class (#3896) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org>
 2021-12-01 13:24:08 -08:00
Rushabh Shah 6fc5bbc19a
HBASE-26468 Region Server doesn't exit cleanly incase it crashes. (#3865) (#3862) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Geoffrey Jacoby <gjacoby@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-12-01 11:06:07 +05:30
Duo Zhang 774484ed44 HBASE-25905 Shutdown of WAL stuck at waitForSafePoint (#3898) 
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
 2021-12-01 08:34:08 +08:00
Duo Zhang 799217ee69 HBASE-26455 TestStochasticLoadBalancerRegionReplicaWithRacks fails consistently (#3879) 
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
Signed-off-by: GeorryHuang <huangzhuoyue@apache.org>
 2021-11-28 22:23:21 +08:00
chenglei 5437532910 HBASE-26476 Make DefaultMemStore extensible for HStore.memstore (#3869) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-11-27 12:06:35 +08:00
chenglei 5101f372e4 HBASE-26465 MemStoreLAB may be released early when its SegmentScanner is scanning (#3859) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Reviewed-by: Anoop Sam John <anoopsamjohn@apache.org>
 2021-11-25 21:35:55 +08:00
Yutong Xiao 6aa2daa2c2 HBASE-26477 Fix MiniMapReduceCluster failure under Java17 (#3878) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-11-25 16:27:23 +08:00
Duo Zhang fa34cfc7d1 Revert "Backport HBASE-26477 Fix MiniMapReduceCluster failure under Java17 to branch-2 (#3878)" 
This reverts commit ccf150434d.
 2021-11-25 16:27:11 +08:00
Yutong Xiao ccf150434d
Backport HBASE-26477 Fix MiniMapReduceCluster failure under Java17 to branch-2 (#3878) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-11-25 16:25:44 +08:00
liangxs 2798ef132b HBASE-26479 Print too slow/big scan's operation_id in region server log (#3870) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-11-25 11:43:01 +08:00
Andrew Purtell 52ce87780a Revert "HBASE-26479 Print too slow/big scan's operation_id in region server log (#3870)" 
This reverts commit b437a96272.
 2021-11-24 17:34:01 -05:00
liangxs b437a96272 HBASE-26479 Print too slow/big scan's operation_id in region server log (#3870) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-11-24 23:38:16 +08:00
Duo Zhang 0a51d2b5be HBASE-26475 The flush and compact methods in HTU should skip processing secondary replicas (#3868) 
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
 2021-11-23 18:51:25 +08:00
Xiaolin Ha f8d03d1227 HBASE-26249 Ameliorate compaction made by bulk-loading files (#3831) (#3874) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-11-23 15:52:47 +08:00
haxiaolin 61406252a5 Revert "Backport HBASE-26249 Ameliorate compaction made by bulk-loading files (#3831) (#3874)" 
This reverts commit 5843a9a44c.
 2021-11-23 15:52:34 +08:00
Xiaolin Ha 5843a9a44c
Backport HBASE-26249 Ameliorate compaction made by bulk-loading files (#3831) (#3874) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-11-23 15:45:21 +08:00
zhengzhuobinzzb 4312f817f3 HBASE-26467 Fix bug for MemStoreLABImpl.forceCopyOfBigCellInto(Cell) (#3858) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Reviewed-by: chenglei <chenglei@apache.org>
 2021-11-21 21:02:18 +08:00
ss77892 e549384437 HBASE-26463 Unreadable table names after HBASE-24605 (#3853) 
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-11-21 20:49:13 +08:00
Xiaolin Ha 8d3562c6fa
HBASE-26421 Use HFileLink file to replace entire file's reference when splitting (#3854) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-11-18 21:37:10 +08:00
Josh Elser 4a8da47f74 HBASE-26267 Don't try to recover WALs from a WAL dir which doesn't exist (#3679) 
We currently cause an error to be thrown by trying to list a
non-existent directory. We see that the master region directory exists
on the filesystem, but forget to make sure that the master region's WAL
directory also exists before we try to list it.
 2021-11-16 16:27:45 -05:00
Andrew Purtell e65f28cc57 HBASE-26444 BucketCacheWriter should log only the BucketAllocatorException message, not the full stack trace (#3840) 
Signed-off-by: Anoop <anoopsamjohn@apache.org>
 2021-11-15 16:13:36 -08:00
Andrew Purtell 23024edba2 HBASE-26443 Some BaseLoadBalancer log lines should be at DEBUG level (#3838) 
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
 2021-11-15 16:06:59 -08:00
chenglei 98a4fb8086 HBASE-26438 Fix flaky test TestHStore.testCompactingMemStoreCellExceedInmemoryFlushSize (#3834) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-11-12 23:03:37 +08:00
Zhuoyue Huang ad36941932 HBASE-26450 Server configuration will overwrite HStore configuration after using shell command 'update_config' (#3843) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Baiqiang Zhao <zhaobaiqiang@apache.org>
 2021-11-12 11:43:48 +08:00
Yutong Xiao 9db7630d23 HBASE-26410 Fix HBase TestCanaryTool for Java17 (#3809) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-11-09 22:21:01 +08:00
clarax cc2e4aec87
HBASE-26337 Optimization for weighted random generators (#3828) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-11-09 21:27:44 +08:00
Andrew Purtell 1302073dfe HBASE-26430 Increase log level in DefaultHeapMemoryTuner from DEBUG to INFO (#3820) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-11-08 18:35:39 -08:00
Andrew Purtell 201b3c7fc7 HBASE-26429 HeapMemoryManager fails memstore flushes with NPE if enabled (#3819) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-11-08 18:32:17 -08:00
Xiaolin Ha 2fbb375cd5 HBASE-25322 Redundant Reference file in bottom region of split (#3814) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-11-08 14:48:55 +08:00
haxiaolin ebd89b58eb Revert "HBASE-25322 Redundant Reference file in bottom region of split (#3814)" 
This reverts commit a6edf44175.
 2021-11-08 11:40:43 +08:00
Xiaolin Ha a6edf44175 HBASE-25322 Redundant Reference file in bottom region of split (#3814) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-11-05 18:17:47 +08:00
clarax e06208f3d6
HBASE-26311 Balancer gets stuck in cohosted replica distribution (#3804) 
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.org>
 2021-11-03 11:58:38 -07:00
clarax 9eb3706021
HBASE-26309 Balancer tends to move regions to the server at the end of list (#3811) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-11-02 21:58:50 +08:00
XinSun 063e0e2e92 HBASE-26406 Can not add peer replicating to non-HBase (#3806) 
Signed-off-by: Rushabh Shah <shahrs87@gmail.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>

(cherry picked from commit b9b7fec57f)
 2021-11-02 15:03:33 +08:00
chenglei e89f62e611
HBASE-26384 Segment already flushed to hfile may still be remained in CompactingMemStore (#3779) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-11-01 23:05:57 +08:00
Andrew Purtell f2c58fcf68 HBASE-26353 Support loadable dictionaries in hbase-compression-zstd (#3787) 
ZStandard supports initialization of compressors and decompressors with a
precomputed dictionary, which can dramatically improve and speed up compression
of tables with small values. For more details, please see

  The Case For Small Data Compression
  https://github.com/facebook/zstd#the-case-for-small-data-compression

Signed-off-by: Duo Zhang <zhangduo@apache.org>

Conflicts:
	hbase-compression/hbase-compression-zstd/src/main/java/org/apache/hadoop/hbase/io/compress/zstd/ZstdCodec.java
 2021-10-27 07:57:16 -07:00
chenglei b4177577ac HBASE-26394 Cache in RSRpcServices.executeProcedures does not take effect (#3794) 
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-10-26 15:44:17 +08:00
clarax 32eb23c8f3
HBASE-26327 Replicas cohosted on a rack shouldn't keep triggering balancer (#3788) 
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.org>
 2021-10-25 15:44:11 -07:00
Anoop Sam John 444dec9289 HBASE-26190 High rate logging of BucketAllocatorException: Allocation too big - Addendum (#3792) 2021-10-24 09:22:26 +05:30
clarax f299d56934
HBASE-26308 Sum of multiplier of cost functions is not populated properly when we have a shortcut for trigger (#3783) 2021-10-22 09:46:58 -07:00
Sourabh Dhanotia eea2ea8576
HBASE-26368 Fix checkstyle issues for HRegionServer for branch-2 (#3781) (#3764) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-10-22 12:51:29 +05:30
binlijin 0eaadd92d8 HBASE-26385 Clear CellScanner when replay (#3773) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Pankaj Kumar <pankajkumar@apache.org>
 2021-10-21 21:56:39 +08:00
Andrew Purtell dce45ec16a HBASE-26383 HBCK incorrectly reports inconsistencies for recently split regions following a master failover (#3775) 
A regression was introduced by HBASE-25847 which changed regionInfo#isParentSplit
to regionState#isSplit. The region state after restart is CLOSED instead of SPLIT.
We need to check both regionState and regionInfo for split status.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-10-20 15:55:07 -07:00
Viraj Jasani 5c4a053f2a
HBASE-26371 Prioritize meta region move over other region moves in region_mover (#3771) (#3767) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2021-10-20 23:06:25 +05:30
Andrew Purtell b6bb18022f HBASE-26316 Per-table or per-CF compression codec setting overrides (#3730) 
We get and retain Compressor instances in HFileBlockDefaultEncodingContext,
and could in theory call Compressor#reinit when setting up the context,
to update compression parameters like level and buffer size, but we do
not plumb through the CompoundConfiguration from the Store into the
encoding context. As a consequence we can only update codec parameters
globally in system site conf files.

Fine grained configurability is important for algorithms like ZStandard
(ZSTD), which offers more than 20 compression levels, where at level 1
it is almost as fast as LZ4, and where at higher levels it utilizes
computationally expensive techniques to rival LZMA at compression ratio
but trades off significantly for reduced compresson throughput. The ZSTD
level that should be set for a given column family or table will vary by
use case.

Signed-off-by: Viraj Jasani <vjasani@apache.org>

Conflicts:
	hbase-compression/hbase-compression-zstd/src/main/java/org/apache/hadoop/hbase/io/compress/zstd/ZstdDecompressor.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/io/compress/HFileTestBase.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestDataBlockEncoders.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/io/encoding/TestSeekToBlockWithEncoders.java
 2021-10-19 13:59:04 -07:00
KulwantSingh011 7e574c9f1d
HBASE-26190 High rate logging of BucketAllocatorException: Allocation too big (#3752) 
Reduce the frequency of allocation failed traces by printing them
preiodically (once per minute). Record the allocation failures in the
Bucket Cache Stats and let the stat thread dump cumulative allocation
failures alongside other traces it dumps.

Also, this change adds trace for the Table name, Column Family and
HFileName for the most recent allocation failure in last 1 minute.

Signed-off-by: Anoop <anoopsamjohn@apache.org>
 2021-10-19 13:21:11 +05:30
Josh Elser 45b09aa4d7 HBASE-26350 Add a DEBUG when we fail the SASL handshake 
Closes #3743

Signed-off-by: Pankaj <pankajkumar@apache.org>
Signed-off-by: Tak Lon (Stephen) Wu <taklwu@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2021-10-14 15:27:32 -04:00
Tak Lon (Stephen) Wu 3ce9052188
HBASE-24833: Bootstrap should not delete the META table directory if … (#2237) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Zach York <zyork@apache.org>
 2021-10-12 10:56:00 -07:00
Duo Zhang 10584d70d2
HBASE-26306 Backport "HBASE-26220 Use P2P communicate between region servers to sync the list for bootstrap node" to branch-2 (#3727) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-10-10 22:56:12 +08:00
Andrew Purtell 5af02201d2 HBASE-26324 Reuse compressors and decompressors in WAL CompressionContext (#3728) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2021-10-07 09:46:38 -07:00
Andrew Purtell 18b9fa8a3c HBASE-26259 Fallback support to pure Java compression (#3691) 
This change introduces provided compression codecs to HBase as
new Maven modules. Each module provides compression codec support
that formerly required Hadoop native codecs, which in turn relies
on native code integration, which may or may not be available on
a given hardware platform or in an operational environment. We
now provide codecs in the HBase distribution for users whom for
whatever reason cannot or do not wish to deploy the Hadoop native
codecs.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>

Conflicts:
	hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileEncryption.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileSeek.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestCompressedWAL.java
 2021-10-06 13:48:33 -07:00
Duo Zhang 706082d513
HBASE-26294 Backport "HBASE-26181 Region server and master could use itself as ConnectionRegistry" to branch-2 (#3708) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-10-05 22:11:01 +08:00
imbajin 9a33e234e7 HBASE-24601: Change default Hfile storage policy from HOT to NONE for HDFS (#1940) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-10-02 16:37:39 +08:00
Yutong Xiao a2c09ae5bc HBASE-26305 Move NavigableSet add operation to writer thread in BucketCache (#3705) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Reviewed-by: chenglei <chenglei@apache.org>
 2021-10-01 21:06:57 +08:00
chenglei 0748165ab0 HBASE-26295 BucketCache could not free BucketEntry which restored fro… (#3699) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-09-29 21:46:57 +08:00
GeorryHuang 08800e19d6
HBASE-26251 StochasticLoadBalancer metrics should update even if balancer doesn't run (#3706) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Reviewed-by: Bryan Beaudreault <bbeaudreault@hubspot.com>
 2021-09-29 21:25:39 +08:00
richardantal 69749ed01e HBASE-26289 Hbase scan setMaxResultsPerColumnFamily not giving right results (#3688) 
Co-authored-by: Duo Zhang <zhangduo@apache.org>

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
 2021-09-29 10:22:24 +02:00
Duo Zhang 08a2efde16 HBASE-26293 Use reservoir sampling when selecting bootstrap nodes (#3702) 
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
 2021-09-28 23:41:28 +08:00
bitterfox 786e09a936 HBASE-26238 Short message by Result#compareResults for VerifyReplication to avoid OOME (#3647) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-09-28 23:16:20 +08:00
clarax 73f42b27bf HBASE-26297 Balancer run is improperly triggered by accuracy error of double comparison (#3698) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-09-26 23:35:50 +08:00
clarax 404789adde HBASE-26178 Improve data structure and algorithm for BalanceClusterState to improve computation speed for large cluster (#3682) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-09-25 21:48:51 +08:00
Andrew Purtell 59d133ef95 Amend HBASE-26274 Create an option to reintroduce BlockCache to mapreduce job 
Compliation fixes for CombinedBlockCache and TestClientSideRegionScanner.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2021-09-22 12:27:59 -07:00
Tak Lon (Stephen) Wu e0325c40f6 HBASE-26274 Create an option to reintroduce BlockCache to mapreduce job (#3684) 
Introduce `hfile.onheap.block.cache.fixed.size`
and default to disable. when using ClientSideRegionScanner
it will be enabled with a fixed size for caching
INDEX/LEAF_INDEX block when a client, e.g.
snapshot scanner, scans the entire HFile
and does not need to seek/reseek to index
block multiple times.

Signed-off-by: Josh Elser <elserj@apache.org>
 2021-09-22 09:38:20 -07:00
meiyi 84b5511d13 HBASE-26261 Store configuration loss when use update_config (#3664) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-09-22 18:35:28 +08:00
meiyi 8f6c5e1f02 HBASE-26270 Provide getConfiguration method for Region and Store interface (#3671) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
 2021-09-22 11:32:32 +08:00
Duo Zhang e327fa43c0 HBASE-26288 Revisit the usage of MetaTableLocator when HRegionServer.TEST_SKIP_REPORTING_TRANSITION is true (#3689) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-09-22 00:27:11 +08:00
Duo Zhang ef85204522 HBASE-26285 Remove MetaTableLocator usages in non-migration code (#3686) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-09-21 23:32:13 +08:00
chenglei b21b250570 HBASE-26281 DBB got from BucketCache would be freed unexpectedly before RPC completed (#3680) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-09-17 22:28:56 +08:00
Xiaolin Ha f670fc5d4d HBASE-26229 Limit count and size of L0 files compaction in StripeCompactionPolicy (#3646) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-09-09 18:28:45 +08:00
hapihu 68db7f5ed0 [HBASE-26243]Fix typo for file "hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java" (#3650) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-09-09 12:21:54 +08:00
Rushabh Shah 820ba21b2a HBASE-26106 AbstractFSWALProvider#getArchivedLogPath doesn't look for wal file in all oldWALs directory. (#3636) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>

Conflicts:
	hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestWALRecordReader.java
 2021-09-02 10:26:22 -07:00
XinSun ccd90269d2 HBASE-25773 TestSnapshotScannerHDFSAclController.setupBeforeClass is flaky (#3651) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 345d7256c8)
 2021-09-02 10:33:41 +08:00
Bryan Beaudreault e07928df4c HBASE-26147 Add a dry run mode to the balancer, where moves are calculated but not actually executed 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>

(Re-application of HBASE-26147 with the correct author metadata)
 2021-09-01 21:58:30 -04:00
Josh Elser 1e74f08403 Revert "HBASE-26147 Add a dry run mode to the balancer, where moves are calculated but not actually executed" 
This reverts commit 5b58d11c91.

I accidentally applied this as myself instead of as Bryan. Reverting it
to apply the patch properly.
 2021-09-01 21:57:46 -04:00
Josh Elser 5b58d11c91 HBASE-26147 Add a dry run mode to the balancer, where moves are calculated but not actually executed 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org
 2021-09-01 21:47:53 -04:00
Tak Lon (Stephen) Wu c639d6f353 HBASE-26136 Backport HBASE-25723 "Temporarily remove the trace support for RegionScanner.next" to branch-2 (#3623) 
13/17 commits of HBASE-22120, original commit 7f90c2201f

Co-authored-by: Duo Zhang <zhangduo@apache.org>

Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-09-01 15:29:09 -07:00
Tak Lon (Stephen) Wu d0a53e3f29 HBASE-26133 Backport HBASE-25591 "Upgrade opentelemetry to 0.17.1" to branch-2 (#3608) 
10/17 commits of HBASE-22120, original commit f6ff519dd0

Co-authored-by: Duo Zhang <zhangduo@apache.org>

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
 2021-09-01 15:29:09 -07:00
Tak Lon (Stephen) Wu 8b7450f5d8 HBASE-26132 Backport HBASE-25535 "Set span kind to CLIENT in AbstractRpcClient" to branch-2 (#3607) 
9/17 commits of HBASE-22120, original commit bb8c4967f8

Co-authored-by: Duo Zhang <zhangduo@apache.org>

Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-09-01 15:29:09 -07:00
Tak Lon (Stephen) Wu 9cce94a2d0 HBASE-26131 Backport HBASE-25484 "Add trace support for WAL sync" to branch-2 (#3597) 
8/17 commits of HBASE-22120, original commit 2be2c63f0d

Co-authored-by: Duo Zhang <zhangduo@apache.org>

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by:  Peter Somogyi <psomogyi@apache.org>
 2021-09-01 15:29:09 -07:00
Tak Lon (Stephen) Wu 321dcd4e77 HBASE-26130 Backport HBASE-25455 "Add trace support for HRegion read/… (#3594) 
7/17 commits of HBASE-22120, original commit 03e12bfa4a

Co-authored-by: Duo Zhang <zhangduo@apache.org>

Signed-off-by:  Duo Zhang <zhangduo@apache.org>
 2021-09-01 15:29:09 -07:00
Tak Lon (Stephen) Wu 83a27389a0 HBASE-26129 Backport HBASE-25481 "Add host and port attribute when tracing rpc call at client side" to branch-2 (#3589) 
6/17 commits of HBASE-22120, original commit ae2c62ffaa

Co-authored-by: Duo Zhang <zhangduo@apache.org>

Signed-off-by:  Peter Somogyi  <psomogyi@apache.org>
 2021-09-01 15:29:09 -07:00
Tak Lon (Stephen) Wu 58b5c007b9 HBASE-26127 Backport HBASE-23898 "Add trace support for simple apis i… (#3556) 
4/17 commits of HBASE-22120, original commit 805b2ae2ad

Co-authored-by: Duo Zhang <zhangduo@apache.org>

Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-09-01 15:29:09 -07:00
Tak Lon (Stephen) Wu 4dfc59c19e HBASE-26125 Backport HBASE-25401 "Add trace support for async call in rpc client" to branch-2 (#3543) 
2/17 commits of HBASE-22120

Co-authored-by: Duo Zhang <zhangduo@apache.org>

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
 2021-09-01 15:29:09 -07:00
Tak Lon (Stephen) Wu 665305cc3b HBASE-26124 Backport HBASE-25373 "Remove HTrace completely in code base and try to make use of OpenTelemetry" to branch-2 (#3529) 
1/17 commits of HBASE-22120

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
 2021-09-01 15:29:09 -07:00
chenglei dbf43dcfbd HBASE-26210 HBase Write should be doomed to hang when cell size exceeds InmemoryFlushSize for CompactingMemStore (#3604) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-09-01 18:17:13 +08:00
Andrew Purtell 8be2ad2d42 HBASE-25642 Fix or stop warning about already cached block (#3638) 
Our logs have as a fairly common occurrence: 2021-03-05 22:24:31,034 WARN
[StoreFileOpener-foo-1] hfile.BlockCacheUtil: Caching an already cached
block: blah.bub. This is harmless and can happen in rare cases (see HBASE-8547)

Because it is harmless, log at DEBUG level, not WARN.

Signed-off-by: Baiqiang Zhao <zhaobaiqiang@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Reviewed-by: Rushabh Shah <rushabh.shah@salesforce.com>
 2021-08-31 09:26:10 -07:00
Duo Zhang c08e877d6a HBASE-26235 We could start RegionServerTracker before becoming active master (#3645) 
Signed-off-by: Yulin Niu <niuyulin@apache.org>
 2021-08-30 12:22:41 +08:00
keyserbrian1 1e49e94051 HBASE-26232 SEEK_NEXT_USING_HINT is ignored on reversed Scans (#3639) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-08-29 21:42:48 +08:00
GeorryHuang 033ff757fd HBASE-26216 Move HRegionServer.abort(String) to Abortable as a default method (#3642) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-08-28 23:06:54 +08:00
Andrew Purtell 3b1482ef60 HBASE-26189 Reduce log level of CompactionProgress notice to DEBUG (#3579) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2021-08-27 10:15:28 -07:00
Pankaj 30bb72bdb8 HBASE-26227 Forward port HBASE-26223 test code to branch-2.4+ (#3632) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-08-27 16:46:54 +08:00
lpalasthy cd9e45ce8d HBASE-26219 Negative time is logged while waiting on regionservers (#3622) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-08-25 16:25:24 +02:00
Duo Zhang 931a7a5a61 HBASE-26180 Introduce a initial refresh interval for RpcConnectionRegistry (#3601) 
Signed-off-by: Xin Sun <ddupgs@gmail.com>
 2021-08-25 18:51:40 +08:00
GeorryHuang dacd3404a4 HBASE-26215 The backup master status page should use ActiveMasterManager instead of MasterAddressTracker (#3621) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2021-08-25 16:42:45 +08:00
Duo Zhang 5dad395869 HBASE-26173 Return only a sub set of region servers as bootstrap nodes (#3599) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2021-08-24 23:17:56 +08:00
Duo Zhang 6ad7eb88c3 HBASE-26172 Deprecated MasterRegistry (#3566) 
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2021-08-19 16:30:48 +08:00
Michael Stack 101602007a
HBASE-24337 Backport HBASE-23968 to branch-2 (#3588) 
Signed-off-by: stack <stack@apache.org>
 2021-08-18 10:19:35 -07:00
Wellington Ramos Chevreuil f2f99aea34 Amend HBASE-26187 UTs, adding a check for Split procedure completion before proceeding with test logic. (#3598) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit f2e2140b57)
 2021-08-18 16:41:08 +01:00