Commit Graph

18226 Commits

Author SHA1 Message Date
huaxiangsun 8482a285c6 HBASE-26864 SplitTableRegionProcedure calls openParentRegions() at a … (#4261)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2022-03-26 12:15:24 -07:00
Andrew Purtell 6902cb2568 HBASE-26826 Backport StoreFileTracker (HBASE-26067, HBASE-26584, and others) to branch-2.5
Previous cherry picks:

commit 6aaef89 HBASE-26064 Introduce a StoreFileTracker to abstract the store file tracking logic
commit 43b40e9 HBASE-25988 Store the store file list by a file #3578)
commit 6e05376 HBASE-26079 Use StoreFileTracker when splitting and merging #3617)
commit 090b2fe HBASE-26224 HBASE-26224 Introduce a MigrationStoreFileTracker to support migratin… #3656)
commit 0ee1689 HBASE-26246 Persist the StoreFileTracker configurations to TableDescriptor when creating table #3666)
commit 2052e80 HBASE-26248 Should find a suitable way to let users specify the store… #3665)
commit 5ff0f98 HBASE-26264 Add more checks to prevent misconfiguration on store file… #3681)
commit fc4f6d1 HBASE-26280 HBASE-26280 Use store file tracker when snapshoting #3685)
commit 06db852 HBASE-26326 CreateTableProcedure fails when FileBasedStoreFileTracker… #3721)
commit e4e7cf8 HBASE-26386 Refactor StoreFileTracker implementations to expose the s… #3774)
commit 08d1171 HBASE-26328 Clone snapshot doesn't load reference files into FILE SFT impl #3749)
commit 8bec26e HBASE-26263 [Rolling Upgrading] Persist the StoreFileTracker configur… #3700)
commit a288365 HBASE-26271: Cleanup the broken store files under data directory #3786)
commit d00b5fa HBASE-26454 CreateTableProcedure still relies on temp dir and renames… #3845)
commit 771e552 HBASE-26286: Add support for specifying store file tracker when restoring or cloning snapshot
commit f16b7b1 HBASE-26265 Update ref guide to mention the new store file tracker im… #3942)
commit 755b3b4 HBASE-26585 Add SFT configuration to META table descriptor when creating META #3998)
commit 39c42c7 HBASE-26639 The implementation of TestMergesSplitsAddToTracker is pro… #4010)
commit 6e1f5b7 HBASE-26586 Should not rely on the global config when setting SFT implementation for a table while upgrading #4006)
commit f1dd865 HBASE-26654 ModifyTableDescriptorProcedure shoud load TableDescriptor… #4034)
commit 8fbc9a2 HBASE-26674 Should modify filesCompacting under storeWriteLock #4040)
commit 5aa0fd2 HBASE-26675 Data race on Compactor.writer #4035)
commit 3021c58 HBASE-26700 The way we bypass broken track file is not enough in Stor… #4055)
commit a8b68c9 HBASE-26690 Modify FSTableDescriptors to not rely on renaming when wr… #4054)
commit dffeb8e HBASE-26587 Introduce a new Admin API to change SFT implementation (#… #4080)
commit b265fe5 HBASE-26673 Implement a shell command for change SFT implementation #4113)
commit 4cdb380 HBASE-26640 Reimplement master local region initialization to better … #4111)
commit 77bb153 HBASE-26707: Reduce number of renames during bulkload (#4066) #4122)
commit a4b192e HBASE-26611 Changing SFT implementation on disabled table is dangerous #4082)
commit d3629bb HBASE-26837 Set SFT config when creating TableDescriptor in TestClone… #4226)
commit 541d748 HBASE-26881 Backport HBASE-25368 to branch-2 (#4267)

Fixups for precommit error prone, checkstyle, and javadoc warnings after applying cherry picks.

Signed-off-by: Josh Elser <elserj@apache.org>
Reviewed-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
2022-03-26 09:30:58 -07:00
Wellington Ramos Chevreuil 2bbddfee1e HBASE-26881 Backport HBASE-25368 to branch-2 (#4267)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-26 09:30:58 -07:00
Duo Zhang 6a4b2428df HBASE-26837 Set SFT config when creating TableDescriptor in TestCloneSnapshotProcedure (#4226)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
(cherry picked from commit 5b28d76652)
2022-03-26 09:30:58 -07:00
Duo Zhang 9c8d1e4625 HBASE-26611 Changing SFT implementation on disabled table is dangerous (#4082)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2022-03-26 09:30:58 -07:00
BukrosSzabolcs 4f9fbd8d58 HBASE-26707: Reduce number of renames during bulkload (#4066) (#4122)
Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>

Conflicts:
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestBulkloadBase.java
2022-03-26 09:30:58 -07:00
Duo Zhang c76c08308c HBASE-26640 Reimplement master local region initialization to better work with SFT (#4111)
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2022-03-26 09:30:58 -07:00
LiangJun He 4c0786e2b0 HBASE-26673 Implement a shell command for change SFT implementation (#4113)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-26 09:30:58 -07:00
Duo Zhang 6c43e62163 HBASE-26587 Introduce a new Admin API to change SFT implementation (#4030) (#4080)
Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
Reviewed-by: Josh Elser <elserj@apache.org>
2022-03-26 09:30:58 -07:00
Duo Zhang f0e1bc81f9 HBASE-26690 Modify FSTableDescriptors to not rely on renaming when writing TableDescriptor (#4054)
Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
2022-03-26 09:30:58 -07:00
Duo Zhang 46c10f78ff HBASE-26700 The way we bypass broken track file is not enough in StoreFileListFile (#4055)
Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
2022-03-26 09:30:58 -07:00
Duo Zhang 520b7e7c55 HBASE-26675 Data race on Compactor.writer (#4035)
Signed-off-by: Xin Sun <ddupgs@gmail.com>
2022-03-26 09:30:58 -07:00
Duo Zhang 9435769420 HBASE-26674 Should modify filesCompacting under storeWriteLock (#4040)
Signed-off-by: Josh Elser <elserj@apache.org>
2022-03-26 09:30:58 -07:00
Duo Zhang af921c3546 HBASE-26654 ModifyTableDescriptorProcedure shoud load TableDescriptor while executing (#4034)
Signed-off-by: GeorryHuang <huangzhuoyue@apache.org>
2022-03-26 09:30:58 -07:00
Duo Zhang 90981be650 HBASE-26586 Should not rely on the global config when setting SFT implementation for a table while upgrading (#4006)
Signed-off-by: GeorryHuang <huangzhuoyue@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
2022-03-26 09:30:58 -07:00
Duo Zhang 5ffa8315cf HBASE-26639 The implementation of TestMergesSplitsAddToTracker is problematic (#4010)
Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>

Conflicts:
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/storefiletracker/TestStoreFileTracker.java
2022-03-26 09:30:58 -07:00
Wellington Ramos Chevreuil 45d7f3118a HBASE-26585 Add SFT configuration to META table descriptor when creating META (#3998)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>

(cherry picked from commit baeb51ff8a)
2022-03-26 09:30:58 -07:00
Wellington Ramos Chevreuil 4776c88e82 HBASE-26265 Update ref guide to mention the new store file tracker im… (#3942) 2022-03-26 09:30:58 -07:00
BukrosSzabolcs 16ad5f777d HBASE-26286: Add support for specifying store file tracker when restoring or cloning snapshot
Closes #3851

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2022-03-26 09:30:58 -07:00
Wellington Ramos Chevreuil 6a794e397b HBASE-26454 CreateTableProcedure still relies on temp dir and renames… (#3845)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-26 09:30:58 -07:00
BukrosSzabolcs d6a409a695 HBASE-26271 Cleanup the broken store files under data directory (#3786)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>

Conflicts:
	hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
	hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
2022-03-26 09:30:58 -07:00
GeorryHuang baac21b047 HBASE-26263 [Rolling Upgrading] Persist the StoreFileTracker configurations to TableDescriptor for existing tables (#3700)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Reviewed-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
2022-03-26 09:30:58 -07:00
Wellington Ramos Chevreuil b5a3225999 HBASE-26328 Clone snapshot doesn't load reference files into FILE SFT impl (#3749)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-26 09:30:58 -07:00
Duo Zhang 08d108d879 HBASE-26386 Refactor StoreFileTracker implementations to expose the set method (#3774)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2022-03-26 09:30:58 -07:00
Wellington Ramos Chevreuil 4b493b226b HBASE-26326 CreateTableProcedure fails when FileBasedStoreFileTracker… (#3721)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2022-03-26 09:30:58 -07:00
Duo Zhang 4ae866f3bc HBASE-26280 Use store file tracker when snapshoting (#3685)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Reviewed-by: Josh Elser <elserj@apache.org>
2022-03-26 09:30:58 -07:00
Duo Zhang 9ed3f3f161 HBASE-26264 Add more checks to prevent misconfiguration on store file tracker (#3681)
Signed-off-by: Josh Elser <elserj@apache.org>
2022-03-26 09:30:58 -07:00
Duo Zhang 864fb9a8ca HBASE-26248 Should find a suitable way to let users specify the store file tracker implementation (#3665)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>

Conflicts:
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/storefiletracker/TestRegionWithFileBasedStoreFileTracker.java
2022-03-26 09:30:58 -07:00
Wellington Ramos Chevreuil b80efeae09 HBASE-26246 Persist the StoreFileTracker configurations to TableDescriptor when creating table (#3666)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-26 09:30:58 -07:00
Duo Zhang b7c1adbbfb HBASE-26224 Introduce a MigrationStoreFileTracker to support migrating from different store file tracker implementations (#3656)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>

Conflicts:
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/storefiletracker/TestMigrationStoreFileTracker.java
2022-03-26 09:30:58 -07:00
Andrew Purtell d08857df53 HBASE-26079 Use StoreFileTracker when splitting and merging (#3617)
Signed-off-by: Duo Zhang <zhangduo@apache.org>

Conflicts:
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHStoreFile.java
2022-03-26 09:30:58 -07:00
Duo Zhang 60135108f4 HBASE-25988 Store the store file list by a file (#3578)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2022-03-26 09:30:58 -07:00
Duo Zhang 073656bf88 HBASE-26064 Introduce a StoreFileTracker to abstract the store file tracking logic
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>

Conflicts:
	hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreCompactor.java
	hbase-server/src/main/java/org/apache/hadoop/hbase/mob/DefaultMobStoreFlusher.java
	hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultStoreEngine.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScannerClosure.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStripeStoreEngine.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestDateTieredCompactor.java
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/compactions/TestStripeCompactor.java
2022-03-26 09:30:58 -07:00
Sean Busbey 64a2e9c088 HBASE-26871 shaded mapreduce and shaded byo-hadoop client artifacts contain no classes (#4279)
- revert poms for both modules
- add a test to the shaded artifact validation that checks for java classes

Signed-off-by: Andrew Purtell <apurtell@apache.org>

Conflicts:
	hbase-shaded/hbase-shaded-client-byo-hadoop/pom.xml
	hbase-shaded/hbase-shaded-mapreduce/pom.xml
2022-03-26 09:12:05 -07:00
Bryan Beaudreault cfd0a1786a HBASE-26876 Use toStringBinary for rowkey in RegionServerCallable error string (#4269)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
2022-03-26 06:49:42 +08:00
Bryan Beaudreault c41c18bb8d HBASE-26875 RpcRetryingCallerImpl translateException ignores return value of recursive call (#4270)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-25 09:01:27 -07:00
Bryan Beaudreault 1404539731 HBASE-26878 TableInputFormatBase should cache RegionSizeCalculator (#4271)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-24 16:16:44 -07:00
Xiaolin Ha 7906703569 HBASE-26175 MetricsHBaseServer should record all kinds of Exceptions (#4248)
Signed-off-by: Pankaj Kumar <pankajkumar@apache.org>
2022-03-24 19:10:07 +08:00
Bryan Beaudreault 2d1dc5349e HBASE-21065 Try ROW_INDEX_V1 encoding on meta table (#4268)
Set encoding and blooms on meta as default. Also shutdown access to the
initial meta schema creating method; get from TableDescriptors if you
need access to schema or edit it as you would any other table if you
want to edit it.

Co-authored-by: Michael Stack <stack@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-23 16:53:31 -07:00
chenglei 1bee70578d
HBASE-26869 RSRpcServices.scan should deep clone cells when RpcCallContext is null (#4266) 2022-03-23 19:32:46 +08:00
Nick Dimiduk fc9aecbaab HBASE-26531 Trace coprocessor exec endpoints (addendum)
Stuck by this again. branch-2 needs this configuration instance to be initialized as an
HBaseConfiguration.
2022-03-22 14:10:39 +01:00
Nick Dimiduk b910b11731 HBASE-26531 Trace coprocessor exec endpoints
Trace table ExecService invocations as table operations. Ensure span relationships for both table
and master invocations.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-22 14:10:39 +01:00
Nick Dimiduk ae2000498a HBASE-26834 Adapt ConnectionRule for both sync and async connections
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-21 16:52:35 +01:00
Duo Zhang afd10a6fbd HBASE-26870 Log4j2 integration is incorrect in nighly's client integration test (#4250)
Signed-off-by: Sean Busbey <busbey@apache.org>
(cherry picked from commit 8640453d70)
2022-03-21 11:19:04 +08:00
Duo Zhang c2ab765334 Revert "HBASE-26813 Remove javax.ws.rs-api dependency (#4191)"
MiniYARNCluster needs it

This reverts commit f497465d38.
2022-03-19 19:46:08 +08:00
Nick Dimiduk b11b18bf91 HBASE-26855 Delete unnecessary dependency on jaxb-runtime jar (#4236)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-18 19:15:05 +01:00
Duo Zhang 31dd5ec054 HBASE-26861 Fix flaky TestSnapshotFromMaster.testSnapshotHFileArchiving (#4242)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
(cherry picked from commit 520ad638d9)
2022-03-19 00:46:36 +08:00
Duo Zhang 6be8c33789 HBASE-26858 Refactor TestMasterRegionOnTwoFileSystems to avoid dead loop (#4238)
Signed-off-by: Xiaolin Ha <haxiaolin@apache.org>
(cherry picked from commit 31db8e3b3b)
2022-03-19 00:46:08 +08:00
Duo Zhang 2e8f0a0326 HBASE-26833 Addendum fix NPE in TestReplicationSource.testTerminateClearsBuffer
(cherry picked from commit bd65ad42a2)
2022-03-17 23:23:20 +08:00
Duo Zhang c413bb1120 HBASE-26848 Addendum make tmpdir absolute
(cherry picked from commit caea843e94)
2022-03-17 13:03:54 +08:00