8084 Commits

Author SHA1 Message Date
Toshihiro Suzuki
00aa3bc9fc
HBASE-24884 BulkLoadHFilesTool/LoadIncrementalHFiles should accept -D options from command line parameters (#2275)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-08-19 19:55:34 +09:00
Nick Dimiduk
9c85f3641f HBASE-24583 Normalizer can't actually merge empty regions...
when neighbor is larger than average size

* add `testMergeEmptyRegions` to explicitly cover different
  interleaving of 0-sized regions.
* fix bug where merging a 0-size region is skipped due to large
  neighbor.
* remove unused `splitPoint` from `SplitNormalizationPlan`.
* generate `toString`, `hashCode`, and `equals` methods from Apache
  Commons Lang3 template on `SplitNormalizationPlan` and
  `MergeNormalizationPlan`.
* simplify test to use equality matching over `*NormalizationPlan`
  instances as plain pojos.
* test make use of this handy `TableNameTestRule`.
* fix line-length issues in `TestSimpleRegionNormalizer`

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
Signed-off-by: Aman Poonia <aman.poonia.29@gmail.com>
2020-08-12 10:11:40 -07:00
Viraj Jasani
8ccf643fdc
HBASE-24750 : All ExecutorService should use guava ThreadFactoryBuilder (#2214)
Closes #2196

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Ted Yu <tyu@apache.org>
Signed-off-by: niuyulin <nyl353@163.com>
2020-08-12 15:57:53 +05:30
Duo Zhang
96ea136a8e HBASE-24856 Fix error prone error in FlushTableSubprocedure (#2242)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-08-12 10:01:25 +08:00
Bharath Vissapragada
0533937b16
HBASE-24846: Address compaction races in TestFIFOCompactionPolicy (#2203)
Looped through the test 100 times and it passes. Without the patch it fails
every ~10 runs or so.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2020-08-10 17:21:27 -07:00
Bharath Vissapragada
cd97d1a452
Revert "HBASE-21905 (addendum): Address compaction races in TestFIFOCompactionPolicy (#2203)"
This reverts commit c372ff19cf42cf7aa3df9db744a8eff60577ca3e.

Reverting to apply as HBASE-24846.
2020-08-10 17:21:05 -07:00
Nick Dimiduk
17a0c2aabf HBASE-24835 Normalizer should log a successful run at INFO level
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-10 10:13:54 -07:00
Toshihiro Suzuki
22bf9a38c9
HBASE-24680 Refactor the checkAndMutate code on the server side (#2184)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2020-08-10 18:57:17 +09:00
bsglz
ebf493f075
HBASE-24694 Support flush a single column family of table (#2218)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-08-10 10:36:13 +01:00
Duo Zhang
ef7b9eb36e HBASE-23157 WAL unflushed seqId tracking may wrong when Durability.ASYNC_WAL is used (#762)
Signed-off-by: stack <stack@apache.org>
2020-08-10 10:54:30 +08:00
chenglei
457234c695 HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length(addendum) (#2055)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-08-10 10:04:05 +08:00
stack
cee7431d0a HBASE-24830 Some tests involving RS crash fail with NullPointerException after HBASE-24632 in branch-2 2020-08-09 11:16:37 -07:00
Bo Cui
11cfb0f382 HBASE-21721 FSHLog : reduce write#syncs() times (#2217)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Anoop <anoopsamjohn@apache.org>
2020-08-08 09:41:36 +05:30
Josh Elser
303db63b76 HBASE-24779 Report on the WAL edit buffer usage/limit for replication
Closes #2193

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-08-07 14:33:30 -04:00
niuyulin
e099aa5e20 HBASE-24659 Calculate FIXED_OVERHEAD automatically (#2018)
Co-authored-by: niuyulin <niuyulin@xiaomi.com>

SIgned-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-08-06 14:52:20 -07:00
Bharath Vissapragada
c372ff19cf
HBASE-21905 (addendum): Address compaction races in TestFIFOCompactionPolicy (#2203)
Looped through the test 100 times and it passes. Without the patch it fails
every ~10 runs or so.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2020-08-06 09:40:32 -07:00
Michael Stack
9a564dc2bf
HBASE-24817 Allow configuring WALEntry filters on ReplicationSource (#2198)
Allow specifying base WALEntry filter on construction of
ReplicationSource. Add means of being able to filter WALs by name.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 Add constructor that allows passing a predicate for filtering *in* WALs
 and a list of filters for filtering *out* WALEntries. The latter was
 hardcoded to filter out system-table WALEntries. The former did not
 exist but we'll need it if Replication takes in more than just the
 default Provider.

Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-06 09:29:08 -07:00
sguggilam
9f62e5645d
HBASE-24823 Port HBASE-22762 Print the delta between phases in the split/merge/compact/flush transaction journals
Closes #2204

Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-06 18:51:38 +05:30
gkanade
bfa558a0e3 HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751
writer null check if not initialized yet during syncrunner run (#2201)

Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: AnoopSamJohn<anoopsamjohn@apache.org>
(cherry picked from commit b0863c5832024033bc13efa3edb7c57b3b753996)
2020-08-05 23:41:13 -05:00
Sean Busbey
ec39d59161 Revert "writer null check if not initialized yet during syncrunner run (#2201)"
bad commit message

This reverts commit b0863c5832024033bc13efa3edb7c57b3b753996.
2020-08-05 23:39:18 -05:00
Zheng Wang
64fe71dba3
HBASE-24821 simplify the logic of getRegionInfo in TestFlushFromClient to reduce redundancy code
Closes #2200

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-05 20:01:12 +05:30
gkanade
b0863c5832
writer null check if not initialized yet during syncrunner run (#2201)
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: AnoopSamJohn<anoopsamjohn@apache.org>
2020-08-05 18:44:12 +05:30
Viraj Jasani
8979202c7a
HBASE-24795 : RegionMover to deal with unknown region while (un)loading
* RegionMover to ignore move failures for split/merged regions with ack mode
* Refactor MoveWithAck and MoveWithoutAck as high level classes
* UT for RegionMover gracefully handling split/merged regions while loading regions and throwing failure while loading offline regions

Closes #2172

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Ted Yu <tyu@apache.org>
2020-08-04 15:59:12 +05:30
Michael Stack
8c4f8e28e4
HBASE-24808 skip empty log cleaner delegate class names (WAS => cleaner.CleanerChore: Can NOT create CleanerDelegate= ClassNotFoundException) (#2181)
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-08-03 13:15:19 -07:00
Michael Stack
86d2e37bc6
HBASE-24767 Change default to false for HBASE-15519 per-user metrics Set hbase.regionserver.user.metrics.enabled default to false; i.e. off. (#2132)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-08-03 11:36:19 -07:00
Zheng Wang
8b1c86a54d
HBASE-24704 Make Table Schema easier to view with multiple families
Closes #2182

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-02 23:39:09 +05:30
Sean Busbey
0806349ada
HBASE-24805 HBaseTestingUtility.getConnection should be threadsafe
* refactor how we use connection to rely on the access method
* refactor initialization and cleanup of the shared connection
* incompatibly change HCTU's Configuration member variable to be final so it can be safely accessed from multiple threads.

Closes #2180

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 86ebbdd8a2df89de37c2c3bd50e64292eaf28b11)
2020-08-02 00:51:03 -05:00
Anoop Sam John
86fccba0d0
HBASE-24695 FSHLog - close the current WAL file in a background thread. (#2183)
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-08-01 22:46:32 +05:30
bsglz
1c45c8c7db
HBASE-24404 Support flush a single column family of region (#2098)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-07-31 09:31:14 +01:00
Sean Busbey
cb179467fe HBASE-24794 hbase.rowlock.wait.duration should not be <= 0 (#2174)
if hbase.rowlock.wait.duration is <=0 then log a message and treat it as a value of 1ms.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 840a55761b4b3db6bfcf8ca5b7ae67509fc21566)
2020-07-30 12:50:48 -05:00
Sandeep Pal
6c61c50df7
HBASE-24788: Fix the connection leaks on getting hbase admin from unclosed connection (#2173)
Signed-off-by: Ted Yu <tyu@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-07-30 10:32:32 -07:00
ddupg
13f9f7a2c1
HBASE-24797 Move log code out of loop
Closes #2175

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-30 15:18:48 +05:30
Bharath Vissapragada
06236dbfcc
HBASE-20226: Parallelize region manifest deletes (#2159)
We observed this delete call to be a bottleneck for table with lots of
regions. Patch attempts to parallelize them.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
(cherry picked from commit f07f30ae24e6d0e9151bb76aebf78928ec9572e3)
2020-07-29 11:08:19 -07:00
WenFeiYi
bba70f08ea
HBASE-24752 NPE/500 accessing webui on master startup
Closes #2148

Signed-off-by: Ted Yu <yuzhihong@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-29 20:24:22 +05:30
Michael Stack
20990b9c04
HBASE-24766 Document Remote Procedure Execution (#2131)
Add outline of how the mechanism works generally.

Signed-off-by: Anoop Sam John anoopsamjohn@apache.org
Signed-off-by: Viraj Jasani vjasani@apache.org
Signed-off-by: Jan Hentschel jan.hentschel@ultratendency.com
2020-07-29 07:44:01 -07:00
stack
edb4cd534c HBASE-24632 Enable procedure-based log splitting as default in hbase3 Add deprecation of 'classic' zk-based WAL splitter.
Also fix three bugs:

 * We were trying to delete non-empty directory; weren't doing
 accounting for meta WALs where meta had moved off the server
 (successfully)
 * We were deleting split WALs rather than archiving them.
 * We were not handling corrupt files.

Deprecations and removal of tests of old system.
2020-07-27 08:23:29 -07:00
Wellington Ramos Chevreuil
fce52fe6bb HBASE-24758 Avoid flooding replication source RSes logs when no sinks… (#2118)
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>

(cherry picked from commit 8c0d7fa5b8971de8bc7062675ba96a6091263776)
2020-07-27 12:43:57 +01:00
WenFeiYi
2bb76c0a51
HBASE-24665 (#2150)
Co-authored-by: wen_yi <liu.wenwen@immomo.com>
 Signed-off-by: Anoop <anoopsamjohn@apache.org>
 Signed-off-by: Ramkrishna <ramkrishna@apache.org>
 Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-27 16:39:49 +05:30
Viraj Jasani
6cb51cc0f0
HBASE-24757 : ReplicationSink should limit row count in batch mutation based on hbase.rpc.rows.warning.threshold (#2136)
Closes #2127

Signed-off-by: stack <stack@apache.org>
2020-07-24 20:32:40 +05:30
Guanghao Zhang
70ab0dc324
HBASE-24743 Reject to add a peer which replicate to itself earlier (#2124)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-07-24 08:09:02 +08:00
Viraj Jasani
ce4e692699
HBASE-24718 : Generic NamedQueue framework for multiple use-cases (Refactor SlowLog responses) (#2110)
Closes #2052

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-07-22 11:36:19 +05:30
surbhi
33102a1265 HBASE-22146 Removing a namespace-level space quota does not remove policies against contained tables
Closes #1935

Signed-off-by: Josh Elser <elserj@apache.org>
2020-07-21 21:24:06 -04:00
Viraj Jasani
5bb76bf22d
Revert "HBASE-24743 Reject to add a peer which replicate to itself earlier (#2071)"
This reverts commit 5db3ec2cfb0bc44c99d7bd3bc61a45ecd258c3b6.

TestReplicationAdmin and TestReplicationShell are broken on branch-2 and master respectively
2020-07-21 20:20:51 +05:30
Mingliang Liu
3c91c33452
HBASE-24696 Include JVM information on Web UI under "Software Attributes"
Closes #2087

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-07-21 16:55:44 +05:30
Sanjeet Nishad
70d1b35a4b
HBASE-24714 Error message is displayed in UI of table's compaction state if any region of that table is not open
Closes #2047

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-21 12:35:08 +05:30
sguggilam
19d0f364ed
HBASE-24740 Enable journal logging for HBase snapshot operation (#2104)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
(cherry picked from commit 430602aed82c09cfac6781dbd3dc5eba0dfef4af)
2020-07-20 18:56:37 -07:00
Guanghao Zhang
5db3ec2cfb HBASE-24743 Reject to add a peer which replicate to itself earlier (#2071)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-07-21 09:06:54 +08:00
stack
620470607e Revert "HBASE-24718 : Generic NamedQueue framework for multiple use-cases (Refactor SlowLog responses)"
Causes TestAdminShell and TestThriftHBaseServiceHandler to fail 100% of the time in branch-2.

This reverts commit 8ae3480e70e8900c0c53b60f9bac69967813f1fc.
2020-07-20 14:50:03 -07:00
Wellington Ramos Chevreuil
8c89582633 HBASE-24747 Log an ERROR if HBaseSaslRpcServer initialisation fails with an uncaught exception (#2079)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
(cherry picked from commit bd42c75cac47123368b031a0645a557106be8018)
2020-07-20 10:29:04 +01:00
Zheng Wang
0a8a7fa2ed
HBASE-24709 Support MoveCostFunction use a lower multiplier in offpeak hours
Closes #2099

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-07-19 20:11:20 +05:30