Commit Graph

17620 Commits

Author SHA1 Message Date
Nick Dimiduk acfa08cd87 HBASE-24627 Normalize one table at a time
Introduce an additional method to our Admin interface that allow an
operator to selectivly run the normalizer. The IPC protocol supports
general table name select via compound filter.

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-19 09:22:21 -07:00
Toshihiro Suzuki 00aa3bc9fc
HBASE-24884 BulkLoadHFilesTool/LoadIncrementalHFiles should accept -D options from command line parameters (#2275)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-08-19 19:55:34 +09:00
Zheng Wang 7335dbc834
HBASE-24854 Correct the help content of assign and unassign commands in hbase shell
Closes #2241

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-08-13 15:39:55 +05:30
Guanghao Zhang 4bf833418f HBASE-24876 Fix the flaky job url in hbase-personality.sh (#2253)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-08-13 16:52:29 +08:00
Nick Dimiduk 9c85f3641f HBASE-24583 Normalizer can't actually merge empty regions...
when neighbor is larger than average size

* add `testMergeEmptyRegions` to explicitly cover different
  interleaving of 0-sized regions.
* fix bug where merging a 0-size region is skipped due to large
  neighbor.
* remove unused `splitPoint` from `SplitNormalizationPlan`.
* generate `toString`, `hashCode`, and `equals` methods from Apache
  Commons Lang3 template on `SplitNormalizationPlan` and
  `MergeNormalizationPlan`.
* simplify test to use equality matching over `*NormalizationPlan`
  instances as plain pojos.
* test make use of this handy `TableNameTestRule`.
* fix line-length issues in `TestSimpleRegionNormalizer`

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
Signed-off-by: Aman Poonia <aman.poonia.29@gmail.com>
2020-08-12 10:11:40 -07:00
WenFeiYi 28a05a0f87
HBASE-24844 RecoverableZookeeper#close followed by ExecutorService shutdown
* Resolves Exception on standalone (master) shutdown

Closes #2239

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-12 17:10:56 +05:30
Viraj Jasani 8ccf643fdc
HBASE-24750 : All ExecutorService should use guava ThreadFactoryBuilder (#2214)
Closes #2196

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Ted Yu <tyu@apache.org>
Signed-off-by: niuyulin <nyl353@163.com>
2020-08-12 15:57:53 +05:30
Duo Zhang 96ea136a8e HBASE-24856 Fix error prone error in FlushTableSubprocedure (#2242)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-08-12 10:01:25 +08:00
Nick Dimiduk 8737afb01e HBASE-24843 Sort the constants in `hbase_constants.rb`
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-08-11 16:13:42 -07:00
Duo Zhang 7f5f5776b7 HBASE-24841 Change the jenkins job urls in our jenkinsfile (#2227)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-11 09:07:30 +08:00
Bharath Vissapragada 0533937b16
HBASE-24846: Address compaction races in TestFIFOCompactionPolicy (#2203)
Looped through the test 100 times and it passes. Without the patch it fails
every ~10 runs or so.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2020-08-10 17:21:27 -07:00
Bharath Vissapragada cd97d1a452
Revert "HBASE-21905 (addendum): Address compaction races in TestFIFOCompactionPolicy (#2203)"
This reverts commit c372ff19cf.

Reverting to apply as HBASE-24846.
2020-08-10 17:21:05 -07:00
Nick Dimiduk 17a0c2aabf HBASE-24835 Normalizer should log a successful run at INFO level
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-10 10:13:54 -07:00
huaxiangsun bd30555ac8
HBASE-24824 Add more stats in PE for read replica (#2205) (#2212)
Signed-off-by: stack <stack@apache.org>
2020-08-10 09:58:50 -07:00
Toshihiro Suzuki 22bf9a38c9
HBASE-24680 Refactor the checkAndMutate code on the server side (#2184)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2020-08-10 18:57:17 +09:00
bsglz ebf493f075
HBASE-24694 Support flush a single column family of table (#2218)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-08-10 10:36:13 +01:00
Duo Zhang aab8ce9e01 HBASE-24838 The pre commit job fails to archive surefire reports (#2224)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-08-10 11:13:56 +08:00
Duo Zhang ef7b9eb36e HBASE-23157 WAL unflushed seqId tracking may wrong when Durability.ASYNC_WAL is used (#762)
Signed-off-by: stack <stack@apache.org>
2020-08-10 10:54:30 +08:00
chenglei 457234c695 HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length(addendum) (#2055)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-08-10 10:04:05 +08:00
stack cee7431d0a HBASE-24830 Some tests involving RS crash fail with NullPointerException after HBASE-24632 in branch-2 2020-08-09 11:16:37 -07:00
Viraj Jasani 4a2f3f89f4
HBASE-24788 : TableOutputFormat source compatibility (ADDENDUM)
Brought back InterruptedException in throws clause of getRecordWriter and checkOutputSpecs of TableOutputFormat
2020-08-08 14:30:40 +05:30
Bo Cui 11cfb0f382 HBASE-21721 FSHLog : reduce write#syncs() times (#2217)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Anoop <anoopsamjohn@apache.org>
2020-08-08 09:41:36 +05:30
Josh Elser 303db63b76 HBASE-24779 Report on the WAL edit buffer usage/limit for replication
Closes #2193

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-08-07 14:33:30 -04:00
bsglz cb3dd990f7 HBASE-24826 Add some comments for processlist in hbase shell (#2207)
Signed-off-by: Elliot Miller <elliot_miller@apple.com>
Signed-off-by: stack <stack@apple.com>
2020-08-06 19:29:33 -07:00
niuyulin e099aa5e20 HBASE-24659 Calculate FIXED_OVERHEAD automatically (#2018)
Co-authored-by: niuyulin <niuyulin@xiaomi.com>

SIgned-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-08-06 14:52:20 -07:00
Bharath Vissapragada c372ff19cf
HBASE-21905 (addendum): Address compaction races in TestFIFOCompactionPolicy (#2203)
Looped through the test 100 times and it passes. Without the patch it fails
every ~10 runs or so.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2020-08-06 09:40:32 -07:00
Michael Stack 9a564dc2bf
HBASE-24817 Allow configuring WALEntry filters on ReplicationSource (#2198)
Allow specifying base WALEntry filter on construction of
ReplicationSource. Add means of being able to filter WALs by name.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 Add constructor that allows passing a predicate for filtering *in* WALs
 and a list of filters for filtering *out* WALEntries. The latter was
 hardcoded to filter out system-table WALEntries. The former did not
 exist but we'll need it if Replication takes in more than just the
 default Provider.

Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-06 09:29:08 -07:00
sguggilam 9f62e5645d
HBASE-24823 Port HBASE-22762 Print the delta between phases in the split/merge/compact/flush transaction journals
Closes #2204

Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-06 18:51:38 +05:30
gkanade bfa558a0e3 HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751
writer null check if not initialized yet during syncrunner run (#2201)

Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: AnoopSamJohn<anoopsamjohn@apache.org>
(cherry picked from commit b0863c5832)
2020-08-05 23:41:13 -05:00
Sean Busbey ec39d59161 Revert "writer null check if not initialized yet during syncrunner run (#2201)"
bad commit message

This reverts commit b0863c5832.
2020-08-05 23:39:18 -05:00
Zheng Wang 64fe71dba3
HBASE-24821 simplify the logic of getRegionInfo in TestFlushFromClient to reduce redundancy code
Closes #2200

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-05 20:01:12 +05:30
gkanade b0863c5832
writer null check if not initialized yet during syncrunner run (#2201)
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: AnoopSamJohn<anoopsamjohn@apache.org>
2020-08-05 18:44:12 +05:30
Viraj Jasani 8979202c7a
HBASE-24795 : RegionMover to deal with unknown region while (un)loading
* RegionMover to ignore move failures for split/merged regions with ack mode
* Refactor MoveWithAck and MoveWithoutAck as high level classes
* UT for RegionMover gracefully handling split/merged regions while loading regions and throwing failure while loading offline regions

Closes #2172

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Ted Yu <tyu@apache.org>
2020-08-04 15:59:12 +05:30
Michael Stack 8c4f8e28e4
HBASE-24808 skip empty log cleaner delegate class names (WAS => cleaner.CleanerChore: Can NOT create CleanerDelegate= ClassNotFoundException) (#2181)
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-08-03 13:15:19 -07:00
Michael Stack 86d2e37bc6
HBASE-24767 Change default to false for HBASE-15519 per-user metrics Set hbase.regionserver.user.metrics.enabled default to false; i.e. off. (#2132)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-08-03 11:36:19 -07:00
Zheng Wang 8b1c86a54d
HBASE-24704 Make Table Schema easier to view with multiple families
Closes #2182

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-02 23:39:09 +05:30
Sean Busbey 0806349ada
HBASE-24805 HBaseTestingUtility.getConnection should be threadsafe
* refactor how we use connection to rely on the access method
* refactor initialization and cleanup of the shared connection
* incompatibly change HCTU's Configuration member variable to be final so it can be safely accessed from multiple threads.

Closes #2180

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 86ebbdd8a2)
2020-08-02 00:51:03 -05:00
Anoop Sam John 86fccba0d0
HBASE-24695 FSHLog - close the current WAL file in a background thread. (#2183)
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-08-01 22:46:32 +05:30
Elliot 271a3cbf13 HBASE-24803 Unify hbase-shell ::Shell::Commands::Command#help behavior (#2178)
- Update all hbase-shell commands to return help text implicitly
- Add hbase-shell test to ensure all help methods return unindented text
- Set the outermost indentation to zero for the help text of all hbase-shell
  commands by using the squiggly heredoc added in Ruby 2.3.0.
- Document return value of hbck_chore_run and add usage examples to truncate commands
- Update docs to reflect current return value of list command

Signed-off-by: stack <stack@apache.org>
2020-07-31 07:42:54 -07:00
bsglz 1c45c8c7db
HBASE-24404 Support flush a single column family of region (#2098)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-07-31 09:31:14 +01:00
Sean Busbey cb179467fe HBASE-24794 hbase.rowlock.wait.duration should not be <= 0 (#2174)
if hbase.rowlock.wait.duration is <=0 then log a message and treat it as a value of 1ms.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 840a55761b)
2020-07-30 12:50:48 -05:00
Sandeep Pal 6c61c50df7
HBASE-24788: Fix the connection leaks on getting hbase admin from unclosed connection (#2173)
Signed-off-by: Ted Yu <tyu@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-07-30 10:32:32 -07:00
ddupg 13f9f7a2c1
HBASE-24797 Move log code out of loop
Closes #2175

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-30 15:18:48 +05:30
Bharath Vissapragada 06236dbfcc
HBASE-20226: Parallelize region manifest deletes (#2159)
We observed this delete call to be a bottleneck for table with lots of
regions. Patch attempts to parallelize them.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
(cherry picked from commit f07f30ae24)
2020-07-29 11:08:19 -07:00
WenFeiYi bba70f08ea
HBASE-24752 NPE/500 accessing webui on master startup
Closes #2148

Signed-off-by: Ted Yu <yuzhihong@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-29 20:24:22 +05:30
Michael Stack 20990b9c04
HBASE-24766 Document Remote Procedure Execution (#2131)
Add outline of how the mechanism works generally.

Signed-off-by: Anoop Sam John anoopsamjohn@apache.org
Signed-off-by: Viraj Jasani vjasani@apache.org
Signed-off-by: Jan Hentschel jan.hentschel@ultratendency.com
2020-07-29 07:44:01 -07:00
niuyulin 18d2c72207 HBASE-24669 Logging of ppid should be consistent across all occurrences
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-07-29 06:34:02 -07:00
Elliot 568d73d7db HBASE-11676 Scan FORMATTER is not applied for columns using non-printable name in shell (#2161)
- In HBase::Table, the instance variable @converters is used to map column
  names to converters. This patch fixes how HBase::Table#_get_internal and
  HBase::Table#_scan_internal generate the column name key used to access
  @converters.
- Refactor parsing of family:qualifier:converter specifications so that the
  code is more readable and reusable. As part of this change, I added two
  private methods and marked HBase::Table#set_converter as deprecated for
  removal in HBase 4.0.0.
- Add unit testing for the fixed bug

Signed-off-by: stack <stack@apache.org>
2020-07-28 20:44:35 -07:00
Pankaj 6ded070f82 HBASE-24738 [Shell] processlist command fails with ERROR: Unexpected end of file from server when SSL enabled (#2123)
Signed-off-by: Elliot Miller <elliot_miller@apple.com>
2020-07-28 09:11:57 -07:00
Toshihiro Suzuki 2720a9d93e HBASE-24775 [hbtop] StoreFile size should be rounded off (#2144)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-28 08:16:20 +09:00