Commit Graph

8428 Commits

Author SHA1 Message Date
Bharath Vissapragada 60c9ae58fb
HBASE-21905: [FIFOCompaction] Don't compact already inflight store files (#1829)
This one surfaced as a flake test but turns out to be a legit bug
in FIFOCompaction code. FifoCompaction does not check if an empty
store file is already being compacted by an in-flight compaction
request and still enqueues. It can potentially race with a running
compaction (as in this test case, see jira for the exact exception).

Fixes the edge case and cleans up the test code a bit.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-06-03 16:37:22 -07:00
Viraj Jasani 8de8c44029
HBASE-24492 : Remove infinite loop from ProtobufLogReader#readNext (#1831)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-06-03 13:41:15 +05:30
niuyulin f71f1cdfa0
HBASE-24412 Canary support check only one column family per RegionTask (#1817)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-06-03 09:15:10 +08:00
XinSun 901187ae2f
HBASE-22738 Fallback to default group to choose RS when there are no RS in current group (#1789)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-06-03 08:26:05 +08:00
Nick Dimiduk cb5c0d2782 HBASE-24418 Consolidate Normalizer implementations
Simplify our Normalizer story to have just a single, configurable
implementation.

* fold the features of `MergeNormalizer` into
  `SimpleRegionNormalizer`, removing the intermediate abstract class.
* configuration keys for merge-only features now share a common
  structure.
* add configuration to selectively disable normalizer split/merge
  operations.
* `RegionNormalizer` now extends `Configurable` instead of creating a
  new instance of `HBaseConfiguration` or snooping one off of other
  fields.
* avoid the extra RPCs by using `MasterServices` instead of
  `MasterRpcServices`.
* boost test coverage of all the various flags and feature
  combinations.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
2020-06-02 11:38:39 -07:00
Wellington Ramos Chevreuil e5345b3a7c
HBASE-21406 "status 'replication'" should not show SINK if the cluste… (#1761)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2020-06-02 09:27:16 +01:00
Duo Zhang bad2d4e409
HBASE-24474 Rename LocalRegion to MasterRegion (#1811)
Signed-off-by: Michael Stack <stack@apache.org>
2020-06-02 10:37:43 +08:00
Bharath Vissapragada f5b90fcc44
HBASE-24479: Deflake TestCompaction#testStopStartCompaction (#1820)
Polling of active compaction count is racy. Tightened the asserts
to be more reliable.

Signed-off-by: Reid Chan <reidchan@apache.org>
2020-05-31 23:54:17 -07:00
Bharath Vissapragada 4f49a96258
HBASE-24477: Move ConfigurationObserver and related classes to hbase-common (#1815)
This utility is useful for any module that wants to detect
dynamic config changes. Having it to hbase-common makes it
accessible to all the other modules.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-05-31 11:18:16 -07:00
bsglz b4a4debdd9
HBASE-24475 Clean up the master thread name getting in SplitLogManager and AssignmentManager (#1810) 2020-05-30 22:23:04 -07:00
jacob-leblanc c120260bce
HBASE-24454 - Read ioErrorStartTime to local temporary variable to avoid issue when it is set to -1 between greater than zero check and calculation of error duration (#1816)
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
2020-05-30 21:06:40 -07:00
Duo Zhang 36a6ef9cf9
HBASE-24471 The way we bootstrap meta table is confusing (#1806)
Signed-off-by: Michael Stack <stack@apache.org>
2020-05-30 16:03:52 +08:00
Viraj Jasani f0f9a20cd7
HBASE-23941 : FilterBy operator support in get_slowlog_responses API (#1793)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-05-30 12:59:59 +05:30
Michael Stack bda2094ae5 HBASE-22287 inifinite retries on failed server in RSProcedureDispatcher (#1800)
Adds backoff in place of retry every 100ms.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-05-29 10:04:02 -07:00
xincunSong da2e03bb3b
HBASE-24470 : Add store file info when log meessages in HStore. (#1804)
Signed-off-by: binlijin <binlijin@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-05-29 18:58:57 +05:30
Viraj Jasani bf869def43
HBASE-24428 : Fix for testSplitCompactWithPriority (ADDENDUM) 2020-05-29 18:15:40 +05:30
bsglz 7b7e25a50b
HBASE-24451 Remove the HasThread because the related bug had been fixed since jdk7 (#1797)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-05-29 17:33:18 +05:30
Duo Zhang f06248ef84
HBASE-24443 Refactor TestCustomSaslAuthenticationProvider (#1790)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-05-28 20:44:31 +08:00
Viraj Jasani 48e9835adb
HBASE-24428 : Update compaction priority for recently split daughter regions (#1784)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-05-27 20:56:41 +05:30
binlijin aacb69d098
HBASE-24371 Add more details when print CompactionConfiguration info (#1711)
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2020-05-27 19:41:43 +08:00
Duo Zhang 476cb16232
HBASE-24437 Addendum just start mini dfs cluster, and make the log roll more robust (#1787)
Signed-off-by: stack <stack@apache.org>
2020-05-27 11:25:25 +08:00
Michael Stack e39fd6d11b HBASE-24425 Run hbck_chore_run and catalogjanitor_run on draw of 'HBCK Report' page (#1771)
Signed-off-by: clarax
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-05-26 19:51:50 -07:00
stack a9205f8f4d HBASE-24437 Flaky test, TestLocalRegionOnTwoFileSystems#testFlushAndCompact
Force another WAL roll just in case and make the check more loose (its
about whether the files are present, not their count)
2020-05-26 11:57:19 -07:00
XinSun e06e1dba10
HBASE-24416 RegionNormalizer spliting region should not be limited by hbase.normalizer.min.region.count (#1770)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-05-26 17:03:14 +08:00
Zheng Wang 29c59c0ba8 HBASE-24423 No need to get lock in canSplit because hasReferences will get lock too 2020-05-25 22:52:38 -07:00
WenFeiYi 285e59af25
HBASE-24427 HStore.add log format error (#1773)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-05-25 18:28:12 +05:30
Viraj Jasani 301659dcc7
HBASE-24426 : Provide regionName in warning during failure to move region from CLOSING to CLOSED (#1772)
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-05-25 17:33:15 +05:30
wenbang 006e4d5e46
HBASE-24401 Cell size limit check on append should consider 0 or less value to disable the check (#1742)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-05-25 09:58:01 +08:00
huaxiangsun eaa2762a4e
HBASE-24369 Provide more information about merged child regions in Hbck Overlaps section, which cannot be fixed immediately (#1756)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Clara Xiong
Signed-off-by: stack <stack@apache.org>
2020-05-24 17:21:04 -07:00
Duo Zhang c303f9d329
HBASE-24408 Introduce a general 'local region' to store data on master (#1753)
Signed-off-by: stack <stack@apache.org>
2020-05-23 15:59:51 +08:00
huaxiangsun 2b894dc100
HBASE-24379 CatalogJanitor misreports region holes when there are actually over laps. (#1741)
Signed-off-by: stack <stack@apache.org>
2020-05-22 16:55:54 -07:00
bsglz 0ce85fe63c
HBASE-24407 Correct the comment of clusterRegionLocationMocks in TestStochasticLoadBalancer (#1750) 2020-05-22 14:14:59 -07:00
huaxiangsun 80b64ef4dc
HBASE-24376 MergeNormalizer is merging non-adjacent regions and causing region overlaps/holes. (#1734)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-05-21 10:08:43 -07:00
huaxiangsun a8724e8120
HBASE-24370 Avoid aggressive MergeRegion and GCMultipleMergedRegionsProcedure (#1719)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-05-20 14:18:58 -07:00
Viraj Jasani 86a2692dc4
HBASE-23938 : System table hbase:slowlog to store complete slow/large… (#1681)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
2020-05-20 15:10:29 +05:30
niuyulin 8f4c255b38
HBASE-24386 TableSnapshotScanner support scan limit (#1724)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off by: Viraj Jasani <vjasani@apache.org>
2020-05-19 20:52:28 +05:30
ramkrish86 af8398a0ac
HBASE-24365 MetricsTableWrapperAggregateImpl runnable fails due to exception and never runs (#1729)
Signed-off-by: binlijin <binlijin@gmail.com>
Signed-off by: Viraj Jasani <vjasani@apache.org>
2020-05-18 15:11:02 +05:30
binlijin 6ce1e140ee
HBASE-24377 MemStoreFlusher throw NullPointerException (#1721)
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: Anoop Sam John <anoop.hbase@gmail.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-05-18 09:58:28 +08:00
Baiqiang Zhao 61efb6d5ad
HBASE-24381 The Size metrics in Master Webui is wrong if the size is 0 (#1723) 2020-05-18 09:14:36 +08:00
Viraj Jasani 15627bb722
HBASE-24327 : Flaky connection in TestMasterShutdown (ADDENDUM) 2020-05-16 12:29:16 +05:30
Guanghao Zhang bab4b1cf8c HBASE-23702 hbase.hstore.flusher.count setting to 0 breaks HMaster (#1703)
Co-authored-by: Ctest
Signed-off-by: Anoop Sam John <anoop.hbase@gmail.com>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-05-15 09:21:53 +08:00
Andrew Purtell 94f36fdb8e
Amend HBASE-24350: Extending and Fixing HBaseTable level replication metrics (#1704)
- Rename WALEntryBatch#getWaEntriesWithSize -> getWalEntriesWithSize
2020-05-14 13:42:48 -07:00
stack 32e2682310 HBASE-24368 Let HBCKSCP clear 'Unknown Servers', even if RegionStateNode has RegionLocation == null
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/GCMultipleMergedRegionsProcedure.java
 Edit a log.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/HBCKServerCrashProcedure.java
 Add override of isMatchingRegionLocation. Allow 'null' as a pass in
 HBCKSCP.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java
 Add a method for HBCKSCP to override and be less strict filtering
 assigns.

hbase-server/src/main/resources/hbase-webapps/master/hbck.jsp
 Some doc on what 'Unknown Servers' are.
2020-05-14 13:09:03 -07:00
Sandeep Pal 2e5a664233
HBASE-24350: Extending and Fixing HBaseTable level replication metrics (#1704)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-05-14 10:34:51 -07:00
bsglz 5e32e08782
HBASE-24164 Retain the ReadRequests and WriteRequests of region on we… (#1500)
Signed-off-by: stack <stack@apache.org>
2020-05-13 22:28:45 -07:00
Mingliang Liu a40a0322a7
HBASE-23969 Meta browser should show all `info` columns (#1485)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-05-13 11:50:53 -07:00
Viraj Jasani a782531633
HBASE-24327 : Flaky connection in TestMasterShutdown#testMasterShutdo… (#1690)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-05-13 02:38:47 +05:30
Gkkkk302 9e73244082
HBASE-24355 Fix typos in the HStore#compact annotation (#1699)
Co-authored-by: guo.kangkang <guo.kangkang@immomo.com>

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-05-12 21:07:49 +05:30
Duo Zhang 8601416ee8
HBASE-24309 Avoid introducing log4j and slf4j-log4j dependencies for modules other than hbase-assembly (#1640)
Signed-off-by: stack <stack@apache.org>
2020-05-12 12:03:30 +08:00
stack bb3191c563 HBASE-24338 [Flakey Tests] NPE in TestRaceBetweenSCPAndDTP Addendum2: make same fix for TestRaceBetweenSCPAndTRSP but in a second place (addendum 1 got first place). 2020-05-11 20:48:21 -07:00
stack 2c11d5a0a0 HBASE-24338 [Flakey Tests] NPE in TestRaceBetweenSCPAndDTP
Addendum: make same fix for TestRaceBetweenSCPAndTRSP which does same
thing.
2020-05-11 14:06:56 -07:00
Joseph295 68229c9c99
HBASE-24341 The region should be removed from ConfigurationManager as a ConfigurationObserver when it is closed (#1675)
Co-authored-by: Joseph295 <johnhomsea@gmail.com>
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
2020-05-11 17:31:54 +05:30
Viraj Jasani 6729cafa1d
Revert "HBASE-24327 : Handle shutdown() if master cannot be contacted (#1684)" (#1691)
This reverts commit 34a74534fe.
2020-05-11 16:43:49 +05:30
Geoffrey Jacoby ca81283fe5
HBASE-24321 - Add writable MinVersions and read-only Scan to coproc S… (#1655)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Lars Hofhansl <larsh@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-05-11 12:54:22 +05:30
Viraj Jasani 34a74534fe
HBASE-24327 : Handle shutdown() if master cannot be contacted (#1684)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-05-10 17:09:51 +05:30
huaxiangsun 30f07e8187
HBASE-24256 When fixOverlap hits the max region limit, it is possible to include the same region in multiple merge request (#1584)
Signed-off-by: stack <stack@apache.org>
2020-05-08 10:16:42 -07:00
Reid Chan 0cd70ed89c
HBASE-24345 [ACL] renameRSGroup should require Admin level permission (#1683)
Signed-off-by: binlijin <binlijin@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-05-08 18:56:05 +08:00
Jan Hentschel 03e5a1402d
HBASE-24305 Removed deprecations in ServerName (#1666)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-05-08 09:41:55 +02:00
stack fc283f7a68 HBASE-24338 [Flakey Tests] NPE in TestRaceBetweenSCPAndDTP #1669
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-05-07 13:54:45 -07:00
stack d30268d0ee HBASE-24342 [Flakey Tests] Disable TestClusterPortAssignment.testClusterPortAssignment as it can't pass 100% of the time 2020-05-07 13:22:35 -07:00
niuyulin f4a446c3d2
HBASE-24328 skip duplicate GCMultipleMergedRegionsProcedure while previous finished (#1629)
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
2020-05-06 10:06:49 -07:00
Michael Stack 6f059ecff0 HBASE-24331 [Flakey Test] TestJMXListener rmi port clash (#1657)
Add check that we can make jmx connector in setup. If we can't retry.
2020-05-06 09:29:57 -07:00
Reid Chan c32d18d4fb
HBASE-24336 [Metrics] FSDataInputStream's localBytesRead is wrong (#1667)
Signed-off-by: binlijin <binlijin@gmail.com>
2020-05-06 19:24:05 +08:00
Mohammad Arshad 294c228c67
HBASE-24211: Create table is slow in large cluster when AccessController is enabled. (#1631)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-05-06 15:12:20 +05:30
Duo Zhang c1cb22f0b3
HBASE-24265 Remove hedged rpc call support, implement the logic in MaterRegistry … (#1593)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-05-06 15:55:26 +08:00
Duo Zhang a9a1b9524d
HBASE-24304 Separate a hbase-asyncfs module (#1628)
Signed-off-by: stack <stack@apache.org>
2020-05-06 14:40:21 +08:00
huaxiangsun 045c909bdf
HBASE-24316 GCMulitpleMergedRegionsProcedure is not idempotent (#1660)
It addresses couple issues:
   1. Make sure deleteMergeQualifiers() does not delete the row if there is no columns with "merge" keyword.
   2. GCMulitpleMergedRegionsProcedure now acquire an exclusive lock on the child region.

Signed-off-by: stack <stack@apache.org>
2020-05-05 23:04:48 -07:00
Nick Dimiduk 07077a3950
HBASE-23976 Use nio for file manipulation in TestVerifyBucketCacheFile (#1342)
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
2020-05-05 22:04:26 -07:00
Michael Stack 33eadb5bb1 HBASE-24307 [Flakey Tests] krb server for secure thrift tests throws BindException (#1656)
Move the random free port generate back into hbasecommontestingutility
  from hbasetestingutility.

  Add a create simple kdc server utility that will start a kdc server and
  if a bindexception, create a new one on a new random port in hbase-common.

  Add new BoundSocketMaker helpful when trying to manufacture
  BindExceptions because of port clash.

  Change thrift and http kdc tests to use this new utility (removes
  code duplication around kdc server setup).
2020-05-05 21:36:59 -07:00
stack 24c995da97 HBASE-24118 [Flakey Tests] TestCloseRegionWhileRSCrash
Addendum; add back bit missing off end of test.
2020-05-05 21:16:33 -07:00
Yu Wang 3d96007c37
HBASE-24137:The max merge count of metafixer may be remind in hbase-site.xml (#1478) 2020-05-05 16:14:14 -07:00
Michael Stack 5488124be0 HBASE-24284 [h3/jdk11] REST server won't start Exclude transitive includes of jax-rs 1.x and then explicitly include jax-rs 2.x glassfish impl for REST context when hadoop3. (#1625) 2020-05-05 15:36:01 -07:00
huaxiangsun a240ba8214
HBASE-24273 HBCK's "Orphan Regions on FileSystem" reports regions with referenced HFiles (#1613)
Signed-off-by: stack <stack@apache.org>
2020-05-05 11:10:02 -07:00
Duo Zhang 3340c0024e
HBASE-24314 Some classes still use log4j logger directly (#1637)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-05-05 20:03:55 +08:00
Minwoo Kang 1b11ea2e1f
HBASE-23968 Periodically check whether a system stop is requested in compaction by time. (#1274)
Signed-off-by: stack <stack@apache.org>
2020-05-04 14:45:20 -07:00
Nick Dimiduk e37aafcfc2 HBASE-24260 Add a ClusterManager that issues commands via coprocessor
Implements `ClusterManager` that relies on the new
`ShellExecEndpointCoprocessor` for remote shell command execution.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-05-04 10:53:02 -07:00
Anoop Sam John 7313b4f5aa
HBASE-24311 Add more details in MultiVersionConcurrencyControl STUCK log message (#1636) 2020-05-04 21:26:29 +05:30
stack 255470a139 HBASE-24246 Miscellaneous hbck2 fixMeta bulk merge fixes: better logging around merges/overlap-fixing, 'HBCK Report' overlap listing, and configuration (#1572)
Addendum to address minor feedback on text
2020-05-04 08:27:21 -07:00
Michael Stack ef4cc669b9 HBASE-24246 Miscellaneous hbck2 fixMeta bulk merge fixes: better logging around merges/overlap-fixing, 'HBCK Report' overlap listing, and configuration (#1572)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
 Fix weird brackets around each region name when logging.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetaFixer.java
  Log when we hit the max merge limit. Also up limit to 64.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
 Make logs make more sense to operator.

hbase-server/src/main/resources/hbase-webapps/master/hbck.jsp
 Make RegionName show when you mouseover so long names don't mess up
 display of holes and overlaps.

Address Mingliang Liu liuml07 feedback

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-05-04 08:27:10 -07:00
niuyulin c44bb2e99a HBASE-24221 Support bulkLoadHFile by family (ADDENDUM)
Co-authored-by: niuyulin <niuyulin@xiaomi.com>

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-05-04 15:21:38 +01:00
Wellington Chevreuil b6e81c0e3f Revert "HBASE-24221 Support bulkLoadHFile by family, fix UT (#1633)"
This reverts commit 5d41588df6.
2020-05-04 15:21:01 +01:00
niuyulin 5d41588df6
HBASE-24221 Support bulkLoadHFile by family, fix UT (#1633)
Co-authored-by: niuyulin <niuyulin@xiaomi.com>
2020-05-04 15:18:55 +01:00
xincunSong d67f065594
[HBASE-24288]Allow admin user to create table and do bulkLoad (#1612)
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
Signed-off-by: binlijin <binlijin@gmail.com>
2020-05-01 11:57:33 -07:00
gkanade 9785f5d1c9
HBASE-24199 Procedure related metrics is not consumed in the JMX metric (#1616)
Signed-off-by: stack <stack@apache.org>
2020-05-01 11:45:16 -07:00
Duo Zhang e8504ba817
HBASE-24277 TestZooKeeper is flaky (#1617) 2020-05-01 20:34:42 +08:00
niuyulin 512d00e75d
HBASE-24221 Support bulkLoadHFile by family (#1569)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-04-30 18:33:37 +08:00
bsglz b810c9bb91
HBASE-22710 Wrong result in one case of scan that use raw and version… (#767)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-30 13:51:37 +05:30
xincunSong 856e21d199
HBASE-24282 'scanDetail' log message is missing when responseTooSlow happens on the first scan rpc call (#1604)
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-04-29 16:03:24 -07:00
stack e3c5527ce5 HBASE-24247 Failed multi-merge because two regions not adjacent (legitimately).
Addendum: Restore old semantic where merge of non-adjacent required
'force'
2020-04-29 14:35:48 -07:00
stack 9cff7a4ff4 HBASE-24215 [Flakey Tests] [ERROR] TestSecureRESTServer java.lang.NoClassDefFoundError: com/sun/jersey/core/spi/factory/AbstractRuntimeDelegate
Addendum: Add jersey-servlet to hadoop3 profile.

Made the hadoop3 profile in top-level pom same as it is for hadoop2
when it comes to exclusions. Then backed out previous attempt mostly.
Made the failing test medium-sized so it ran in its own jvm.
2020-04-29 07:39:59 -07:00
Joseph295 f46dba865f
HBASE-24278 The Java doc for minFilesToCompact is wrong (#1599)
Co-authored-by: Joseph295 <johnhomsea@gmail.com>
SIgned-off-by: stack <stack@apache.org>
2020-04-28 22:09:56 -07:00
Duo Zhang 9f52e6b725
HBASE-24249 Move code in FSHDFSUtils to FSUtils and mark related clas… (#1586)
Signed-off-by: stack <stack@apache.org>
2020-04-29 10:44:34 +08:00
Duo Zhang 6928674eb8
HBASE-24228 Merge the code in hbase-hadoop2-compat module to hbase-hadoop-compat (#1563)
Signed-off-by: stack <stack@apache.org>
2020-04-29 10:34:53 +08:00
Michael Stack 8fceec8cb1 HBASE-24247 Failed multi-merge because two regions not adjacent (legitimately) (#1570)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionInfo.java
 Add new isOverlap method that takes list of RegionInfos checking that
 current RegionInfo is overlapped by the passed in Regions.

    Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
    Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-04-28 13:24:04 -07:00
Duo Zhang 39a1bc53f8 Revert "HBASE-24212: Create table is slow in large cluster when AccessController is enabled. (#1546)"
Wrong JIRA issue number and also breaks TestAccessController.testAccessControllerUserPermsRegexHandling

This reverts commit e6cc5eb2f0.
2020-04-27 22:45:52 +08:00
Duo Zhang 84f2e95dd5
HBASE-24264 Disable TestNettyIPC.testHedgedAsyncEcho (#1589) 2020-04-26 23:36:07 +08:00
bsglz ec0d9d767e
HBASE-23264 Resolve a TODO of BucketAllocator: "Why we add the extra … (#801)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-26 15:15:31 +05:30
Josh Elser 8774614a8c HBASE-23881 Ensure Netty client receives at least one response before considering SASL negotiation complete
The PLAIN mechanism test added in the Shade authentication example has
different semantics than GSSAPI mechanism -- the client reports that the
handshake is done after the original challenge is computed. The javadoc
on SaslClient, however, tells us that we need to wait for a response
from the server before proceeding.

The client, best as I can see, does not receive any data from HBase;
however the application semantics (e.g. throw an exception on auth'n
error) do not work as we intend as a result of this bug.

Extra trace logging was also added to debug this, should a similar error
ever happen again with some other mechanism.

Closes #1260

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-04-26 10:24:07 +08:00
Duo Zhang c1f0634462 Revert "HBASE-23381 Ensure Netty client receives at least one response before considering SASL negotiation complete"
Incorrect jira issue number

This reverts commit 78eecd3a7d.
2020-04-26 10:23:40 +08:00
Mohammad Arshad e6cc5eb2f0
HBASE-24211: Create table is slow in large cluster when AccessController is enabled. (#1546)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Pankaj <pankajkumar@apache.org>
2020-04-25 17:41:43 +05:30
Duo Zhang 24961c5c34
HBASE-23848 Addendum fix TestSeekOptimizations (#1574) 2020-04-24 19:50:30 +08:00
niuyulin 6cea5c6462
HBASE-24222 remove FSUtils.checkAccess and replace with FileSystem.access in HBCK (#1557)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-23 23:20:38 +08:00
Viraj Jasani 4ed15e51d9
HBASE-24024 : Reject multi() requests with rows higher than threshold (#1560)
Signed-off-by: Reid Chan <reidchan@apache.org>
2020-04-22 12:46:40 +05:30
Jan Hentschel 75c717d4c2
HBASE-23848 Removed deprecated setStopRow from Scan (#1184)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-22 15:15:17 +08:00
niuyulin bc9184ee00
HBASE-23933 Separate a hbase-balancer module (#1436)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-22 14:55:38 +08:00
Beata Sudi 5f45c8293d
HBASE-24139 : Balancer should avoid leaving idle region servers (#1511)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2020-04-22 09:51:53 +05:30
stack e7b5c34d28 HBASE-24226 Address other hard references to '/tmp' found in Configuration 2020-04-21 17:40:35 -07:00
Viraj Jasani 5c53b606a9
HBASE-24175 : createDirAndSetProperty() (ADDENDUM) 2020-04-21 23:09:07 +05:30
stack 7c2f13eb70 HBASE-24175 [Flakey Tests] TestSecureExportSnapshot FileNotFoundException
Addendum #3; catch more '/tmp' references by hadoop.
2020-04-21 09:41:24 -07:00
stack 2d2e1d965d HBASE-24220 Allow that zk NOTEMPTY multi exception is retryable by running in-series
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReaderImpl.java
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
 Cleanup checkstyle warnings. Don't depend on hbase-client
 ScannerCallable.

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
 Cut down on cluster resource usage.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/AbstractTestDLS.java
 Debug

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController3.java
 Debug

hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftHttpServer.java
 Debug

hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Debug

hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
 Allow that NONEMPTY is retryable by running in series.
2020-04-20 15:51:11 -07:00
Guangxu Cheng bcacc4ce93
HBASE-23896 Snapshot owner cannot delete snapshot when ACL is enabled and Kerberos is not enabled (#1211)
Signed-off-by: binlijin <binlijin@gmail.com>
2020-04-20 09:59:06 +08:00
XinSun 8e5c6da3f0
HBASE-24166 Duplicate implementation for acquireLock between CreateTableProcedure and its parent class (#1486)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-18 10:30:12 +08:00
Joseph295 5ff4376d80
HBASE-24182 log when the region is set to closing status (#1509)
Signed-off-by: stack <stack@apache.org>
2020-04-17 12:20:12 -07:00
Reid Chan db36cf71c5
HBASE-24174 Fix findbugs warning on ServiceAuthorizationManager for master branch (#1537)
Signed-off-by: binlijin <binlijin@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-18 01:24:31 +08:00
gkanade 12aad76109
MetricsTable#updateFlushTime is wrong (#1539)
Authored-by: Gaurav Kanade <gkanade@microsoft.com>
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
2020-04-17 15:40:27 +05:30
Viraj Jasani d212dc4df0
HBASE-24195 : Admin.getRegionServers() should return live servers exc… (#1523)
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-16 20:13:14 +05:30
BukrosSzabolcs a0ef4cb8a2
HBASE-24186: RegionMover ignores replicationId (#1512)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-16 19:53:51 +05:30
Duo Zhang 1f66806c96
HBASE-24170 Remove hadoop-2.0 profile (#1495)
Signed-off-by: stack <stack@apache.org>
2020-04-16 18:57:40 +08:00
stack 853a72bc46 HBASE-24175 [Flakey Tests] TestSecureExportSnapshot FileNotFoundException
Addendum: add test to check for '/tmp' references.
2020-04-15 10:55:40 -07:00
huaxiangsun 3421423dd6
HBASE-24183 [flakey test] replication.TestAddToSerialReplicationPeer (#1514)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-04-15 10:07:57 -07:00
GeorryHuang 38954edb25 HBASE-23994: Add WebUI to Canary (#1292)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-04-14 15:56:19 -07:00
Nick Dimiduk cb60d23cb3 Revert "HBASE-23994:Add WebUI to Canary (#1292)"
This reverts commit daf79de329.
2020-04-14 15:55:56 -07:00
GeorryHuang daf79de329
HBASE-23994:Add WebUI to Canary (#1292)
HBASE-23994: Add WebUI to Canary

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-04-14 15:52:22 -07:00
binlijin 92b30f2638
HBASE-24181 Add region info when log meessages in HRegion. (#1506)
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-14 19:36:16 +08:00
Michael Stack 5f08311b23
HBASE-24134 Down forked JVM heap size from 2800m to 2200m for jdk8 and jdk11 (#1451) (#1503)
Down jdk8 forked jvm heap from 2800 to 2200 and the jdk11 heap from
3200 to 2200. Down the mvn size from 4G to 3.6G

Change how many puts done by TestMultiRespectsLimits because made
the test run the forked heap over 2.5G in size.

Signed-off-by: Sean Busbey <busbey@apache.org>
2020-04-13 21:35:43 -07:00
Jianfei Jiang ba34a2ca30
HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7 (#1453)
* HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7

* HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7

Co-authored-by: 姜建飞 10222269 <jiang.jianfei@zte.com.cn>

Signed-off-by: Mate Szalay-Beko <szalay.beko.mate@gmail.com>
Signed-off-by: Norbert Kalmar <nkalmar@cloudera.com>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangdo@apache.org>
2020-04-13 13:46:37 -07:00
stack 67b2e881ca HBASE-24158 [Flakey Tests] TestAsyncTableGetMultiThreaded 2020-04-13 11:51:47 -07:00
stack 64c7fc6a52 HBASE-24167 [Flakey Tests] TestHRegionWithInMemoryFlush#testWritesWhileScanning gets stuck MVCC 2020-04-13 11:44:07 -07:00
stack b00b5270f6 HBASE-24175 [Flakey Tests] TestSecureExportSnapshot FileNotFoundException
Overwrite yarn configs that write /tmp dir by default.
2020-04-13 11:29:25 -07:00
Reid Chan 22675d8070
HBASE-24112 [RSGroup] Support renaming rsgroup (#1435)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-13 14:04:32 +08:00
stack 72764cf628 HBASE-24153 Revert "Revert "Remove unnecessary super() in MultiVersionConcurrencyControl#MultiVersionConcurrencyControl() (#1468)""
A revert of a revert; i.e. a reapplication to add missing JIRA number

This reverts commit c4026d1beb.
2020-04-12 18:30:38 -07:00
stack c4026d1beb Revert "Remove unnecessary super() in MultiVersionConcurrencyControl#MultiVersionConcurrencyControl() (#1468)"
Forgot to add JIRA number. Reverting to add.

This reverts commit 12a0c1d5fe.
2020-04-12 18:30:11 -07:00
Viraj Jasani b6b8c6973d
HBASE-24140 : Move CandidateGenerator and their implementors out of StochasticLoadBalancer (#1458)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-12 19:13:47 +05:30
leosunli 12a0c1d5fe
Remove unnecessary super() in MultiVersionConcurrencyControl#MultiVersionConcurrencyControl() (#1468)
Signed-off-by: sunlisheng <lisheng.sun08@gmail.com>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-10 15:43:10 -07:00
Sean Busbey 09d20fb4d7
HBASE-24163 MOB compactor implementations should use format specifiers when calling String.format (#1479)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2020-04-10 11:00:53 -05:00
huaxiangsun ac4e94e868
HBASE-24161 [flakey test] locking.TestEntityLocks.testEntityLockTimeout (#1477)
Signed-off-by: stack <stack@apache.org>
2020-04-09 21:13:06 -07:00
Pankaj c92dd28a88
HBASE-24074 ConcurrentModificationException occured in ReplicationSourceManager while refreshing the peer (#1471)
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
Signed-off-by: stack <stack@apache.org>
2020-04-09 16:40:15 -07:00
huaxiangsun 8aa3304427
HBASE-24159 [flakey test] regionserver.TestRegionMergeTransactionOnCluster (#1472)
Delete tables after each test cases to avoid polluting states.

Signed-off-by: stack <stack@apache.org>
2020-04-09 16:27:19 -07:00
Reid Chan 1fb693b998
HBASE-24121 [Authorization] ServiceAuthorizationManager isn't dynamically updatable. And it should be (#1439)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2020-04-09 16:28:22 +08:00
Sean Busbey eb7df0498c
HBASE-23723 Ensure MOB compaction works in optimized mode after snapshot clone (#1446)
* Reorganize MOB compaction tests for more reuse.
* Add tests for mob compaction after snapshot clone operations
* note the original table used to write a given mob hfile and use that to find it later.

Signed-off-by: Esteban Gutierrez <esteban@apache.org>
2020-04-08 14:16:43 -05:00
Sean Busbey 2d78a286b6
HBASE-24138 log more details about balancer decisions for StochasticLoadBalancer (#1455)
- at DEBUG log messages about RegionCountSkewCostFunction region/server totals
- at DEBUG log messages about the decision to balance or not with total costs
- at TRACE log messages about region count on each server RegionCountSkewCostFunction sees
- at TRACE log message with the individual cost functions used in the decision to balance or not

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-08 14:03:33 -05:00
Andrew Purtell ac6060c3f9
HBASE-24099 Use a fair ReentrantReadWriteLock for the region close lock (#1407)
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-04-08 11:36:48 -07:00
Wei-Chiu Chuang 72727ff9be
HBASE-23833. The relocated hadoop-thirdparty protobuf breaks HBase asyncwal (#1301)
* Use Reflection to access shaded Hadoop protobuf classes.

(cherry picked from commit a321e536989083ca3620bf2c53f12c07740bf5b0)

* Update to improve the code:

1. Added license.
2. Added more comments.
3. Wrap byte array instead of copy to make a ByteString.
4. Moved all reflection instantiation to static class loading time.

* Use LiteralByteString to wrap byte array instead of copying it.


Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-04-08 11:31:51 -07:00
huaxiangsun ed830222da
HBASE-24120 Flakey Test: TestReplicationAdminWithClusters timeout (#1441)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-08 09:28:51 -07:00
huaxiangsun d6451a8e2b
HBASE-24114 [Flakey Tests] TestSnapshotScannerHDFSAclController (#1433)
Signed-off-by: stack <stack@apache.org>
2020-04-07 20:36:11 -07:00
Michael Stack 80d4a09ddc HBASE-24128 [Flakey Tests] Add retry on thrift cmdline if client fails plus misc debug (#1442)
hbase-server/src/test/java/org/apache/hadoop/hbase/TestClusterPortAssignment.java
 Saw case where Master failed startup but it came out as an IOE so we
 did not trip the retry logic.

hbase-server/src/test/java/org/apache/hadoop/hbase/TestInfoServers.java
 Add some debug and up timeouts. This test fails frequently for me
 locally.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/locking/TestEntityLocks.java
 Up the wait from 2x 200ms to 10x in case a pause on hardware or GC.
 This test fails locally and up on jenkins.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestClearRegionBlockCache.java
 Debug. Have assert say what bad count was.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactingToCellFlatMapMemStore.java
 Fails on occasion. Found count is off by a few. Tricky to debug. HBASE-24129 to reenable.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java
 Debug. Add wait and check before moving to assert.

hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftHttpServer.java
 Check for null before shutting; can be null if failed start.

hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerCmdLine.java
 Add retry if client messes up connection. Fails for me locally.
2020-04-07 09:29:23 -07:00
Duo Zhang f60f0bdbf0
HBASE-24055 Make AsyncFSWAL can run on EC cluster (#1437)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-04-07 23:41:35 +08:00
binlijin 62718cdb28
HBASE-24077 When encounter RowTooBigException, log the row info. (#1379)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Pankaj <pankajkumar@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-04-07 10:28:35 +08:00
stack 0e30e00c56 HBASE-24118 [Flakey Tests] TestCloseRegionWhileRSCrash
Reapply but as an @Ignore for the flakey test.
2020-04-06 12:38:02 -07:00
stack 0dc2712c90 Revert "HBASE-24118 [Flakey Tests] TestCloseRegionWhileRSCrash"
Reverting in favor of adding an @Ignore on this test until
root cause of flakyness HBASE-24117 is addressed.

This reverts commit 9985c06647.
2020-04-06 12:35:52 -07:00
huaxiangsun da67491a56
HBASE-24105 [Flakey Test] regionserver.TestRegionReplicas (#1425)
Co-authored-by: Huaxiang Sun <huaxiangsun@apache.com>
Signed-off-by: stack <stack@apache.org>
2020-04-05 13:20:23 -07:00
stack 9985c06647 HBASE-24118 [Flakey Tests] TestCloseRegionWhileRSCrash 2020-04-04 20:13:18 -07:00
Viraj Jasani 2a95fffb3b
HBASE-24102 : Remove decommissioned RS from target servers while unlo… (#1417)
Signed-off-by: binlijin <binlijin@gmail.com>
Signed-off-by: Pankaj <pankajkumar@apache.org>
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
2020-04-03 18:28:49 +05:30
huaxiangsun 37aa6690b5
HBASE-24080 [flakey test] TestRegionReplicaFailover.testSecondaryRegionKill fails. (#1421)
Signed-off-by: stack <stack@apache.org>
2020-04-02 14:55:55 -07:00
meiyi b2c9a06dfd
HBASE-24103 [Flakey Tests] TestSnapshotScannerHDFSAclController (#1416)
Signed-off-by: stack <stack@apache.org>
2020-04-02 09:55:47 -07:00
huaxiangsun f00753bacd
HBASE-24073 [flakey test] client.TestAsyncRegionAdminApi messed up compaction state. (#1414)
Addendum:
  For major compaction test, set hbase.hstore.compaction.min to a big number to
  avoid kicking in minor compactions, which will pollute compaction state and
  sometimes, cause major compaction cannot happen.

Co-authored-by: Huaxiang Sun <huaxiangsun@apache.com>
Signed-off-by: stack <stack@apache.org>
2020-04-02 08:16:27 -07:00
niuyulin 0b3918fe56
HBASE-24021 Fail fast when bulkLoadHFiles method catch some IOException (#1343)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-04-02 23:15:14 +08:00
Jan Hentschel 4ddf55d20a
HBASE-23843 Removed deprecated Scan(byte[]) from Scan
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-04-02 11:44:34 +02:00
stack 554678ea4c Revert "HBASE-24051 Allows indirect inheritance to CanUnbuffer (#1406)"
This reverts commit ae1f928b13.
2020-04-01 15:50:25 -07:00
申胜利 ae1f928b13
HBASE-24051 Allows indirect inheritance to CanUnbuffer (#1406)
Signed-off-by: stack <stack@apache.org>
2020-04-01 14:39:00 -07:00
Michael Stack 0595200fb5 HBASE-24097 [Flakey Tests] TestSnapshotScannerHDFSAclController#testRestoreSnapshot (#1405)
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationTrackerZKImpl.java
 Add debug for when assert fails (it fails on occasion locally)

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestHDFSAclHelper.java
 Move this inner class out standalone since it used now by two tests.

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController.java
 Moved out testRestoreSnapshot and made methods in here static so could
 be used by a new adacent test. Also made tablenames unique to methods
 thinking that was root of original issue (wasn't but no harm in doing
 this change) Moved out the inner class TestHDFSAclHelper.

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController2.java
 New class that sets up same context as
 TestSnapshotScannerHDFSAclController but just to run single
 testRestoreSnapshot method.

hbase-server/src/test/java/org/apache/hadoop/hbase/security/token/TestZKSecretWatcher.java
 Some debug.

Signed-off-by: Yi Mei
2020-04-01 08:38:05 -07:00
Viraj Jasani 0dcbf80583
HBASE-23937 : Support Online LargeLogs similar to SlowLogs APIs (#1346)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-04-01 18:22:22 +05:30
Jan Hentschel fb17ecdd07
HBASE-23842 Removed deprecated Scan(byte[], byte[]) from Scan
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-04-01 13:11:21 +02:00
Viraj Jasani 834f89da7b
HBASE-23678 : Builder API for version management - setVersionsWithTim… (#1381)
Signed-off-by: Xu Cang <xucang@apache.org>
2020-04-01 16:16:40 +05:30
binlijin 52c101389e
HBASE-24096 Fix TestRSGroupsBalance (#1402)
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-04-01 14:16:57 +08:00
Bharath Vissapragada 6f213e9d5a HBASE-24075: Fix a race between master shutdown and metrics (re)init
JMXCacheBuster resets the metrics state at various points in time. These
events can potentially race with a master shutdown. When the master is
tearing down, metrics initialization can touch a lot of unsafe state,
for example invalidated FS objects. To avoid this, this patch makes
the getMetrics() a no-op when the master is either stopped or in the
process of shutting down. Additionally, getClusterId() when the server
is shutting down is made a no-op.

Simulating a test for this is a bit tricky but with the patch I don't
locally see the long stacktraces from the jira.

Signed-off-by: Michael Stack <stack@apache.org>
2020-03-31 16:36:31 -07:00
Jan Hentschel fded2b9ddc
HBASE-23846 Removed deprecated setMaxVersions(int) from Scan
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-31 10:18:27 +02:00
stack 3d4124c92a HBASE-24079 [Flakey Tests] Misc fixes and debug; fix BindException in Thrift tests; add waits on quota table to come online; etc.
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientAsyncPrefetchScanner.java
 Refactor to avoid NPE timing issue referencing lock during Construction.

hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
 Comment

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
 Refactor. Catch NPE during startup and return it instead as failed initialization.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplit.java
 Catch IndexOutOfBounds exception and convert to non-split request.

hbase-server/src/test/java/org/apache/hadoop/hbase/TestCachedClusterId.java
 Make less furious. Make it less flakie.

hbase-server/src/test/java/org/apache/hadoop/hbase/TestServerSideScanMetricsFromClientSide.java
 Debug. Catch exception to log, then rethrow.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java
 Guess that waiting longer on compaction to succeed may help make this
 less flakey.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java
 Be explicit about timestamping to avoid concurrent edit landing
 server-side and messing up test expectation.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMasterRegistry.java
 Add wait on meta before proceeding w/ test.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannersFromClientSide.java
 Be explicit that edits are distinct.

hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketCacheRefCnt.java
 Add @Ignore on RAM test... Fails sporadically.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestRegionMoveAndAbandon.java
 Add wait for all RegionServers going down before proceeding; was
 messing up RS accounting.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
 Make balancer test sloppier; less restrictive; would fail on occasion
 by being just outside test limits.

hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaObserverChoreRegionReports.java
 Add wait on quota table coming up; helps make this less flakie.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
 Be explicity about timestamps; see if helps w/ flakie failure.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
 Catch and ignore if issue in shutdown; don't care if after test.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerReportForDuty.java
 Comment.

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
 Add retry to see if helps w/ odd failure; grant hasn't propagated?

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestCellACLWithMultipleVersions.java
 Explicit w/ timestamps so no accidental overlap of puts.

hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftHttpServer.java
hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerCmdLine.java
 Hack to deal w/ BindException on startup.

hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift2/TestThrift2ServerCmdLine.java
 Use loopback.

hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift2/TestThriftHBaseServiceHandler.java
 Disable flakie test.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-03-30 17:08:17 -07:00
stack 030e833dc9 HBASE-24052 Add debug+fix to TestMasterShutdown
Add check for stopped server at a few more points in Master startup.
Defend against NPE in RSProcedureDispatcher; log and retun instead.
2020-03-30 16:07:35 -07:00
huaxiangsun d318ca1741
HBASE-24073 [flakey test] client.TestAsyncRegionAdminApi messed up compaction state. (#1387)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-30 12:11:26 -07:00
huaxiangsun f365cb5264
Revert "HBASE-24073 [flakey test] client.TestAsyncRegionAdminApi messed up compaction state. (#1378)" (#1386)
This reverts commit d319ce8c6f.
2020-03-30 10:06:13 -07:00
huaxiangsun d319ce8c6f
HBASE-24073 [flakey test] client.TestAsyncRegionAdminApi messed up compaction state. (#1378)
Co-authored-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-03-30 09:29:18 -07:00
Duo Zhang 5d4e020c3d
HBASE-23798 Remove hbase-prototcol module (#1360)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: stack <stack@apache.org>
2020-03-30 09:54:59 +08:00
Jan Hentschel aaae46c976
HBASE-23845 Removed deprecated setMaxVersions from Scan (#1208)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-29 17:46:49 +08:00
WenFeiYi eb1fec1c53
HBASE-24040 WALFactory.Providers.multiwal causes StackOverflowError (#1338)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-27 10:04:52 +08:00
Josh Elser 78eecd3a7d HBASE-23381 Ensure Netty client receives at least one response before considering SASL negotiation complete
The PLAIN mechanism test added in the Shade authentication example has
different semantics than GSSAPI mechanism -- the client reports that the
handshake is done after the original challenge is computed. The javadoc
on SaslClient, however, tells us that we need to wait for a response
from the server before proceeding.

The client, best as I can see, does not receive any data from HBase;
however the application semantics (e.g. throw an exception on auth'n
error) do not work as we intend as a result of this bug.

Extra trace logging was also added to debug this, should a similar error
ever happen again with some other mechanism.

Closes #1260

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-03-26 18:34:47 -04:00
huaxiangsun 50c8f49c31
HBASE-23853 [Flakey Test] TestBlockEvictionFromClient#testBlockRefCountAfterSplits (#1363)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
  Signed-off-by: Viraj Jasani <vjasani@apache.org>
  Signed-off-by: <stack@apache.org>
2020-03-26 14:06:34 -07:00
Peter Somogyi 9042bc0058
HBASE-22555 Re-enable TestMasterOperationsForRegionReplicas (#1345)
Signed-off-by: stack <stack@apache.org>
2020-03-26 11:01:19 +01:00
Duo Zhang f3ee9b8aa3
HBASE-24000 Simplify CommonFSUtils after upgrading to hadoop 2.10.0 (#1335)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-26 17:05:31 +08:00
Guanghao Zhang eed730e492
HBASE-24037 Add ut for root dir and wal root dir are different (#1336)
Signed-off-by: stack <stack@apache.org>
2020-03-25 10:53:14 +08:00
Jan Hentschel aa966e354a
HBASE-23983 Fixed Spotbugs complaint in RegionStates related to ignored return value
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-25 00:02:54 +01:00
Wei-Chiu Chuang 84977eeebb
HBASE-8868. add metric to report client shortcircuit reads. (#1334)
Signed-off-by: stack <stack@apache.net>
2020-03-24 15:30:08 -07:00
stack 2ca0a105bc HBASE-24043 [Flakey Tests] TestAsyncRegionAdminApi, TestRegionMergeTransactionOnCluster fixes and debug
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java
 Edit of log about archiving that shows in middle of a table create;
 try to make it less disorientating.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java
 Loosen assert. Compaction may have produced a single file only. Allow
 for this.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableGetMultiThreaded.java
 Make this test less furious given it is inline w/ a bunch of unit
 tests.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java
 Add debug

hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaObserverChoreRegionReports.java
 Add wait on quota table to show up before moving forward; otherwise,
 attempt at quota setting fails.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
 Debug

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java
 Remove asserts that expected regions to still have a presence in fs
 after merge when a catalogjanitor may have cleaned up parent dirs.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
 Catch exception on way out and log it rather than let it fail test.

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController.java
 Wait on acl table before proceeding.
2020-03-24 14:52:24 -07:00
Reid Chan c3a704ea22
HBASE-24032 [RSGroup] Assign created tables to respective rsgroup automatically instead of manual operations (#1319)
Signed-off-by: binlijin <binlijin@gmail.com>
2020-03-24 13:33:00 +08:00
Reid Chan bdcfd6a4a8
[RSGroup] Forward-port HBASE-22658 to master branch and branch-2.x (#1326)
Signed-off-by: stack <stack@apache.org>
2020-03-24 10:34:12 +08:00
stack afc1746757 HBASE-24034 [Flakey Tests] A couple of fixes and cleanups
hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupMajorCompactionTTL.java
 Remove spurious assert. Just before this it waits an arbitrary 10
 seconds. Compactions could have completed inside this time. The spirit
 of the test remains.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
 Get log cleaner to go down promptly; its sticking around. See if this
 helps with TestMasterShutdown

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
 We get a rare NPE trying to sync. Make local copy of SyncFuture and see
 if that helps.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java
 Compaction  may have completed when not expected; allow for it.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestBlockEvictionFromClient.java
 Add wait before testing. Compaction may not have completed. Let
 compaction complete before progressing and then test for empty cache.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
 Less resources.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java
 Less resources.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
 Wait till online before we try and do compaction (else request is
 ignored)

hbase-server/src/test/java/org/apache/hadoop/hbase/tool/TestCanaryTool.java
 Disable test that fails randomly w/ mockito complaint on some mac os
 x's.

TestMasterShutdown... fix NPE in RSRpcDispatcher... catch it and covert
to false and have master check for successful startup.
2020-03-23 16:26:32 -07:00
stack 2f26d8a623 HBASE-24035 [Flakey Tests] Disable TestClusterScopeQuotaThrottle#testUserNamespaceClusterScopeQuota 2020-03-23 13:44:34 -07:00
Guanghao Zhang 8130858199
HBASE-23741 Data loss when WAL split to HFile enabled (#1254)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-23 14:38:47 +08:00
Duo Zhang eface74407
HBASE-23799 Make our core coprocessors use shaded protobuf (#1280)
Signed-off-by: stack <stack@apache.org>
2020-03-23 08:47:02 +08:00
Huaxiang Sun 542244056c
HBASE-23957 [flakey test] client.TestMultiParallel fails to read hbase-site.xml (#1310)
Signed-off-by: Nick Dimiduk ndimiduk@apache.org
Signed-off-by: stack <stack@apache.org>
2020-03-22 17:43:04 -07:00
Guanghao Zhang 2a60a61a73
HBASE-24033 Add ut for loading the corrupt recovered hfiles (#1322)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-22 22:56:30 +08:00
niuyulin d4ab645531
HBASE-23949 refactor loadBalancer implements for rsgroup balance by table to achieve overallbalanced (#1279)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-22 21:54:57 +08:00
Pankaj ca533a77cb
HBASE-23633 Find a way to handle the corrupt recovered hfiles (#1233)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-22 16:39:56 +08:00
Toshihiro Suzuki 9b4e75a7f6
HBASE-24030 Add necessary validations to HRegion.checkAndMutate() and HRegion.checkAndRowMutate() (#1315)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-22 11:55:52 +09:00
Toshihiro Suzuki b53fa3c51f
HBASE-24031 TestHRegion.testCheckAndMutate_WithFilters is flaky (#1316)
Signed-off-by: Viraj Jasani <virajjasani007@gmail.com>
2020-03-22 05:04:16 +09:00
Michael Stack 5592d0c191 HBASE-23984 [Flakey Tests] TestMasterAbortAndRSGotKilled fails in teardown (#1311)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 Change parameter name and add javadoc to make it more clear what the
 param actually is.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
 Move postOpenDeployTasks so if it fails to talk to the Master -- which
 can happen on cluster shutdown -- then we will do cleanup of state;
 without this the RS can get stuck and won't go down.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/CloseRegionHandler.java
 Add handleException so CRH looks more like UnassignRegionHandler and
 AssignRegionHandler around exception handling. Add a bit of doc on
 why CRH.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
 Right shift most of the body of process so can add in a finally
 that cleans up rs.getRegionsInTransitionInRS is on exception
 (otherwise outstanding entries can stop a RS going down on cluster
 shutdown)

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-20 15:26:58 -07:00
Viraj Jasani 44b3ac6457
HBASE-23977 : Resolve flakes present in TestSlowLogRecorder (ADDENDUM) 2020-03-20 13:27:06 +05:30
Viraj Jasani 26b31e351c
HBASE-23977 : Resolve flakes present in TestSlowLogRecorder (#1286)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-19 15:35:49 +05:30
Michael Stack 3e1cf00c71 HBASE-23993 Use loopback for zk standalone server in minizkcluster (#1291)
hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Have client and server use loopback instead of 'localhost'

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-17 20:26:06 -07:00
stack 9804f73c66 HBASE-23971 protoc warns: "no protobuf syntax specified" (#1278)
Remove overcommit in TestUsersOperationsWithSecureHadoop
(Noticed by ndimiduk)
2020-03-17 15:28:26 -07:00
Wei-Chiu Chuang 60de4c2e31
HBASE-22103. HDFS-13209 in Hadoop 3.3.0 breaks asyncwal. (#1284)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-17 14:36:56 +08:00
Jan Hentschel 1b163d98b9
HBASE-23847 Removed deprecated setStartRow from Scan (#1220)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-13 08:43:17 +08:00
stack 02b6bbe7ce HBASE-23979 Disable TestSlowLogRecorder until parent addressed 2020-03-12 17:24:21 -07:00
WenFeiYi abf0a22259
HBASE-23971 protoc warns: "no protobuf syntax specified" (#1278)
Signed-off-by: stack <stack@apache.org>
2020-03-12 08:32:22 -07:00
stack 26f3702735 HBASE-23974 [Flakey Tests] Allow that server may not yet be cleared from DeadServers in TestHBCKSCP 2020-03-11 23:04:16 -07:00
stack d42e9768d0 HBASE-23956 Use less resources running tests (#1266)
Addendum: TestNamespaceCommands was using up the limit on
nioeventloopgroup.... so create them per run of client.
2020-03-11 22:48:22 -07:00
stack 20658c5139 HBASE-23956 Use less resources running tests (#1266)
Addendum: Two more tests that had issues because parent cut
down the number of default RegionServers launched.
2020-03-11 21:03:10 -07:00
stack 4b64558ebc HBASE-23956 Use less resources running tests (#1266)
Addendum: This test was depending on hard-coded number for
log cleaners; the parent patch made them less in test context.
2020-03-11 20:34:29 -07:00
Duo Zhang c5aa2bc121
HBASE-23077 move entirely to spotbugs (#1265)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-12 10:18:09 +08:00
Minwoo Kang 6f1e8abc30
HBASE-23561 acquire lock before update regionsMap (#1263)
Addendum: tie add and delete of regionMap and encodedRegionsMap

Signed-off-by: stack <stack@apache.org>
2020-03-11 16:57:21 -07:00
Michael Stack aaf9a148a0 HBASE-23956 Use less resources running tests (#1266)
Add being able to configure netty thread counts. Enable socket reuse
(should not have any impact).

hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java
 Rename the threads we create in here so they are NOT named same was
 threads created by Hadoop RPC.

hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java
 Allow configuring eventloopgroup thread count (so can override for
 tests)

hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java
 Enable socket resuse.

hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java
 Enable socket resuse and config for how many threads to use.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java
 Thread name edit; drop the redundant 'Thread' suffix.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java
 Make closeable and shutdown executor when called.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
 Call close on HFileReplicator

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
 HDFS creates lots of threads. Use less of it so less threads overall.

hbase-server/src/test/resources/hbase-site.xml
hbase-server/src/test/resources/hdfs-site.xml
 Constrain resources when running in test context.

hbase-server/src/test/resources/log4j.properties
 Enable debug on netty to see netty configs in our log

pom.xml
 Add system properties when we launch JVMs to constrain thread counts in
 tests

 Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-11 10:37:50 -07:00
stack d019fd2eed HBASE-23963 Split TestFromClientSide; it takes too long to complete timing out
Split TestGetFromClientSide. Means have to also split the superclass
TestFromClientSideWithCoprocessor.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-10 22:13:24 -07:00
niuyulin e345a95461
HBASE-23953 SimpleBalancer bug when second pass to fill up to min (#1262)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-11 10:30:02 +08:00
niuyulin 6e6b241e2b
HBASE-23954 SplitParent region should not be balanced (#1264)
Co-authored-by: niuyulin <niuyulin@xiaomi.com>

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-03-10 11:14:43 -07:00
Michael Stack e1e8f396ca HBASE-23851 Log networks and bind addresses when multicast publisher/listener enabled (#1173)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-10 11:00:00 -07:00
Nick Dimiduk 5750876191
HBASE-23932 Minor improvements to Region Normalizer (#1258)
- consolidate checks made by master on behalf of balancer and
   normalizer: deciding if the master is in a healthy state for
   running any actions at all (skipRegionManagementAction). Normalizer
   now does as balancer did previously.
 - both balancer and normalizer make one final check on above
   conditions between calculating an action plan and executing the
   plan. should make the process more responsive to shutdown
   requests.
 - change normalizer to only consider acting on a region when it is in
   the OPEN state. previously we would normalizer attempt to merge a
   region that was already in a MERGING_NEW,MERGING,MERGED state.
 - fix some typos in variable names.

Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: binlijin <binlijin@gmail.com>
2020-03-10 10:08:37 -07:00
nyl3532016 819965a35a
HBASE-23944 The method setClusterLoad of SimpleLoadBalancer is incorrect when balance by table (#1243)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-07 16:18:41 +08:00
Guanghao Zhang 9b0d214b7b
HBASE-23895 STUCK Region-In-Transition when failed to insert procedure to procedure store (#1221)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-03-07 15:50:09 +08:00
Guanghao Zhang df62dde394
HBASE-23739 BoundedRecoveredHFilesOutputSink should read the table descriptor directly (#1223)
Signed-off-by: Pankaj <pankajkumar@apache.org>
2020-03-06 19:06:59 +08:00
Wellington Ramos Chevreuil 9c67abef18
HBASE-23939 Remove unused variables from HBaseSaslRpcServer (#1239)
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-03-06 10:05:04 +00:00
Abhinaba Sarkar b6eefcaeb7
HBASE-23788 ROW_INDEX_V1 encoder should consider the secondary index size with the encoded data size tracking (#1219)
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
2020-03-04 23:14:43 +05:30
Wei-Chiu Chuang 6d9802fc2e
HBASE-23861. Reconcile Hadoop version. (#1179)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-04 08:01:40 -08:00
stack f411e395d3 HBASE-22978 : Online slow response log (#1228)
Addendum to fix errorprone ERROR
2020-03-03 09:19:21 -08:00
Guanghao Zhang 5dcbe687ae
HBASE-23912 Resolve the TODO of FSTableDescriptor's construct method (#1225)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-03 22:13:16 +08:00
Viraj Jasani c0301e3fdf
HBASE-23868 : Replace usages of HColumnDescriptor(byte [] familyName)… (#1222)
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-03-02 17:20:03 +05:30
Duo Zhang 7f2d823164 HBASE-23818 Cleanup the remaining RSGroupInfo.getTables call in the code base (#1152)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 7386369fec HBASE-23276 Add admin methods to get tables within a group (#1118)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 5de0a5ef7e HBASE-23235 Re-enable TestRSGroupsKillRS.testLowerMetaGroupVersion (#1117)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang c8d892cdef HBASE-23253 Rewrite rsgroup related UTs with the new methods introduced in HBASE-22932 (#813)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
linkaline 72cbb129a4 HBASE-22932 Add rs group management methods in Admin and AsyncAdmin (#657)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-02 15:43:40 +08:00
Guangxu Cheng e8e9eecef1 HBASE-23050 Use RSGroupInfoManager to get rsgroups in master UI's rsgroup part (#776)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 0f4a87c24d HBASE-23081 Add an option to enable/disable rs group feature (#691)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang b10b39ad03 HBASE-22971 Deprecated RSGroupAdminEndpoint and make RSGroup feature always enabled (#595)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 0e29d62a2d HBASE-22987 Calculate the region servers in default group in foreground (#599)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Guanghao Zhang 2bf2781df9 HBASE-22729 Start RSGroupInfoManager as default (#555)
Amending-Author: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang b3399ef365 HBASE-22819 Automatically migrate the rs group config for table after HBASE-22695 (#498)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
linkaline 2e0039c2a3 HBASE-22820 Do not need to persist default rs group now (#482)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 44ddb30914 HBASE-22809 Allow creating table in group when rs group contains no live servers (#464)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang 3709f335a8 HBASE-22695 Store the rsgroup of a table in table configuration (#426)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
Duo Zhang f0a13bb74d HBASE-22676 Move all the code in hbase-rsgroup to hbase-server and remove hbase-rsgroup module (#399)
Signed-off-by: Zheng Hu <openinx@gmail.com>
2020-03-02 15:43:40 +08:00
Duo Zhang 662149e193 HBASE-22662 Move RSGroupInfoManager to hbase-server (#368)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-03-02 15:43:40 +08:00
stack 6777e2c2d1 HBASE-23899 [Flakey Test] Stabilizations and Debug
A miscellaney. Add extra logging to help w/ debug to a bunch of tests.
Fix some issues particular where we ran into mismatched filesystem
complaint. Some modernizations, removal of unnecessary deletes
(especially after seeing tests fail in table delete), and cleanup.
Recategorized one tests because it starts four clusters in the one
JVM from  medium to large. Finally, zk standalone server won't come
on occasion; added debug and thread dumping to help figure why (
manifests as test failing in startup saying master didn't launch).

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
  Fixes occasional mismatched filesystems where the difference is file:// vs file:///
  or we pick up hdfs schema when it a local fs test. Had to do this
  vetting of how we do make qualified on a Path in a few places, not
  just here as a few tests failed with this same issue. Code in here is
  used by a lot of tests that each in turn suffered this mismatch.

  Refactor for clarity

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotV1NoCluster.java
  Unused import.

hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/TestWALProcedureStore.java
  This test fails if tmp dir is not where it expects because tries to
  make rootdir there. Give it a rootdir under test data dir.

hbase-server/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java
  This change is probably useless. I think the issue is actually
  a problem addressed later where our test for zk server being
  up gets stuck and never times out.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSplitOrMergeStatus.java
 Move off deprecated APIs.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
 Log when we fail balance check for DEBUG Currently just says 'false'

hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestSplitWALProcedure.java
 NPEs on way out if setup failed.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
 Add logging when assert fails to help w/ DEBUG

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerAbortTimeout.java
 Don't bother removing stuff on teardown. All gets thrown away anyways.
 Saw a few hangs in here in the teardown where hdfs was down before
 expected messing up shutdown.

hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Add timeout on socket; was seeing check for zk server getting stuck
 and never timing out (test time out in startup)

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotWithTemporaryDirectory.java
 Write to test data dir instead.
 Be careful about how we make qualified paths.

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java
 Remove snowflake configs.

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStatus.java
 Add a hacky pause. Tried adding barriers but didn't work. Needs deep
 dive.

hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Remove code copied from zk and use zk methods directly instead.
 A general problem is that zk cluster doesn't come up occasionally but
 no clue why. Add thread dumping and state check.
2020-02-28 13:39:38 -08:00
Wellington Ramos Chevreuil 00ef6c624a
HBASE-23892 SecureTestCluster should allow its subclasses to pass their Class reference on HBaseKerberosUtils.setSSLConfiguration (#1207)
Signed-off-by: Josh Elser <elserj@apache.org>
2020-02-28 18:35:15 +00:00
Viraj Jasani b24ea32b3b
HBASE-22978 : Online slow response log
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-02-28 19:47:59 +05:30
Pankaj 3c3aae9a2f
HBASE-23740 Invalid StoreFile WARN log message printed for recovered.… (#1198)
* HBASE-23740 Invalid StoreFile WARN log message printed for recovered.hfiles directory

* HBASE-23740 Invalid StoreFile WARN log message printed for recovered.hfiles directory
2020-02-26 17:52:21 +05:30
Toshihiro Suzuki ecbed33092
HBASE-23146 Support CheckAndMutate with multiple conditions (#1114)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-02-26 08:09:04 +09:00
Jan Hentschel 9f223c2236
HBASE-23781 Removed deprecated createTableDescriptor(String) from HBaseTestingUtility
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-25 21:10:09 +01:00
Guanghao Zhang 418234356f HBASE-23864 No need to submit SplitTableRegionProcedure/MergeTableRegionsProcedure when split/merge is disabled (addendum) 2020-02-24 22:11:41 +08:00
Minwoo Kang 3ab0135c0e
HBASE-23561 Look up of Region in Master by encoded region name is O(n) (#1193)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-21 16:40:45 +00:00
Bharath Vissapragada c34dbc3c29 HBASE-23804: Fix default master addr hostname in master registry (#1137)
Master rpc server end point doesn't bind to localhost's
IP address by default. Instead, it looks up the hostname and
binds to the endpoint to which it resolves. MasterRegistry should
do the same when building the default server end point to talk to.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-02-19 20:10:17 -08:00