Commit Graph

8818 Commits

Author SHA1 Message Date
Duo Zhang 1540b89cee
HBASE-25420 Some minor improvements in rpc implementation (#2792)
Signed-off-by: XinSun <ddupgs@gmail.com>
Signed-off-by: stack <stack@apache.com>
2020-12-20 11:26:36 +08:00
Akshay Sudheer 7b1e9cd0d2
HBASE-25404 Procedures table Id under master web UI gets word break to single character (#2783)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-12-19 12:37:21 -08:00
Duo Zhang 33441f540a Revert "HBASE-25368 Filter out more invalid encoded name in isEncodedRegionName(byte[] regionName) (#2753)"
This reverts commit c327680125.
2020-12-19 20:28:30 +08:00
Sandeep Pal f600856a3b
HBASE-25383: Ability to update and remove peer base config
Closes #2778

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Geoffrey Jacoby <gjacoby@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-18 14:19:17 +05:30
stack 888a4fb86a HBASE-25400 [Flakey Tests] branch-2 TestRegionMoveAndAbandon 2020-12-16 10:39:11 -08:00
shahrs87 74d68180e6
HBASE-25246 Backup/Restore hbase cell tags
Closes #2745

Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-16 18:52:36 +05:30
huaxiangsun c327680125
HBASE-25368 Filter out more invalid encoded name in isEncodedRegionName(byte[] regionName) (#2753)
Signed-off-by: stack <stack@apache.com>
2020-12-15 21:52:54 -08:00
huaxiangsun 1c217da2ff
HBASE-25293 Followup jira to address the client handling issue when chaning from meta replica to non-meta-replica at the server side. (#2768)
Signed-off-by: stack <stack@apache.org>
2020-12-15 21:45:39 -08:00
Baiqiang Zhao 1bb9b78787
HBASE-25365 The log in move_servers_rsgroup is incorrect (#2742)
Signed-off-by: stack <stack@apache.or>
2020-12-15 20:25:59 -08:00
Pankaj 9bdac6cd17
HBASE-25378 Legacy comparator in Hfile trailer will fail to load (#2756)
Signed-off-by: stack <stack@apache.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-15 23:56:02 +05:30
Michael Stack a4d42d1fcc HBASE-25389 [Flakey Tests] branch-2 TestMetaShutdownHandler (#2773)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-12-15 09:51:56 -08:00
Bo Cui d50816fe44
HBASE-23340 hmaster /hbase/replication/rs session expired (hbase repl… (#2739)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-12-12 21:10:33 +08:00
Huaxiang Sun e04792959a Revert "HBASE-25293 Followup jira to address the client handling issue when chaning from meta replica to non-meta-replica at the server side."
This reverts commit c1aa3b24e9.
2020-12-11 10:22:41 -08:00
haxl 635c911532
HBASE-25287 Forgetting to unbuffer streams results in many CLOSE_WAIT sockets when loading files (#2699)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-12-10 22:15:39 +08:00
haxl a9b8c10f80
HBASE-25334 TestRSGroupsFallback.testFallback is flaky (#2728)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-12-10 22:07:06 +08:00
Huaxiang Sun c1aa3b24e9 HBASE-25293 Followup jira to address the client handling issue when chaning from meta replica to non-meta-replica at the server side. 2020-12-09 09:12:01 -08:00
haxiaolin 56dd3eba81 HBASE-25363 Improve performance of HFileLinkCleaner by using ReadWriteLock instead of synchronize 2020-12-08 21:09:31 +08:00
Pankaj fb6e498b32
HBASE-25277 postScannerFilterRow impacts Scan performance a lot in HBase 2.x (#2675)
* HBASE-25277 postScannerFilterRow impacts Scan performance a lot in HBase 2.x

1. Added a check for Object class in RegionCoprocessorHost to avoid wrong initialization of hasCustomPostScannerFilterRow
2. Removed dummy implementation of postScannerFilterRow from AccessController, VisibilityController & ConstraintProcessor (which are not required currently)

Signed-off-by Ramkrishna S Vasudevan <ramkrishna@apache.org>
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-12-07 23:00:48 +05:30
stack 7d0a687e57 HBASE-25361 [Flakey Tests] branch-2 TestMetaRegionLocationCache.testStandByMetaLocations (#2736)
Addendum; Reset counter so we actually wait in the new loop added by the
above.
2020-12-05 14:01:56 -08:00
Michael Stack 6791b29771 HBASE-25361 [Flakey Tests] branch-2 TestMetaRegionLocationCache.testStandByMetaLocations (#2736)
Add a bit of a wait before testing if online replicas match the zk count. It might take a
while for all replicas to come online.

Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
2020-12-05 08:56:44 -08:00
Michael Stack 946fa81715 HBASE-25349 [Flakey Tests] branch-2 TestRefreshRecoveredReplication.testReplicationRefreshSource:141 Waiting timed out after [60,000] msec (#2731)
Start the check for recovered queue presence earlier.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-12-02 09:58:12 -08:00
lujiefsi 8938b7a678
HBASE-25332:fix One pontential NPE
Closes #2715

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-01 19:41:38 +05:30
Duo Zhang 86bb037eb0 Revert "HBASE-25334 TestRSGroupsFallback.testFallback is flaky (#2712)"
This reverts commit f71eb27be1.
2020-12-01 16:22:09 +08:00
haxl f71eb27be1
HBASE-25334 TestRSGroupsFallback.testFallback is flaky (#2712)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-12-01 15:24:02 +08:00
Michael Stack 7da90a1e23 HBASE-25345 [Flakey Tests] branch-2 TestReadReplicas#testVerifySecondaryAbilityToReadWithOnFiles (#2727)
Check TEST_SKIP_REPORTING_TRANSITION and if true, skip trying to talk to
update master on state transition -- i.e. reportFileArchivalForQuotas --
as we allow for reportRegionStateTransition (For tests only)

Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-11-30 21:04:51 -08:00
GeorryHuang 264412d30c
HBASE-25339 Method parameter and member variable are duplicated in checkSplittable() of SplitTableRegionProcedure
Closes #2717

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-30 19:49:46 +05:30
Viraj Jasani a8a35da73e
Revert "HBASE-25339 Method parameter and member variable are duplicated in checkSplittable() of SplitTableRegionProcedure"
This reverts commit c76145c505.
2020-11-30 19:48:58 +05:30
GeorryHuang c76145c505
HBASE-25339 Method parameter and member variable are duplicated in checkSplittable() of SplitTableRegionProcedure
Closes #2717

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-30 17:59:45 +05:30
Duo Zhang fc62214e77
HBASE-25341 Fix ErrorProne error which causes nightly to fail (#2718)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-30 14:14:44 +08:00
Akshay Sudheer bfa5cb7d48
HBASE-25321 The sort icons not shown after Upgrade JQuery to 3.5.1 (#2694)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-29 12:08:57 +08:00
Baiqiang Zhao e2c1b10cb7
HBASE-25330 RSGroupInfoManagerImpl#moveServers return is not set of servers moved (#2708)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-29 11:32:51 +08:00
Andrew Purtell 679dd7b8f9
HBASE-25292 Improve InetSocketAddress usage discipline (#2669)
Network identities should be bound late. Remote addresses should be
resolved at the last possible moment, just before connect(). Network
identity mappings can change, so our code should not inappropriately
cache them. Otherwise we might miss a change and fail to operate normally.

Revert "HBASE-14544 Allow HConnectionImpl to not refresh the dns on errors"
Removes hbase.resolve.hostnames.on.failure and related code. We always
resolve hostnames, as late as possible.

Preserve InetSocketAddress caching per RPC connection. Avoids potential
lookups per Call.

Replace InetSocketAddress with Address where used as a map key. If we want
to key by hostname and/or resolved address we should be explicit about it.
Using Address chooses mapping by hostname and port only.

Add metrics for potential nameservice resolution attempts, whenever an
InetSocketAddress is instantiated for connect; and metrics for failed
resolution, whenever InetSocketAddress#isUnresolved on the new instance
is true.

* Use ServerName directly to build a stub key

* Resolve and cache ISA on a RpcChannel as late as possible, at first call

* Remove now invalid unit test TestCIBadHostname

We resolve DNS at the latest possible time, at first call, and do not
resolve hostnames for creating stubs at all, so this unit test cannot
work now.

Reviewed-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-28 21:01:22 +08:00
Lokesh Khurana e6165412e0
HBASE-25237 : 'hbase master stop' shuts down the cluster, not the master only
Closes #2705

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-27 16:02:30 +05:30
Qi Yu dc48071e01
HBASE-25324 Remove unnecessary array to list conversion in SplitLogManager (#2703)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.com>
2020-11-27 08:20:24 +08:00
Mate Szalay-Beko 451a4b06b1 HBASE-25263 Various improvements of column family encryption
This PR is a follow-up of HBASE-25181 (#2539), where several issues were
discussed on the PR:

1. Currently we use PBKDF2WithHmacSHA1 key generation algorithm to generate a
secret key for HFile / WalFile encryption, when the user is defining a string
encryption key in the hbase shell. This algorithm is not secure enough and
not allowed in certain environments (e.g. on FIPS compliant clusters). We are
changing it to PBKDF2WithHmacSHA384. It will not break backward-compatibility,
as even the tables created by the shell using the new algorithm will be able
to load (e.g. during bulkload / replication) the HFiles serialized with the
key generated by an old algorithm, as the HFiles themselves already contain
the key necessary for their decryption.

Smaller issues fixed by this commit:

2. Improve the documentation e.g. with the changes introduced by HBASE-25181
and also by some points discussed on the Jira ticket of HBASE-25263.

3. In EncryptionUtil.createEncryptionContext the various encryption config
checks should throw IllegalStateExceptions instead of RuntimeExceptions.

4. Test cases in TestEncryptionTest.java should be broken down into smaller
tests.

5. TestEncryptionDisabled.java should use ExpectedException JUnit rule to
validate exceptions.

closes #2676

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-11-26 16:44:34 +01:00
Toshihiro Suzuki b142f5dcd2
HBASE-25242 Add Increment/Append support to RowMutations (#2630)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-11-26 13:31:43 +09:00
Andrew Purtell 1726160839
HBASE-24640 Purge use of VisibleForTesting (#2695)
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-11-25 14:10:12 -08:00
Qi Yu 403756ade5
HBASE-25282 Remove processingServers in DeadServer as we can get this… (#2657)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-25 15:16:27 +08:00
Qi Yu 984d578dd4
HBASE-25323 Fix potential NPE when the zookeeper path of RegionServerTracker does not exist when start (#2702)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-25 15:13:09 +08:00
Qi Yu 31d880b5dd
HBASE-25325 Remove unused class ClusterSchemaException (#2704)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-25 15:09:40 +08:00
niuyulin 4affae5938
HBASE-25213 Should request Compaction when bulkLoadHFiles is done (addendum) (#2701)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-25 07:57:18 +08:00
tianhang 390abb520c
HBASE-25299 Deprecate Scan#setRowPrefixFilter because of an unexpected behavior
Closes #2674

Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-24 19:39:25 +05:30
ramkrish86 f221d11227
HBASE-25050 - We initialize Filesystems more than once. (#2419)
* HBASE-25050 - We initialize Filesystems more than once.

* Ensuring that calling the FS#get() will only ensure FS init.

* Fix for testfailures. We should pass the entire path and no the scheme
alone

* Cases where we don't have a scheme for the URI

* Address review comments

* Add some comments on why FS#get(URI, conf) is getting used

* Adding the comment as per Sean's review

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2020-11-24 15:31:04 +05:30
XinSun 1cd8f3cf94
HBASE-24654 Allow unset table's rsgroup (#1994)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-23 17:22:37 +08:00
Bo Cui b0f20399e7
HBASE-25311 UI throws NPE (#2688)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Wellington Chevreuil <wellington.chevreuil@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-23 12:07:42 +08:00
niuyulin a307d706c8
HBASE-25281 Bulkload split hfile too many times due to unreasonable split point (#2667)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Wellington Chevreuil <wellington.chevreuil@gmail.com>
2020-11-23 10:07:19 +08:00
stack eca904e0fb HBASE-25291 Document how to enable the meta replica load balance mode for the client and clean up around hbase:meta read replicas
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSelector.java
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSelectorFactory.java
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSimpleSelector.java
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaMode.java
 Make the leading license comment opener align on all these new
 files.... one star rather than two.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java
 Add some comment better situating the catalogreplicationsource
 specialization.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
 Add comment how this differs from replicationsource factory and why no
 need to keep offset up in zk.

src/main/asciidoc/_chapters/architecture.adoc
 Clean up the read replica of META doc. Introduce the new read replica
 liveness on META and the new 'LoadBalance' feature (with warnings).
 General edit.
2020-11-20 00:04:48 -08:00
Huaxiang Sun 9ecf6ff6a2 HBASE-25126 Add load balance logic in hbase-client to distribute read load over meta replica regions (addendum) 2020-11-20 00:04:48 -08:00
Huaxiang Sun 690b4d8973 HBASE-25126 Add load balance logic in hbase-client to distribute read load over meta replica regions
It adds load balance support for meta lookup in AsyncTableRegionLocator.
The existing meta replica mode is renamed as "HedgedRead", client sends scan request to the primary meta replica region first,
if response is not back within a configured amount of time, it will send scan requests to all meta replica regions and
take the first response. On top of the existing mode, a new mode "LoadBalance" is introduced. In this mode, client first
choose a meta replica region to send scan request. If the response is stale, it may send the request to another meta replica region or
the primary region. In this mode, meta scan requests are load balanced across all replica regions with the primary mode as
the ultimate source of truth.

Two new config knobs are added:

1. hbase.locator.meta.replicas.mode
   Valid options are "None", "HedgedRead" and "LoadBalance", they are case insensitive. The default mode is "None".

2. hbase.locator.meta.replicas.mode.loadbalance.selector
   The load balance alogrithm to select a meta replica to send the requests.
   Only org.apache.hadoop.hbase.client.CatalogReplicaLoadBalanceReplicaSimpleSelector.class
   is supported for now, which is the default as well. The algorithm works as follows:
      a. Clients select a randome meta replica region to send the requests if there is no entry for the location in the stale
         location cache.
      b. If the location from one meta replica region is stale, a stale entry will be created in the statle location cache
         for the region.
      c. Clients select the primary meta region if the location is in the stale location cache.
      d. The stale location cache entries time out in 3 seconds.

If there is no "hbase.locator.meta.replicas.mode" configured, it will check the config knob "hbase.meta.replicas.use".
If "hbase.meta.replicas.use" is configured, the mode will be set to "HedgedRead".
2020-11-20 00:04:48 -08:00
stack 40843bb567 HBASE-25151 warmupRegion frustrates registering WALs on the catalog replicationsource
warmupRegion called by Master on Region move will instatiate
the meta WALProvider as part of its action making it so
it is already created by the time we go to open the
hbsae:meta Region. Accommodate meta walProvider
being already up.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
 Pass regionInfo. Needed internally.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
 Add handling if meta wal provider already instantiated when
 addCatalogReplicationSource runs.

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java
 Add exercising moving meta around between servers. Test replication
 keeps working.
2020-11-20 00:04:48 -08:00
Michael Stack 89cb0c5370 HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable w… (#2451)
* HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable when hbase:meta assigned to RS

Fill in gap left by HBASE-11183 'Timeline Consistent region replicas - Phase 2 design'.
HBASE-11183 left off implementing 'async WAL Replication' on the hbase:meta
Table; hbase:meta Table could only do Phase 1 Region Replicas reading
the primary Regions' hfiles. Here we add 'async WAL Replication' to
hbase:meta so Replicas can be more current with the primary's changes.

Adds a 'special' ReplicationSource that reads hbase:meta WAL files and replicates
all edits to the configured in-cluster endpoint (Defaults to the
RegionReadReplicaEndpoint.class -- set hbase.region.replica.catalog.replication to
target a different endpoint implementation).

Set hbase.region.replica.replication.catalog.enabled to enable async WAL
Replication for hbase:meta region replicas. Its off by default.

The CatalogReplicationSource for async WAL Replication of hbase:meta does
NOT need to keep up WAL offset or a queue of WALs-to-replicate in the
replication queue store as is done in other ReplicationSource implementations;
the CatalogReplicationSource is for Region Replicas only. General
Replication does not replicate hbase:meta. hbase:meta Region Replicas reset
on crash of the primary replica so there is no need to 'recover'
replication that was running on the crashed server.

Because it so different in operation, the CatalogReplicationSource is bolted
on to the side of the ReplicationSourceManager. It is lazily
instantiated to match the lazy instantiation of the hbase:meta
WALProvider, created and started on the open of the first Region of an
hbase:meta table. Thereafter it stays up till the process dies, even if
all hbase:meta Regions have moved off the server, in case a hbase:meta
Region is moved back (Doing this latter simplifies the implementation)

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
  Read configuration to see if we need to wait on setting a Region read-enabled
  (if so, replicas will only flip to enable read after confirming a
  flush of the primary so they for sure are a replica of a known point)

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
 If configured, on open of hbase:meta, ask the ReplicationSourceManager
 to add a ReplicationSource (if it hasn't already).

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java
 Edit log message.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
 If configured, on close of hbase:meta, update ReplicationSourceManager
 that a source Region has closed.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java
 javadoc and make constructor private.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java
 Add logPositionAndCleanOldLogs w/ default of the old behavior so
 CatalogReplicationSource can bypass updating store with WAL position,
 etc.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
 Add creation and start of an CatalogReplicationSource.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java
 Go via ReplicationSource when calling logPostionAndCleanOldLogs so new RS can intercept.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java
 Javadoc.

hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java
 Add utility for reading configurations for hbase:meta region replicas.

hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java
 Javadoc.

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
 Use define.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java
 Specical version of ReplicationSource for Region Replicas on hbase:meta.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSourcePeer.java
 Needs a special peer too (peers are baked into replication though we don't use 'peers' here)

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALProvider.java
 Tests.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-11-20 00:04:48 -08:00
stack 7964d2ec6d HBASE-25068 Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space
Pass WALFactory to Replication instead of WALProvider. WALFactory has all
WALProviders in it, not just the user-space WALProvider. Do this so
ReplicationService has access to all WALProviders in the Server (To be
exploited by the follow-on patch in HBASE-25055)
2020-11-20 00:04:48 -08:00
niuyulin 55399a0320
HBASE-25213 Should request Compaction after bulkLoadHFiles is done (#2587)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-20 14:18:42 +08:00
Baiqiang Zhao 30ef3aa457
HBASE-25306 The log in SimpleLoadBalancer#onConfigurationChange is wrong
Closes #2679

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-19 22:20:03 +05:30
Baiqiang Zhao 8c1e4763b3
HBASE-25298 hbase.rsgroup.fallback.enable should support dynamic configuration (#2668)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-19 09:20:01 +08:00
Mate Szalay-Beko 6a529d3269 HBASE-25261 Upgrade Bootstrap to 3.4.1
HBase UI is currently using in bootstrap 3.3.7. This version is vulnerable to 4
medium CVEs (CVE-2018-14040, CVE-2018-14041, CVE-2018-14042, and CVE-2019-8331).
Details on all the bootstrap versions and vulnerabilities is
here: https://snyk.io/vuln/npm:bootstrap

Upgrading to bootstrap 4 would be nice, but potentially more work to do. We
should at least upgrade to the latest bootstrap 3, which is 3.4.1 currently.

closes #2661

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-11-17 12:44:20 +01:00
gkanade 300b0a650f
HBASE-25026 Create a metric to track full region scans RPCs
Add new metric rpcFullScanRequestCount to track number of requests that are full region scans. Can be used to notify user to check if this is truly intended.
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by Ramkrishna S Vasudevan <ramkrishna@apache.org>
2020-11-17 08:48:59 +05:30
stack c07f27e025 Revert "HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConnectionRegistry (#2652)"
This reverts commit 6210dafc47.

Applied to master when should have been applied to branch. Revert.
2020-11-16 08:59:12 -08:00
Michael Stack 6210dafc47
HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConnectionRegistry (#2652)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-11-16 08:58:05 -08:00
stack 1c85c14994 Revert "HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConnectionRegistry"
This reverts commit adbf81c010a5fbfebfcc4aae68e35dbd12da1601.

Premature. Still some comments to address.
2020-11-16 08:45:20 -08:00
stack 0aff1759a8 HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConnectionRegistry
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-11-16 08:45:20 -08:00
Hossein Zolfi f68f3dd799
HBASE-25273 fix typo in StripeStoreFileManager java doc (#2653)
Co-authored-by: Hossein Zolfi <h.zolfi@inside.sahab.ir>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-11-14 18:47:57 +01:00
Duo Zhang 09aaa68be4 HBASE-25255 Addendum wait for meta loaded instead of master initialized for system table creation 2020-11-13 15:19:09 +08:00
Duo Zhang f89faf3ac8
HBASE-25255 Master fails to initialize when creating rs group table (#2638)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-13 14:28:31 +08:00
WenFeiYi aaeeaa582e
HBASE-25253 Deprecated master carrys regions related methods and configs (#2635)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-12 22:13:24 +08:00
niuyulin 0b6d6fd773
HBASE-25276 Need to throw the original exception in HRegion#openHRegion (#2648)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-12 17:40:24 +08:00
ramkrish86 57d9cae480
HBASE-25187 Improve SizeCachedKV variants initialization (#2582)
* HBASE-25187 Improve SizeCachedKV variants initialization

* HBASE-25187 Improve SizeCachedKV variants initialization

* The BBKeyValue also can be optimized

* Change for SizeCachedKeyValue

* Addressing revew comments

* Fixing checkstyle and spot bugs comments

* Spot bug fix for hashCode

* Minor updates make the rowLen as short and some consturctor formatting

* Change two more places where there was a cast
2020-11-11 17:39:39 +05:30
Mate Szalay-Beko 6a5c928539
HBASE-25181 Add options for disabling column family encryption and choosing hash algorithm for wrapped encryption keys.
Prior to this patch hbase always used the MD5 hash algorithm to store a hash for encryption keys.
This hash is needed to verify the secret key of the subject. (e.g. making
sure that the same secrey key is used during encrypted HFile read and write).
The MD5 algorithm is considered weak, and can not be used in some
(e.g. FIPS compliant) clusters.

In this patch we:
- add a config parameter to globally enable/disable column family encryption (def enabled)
- introduce a backward compatible way of specifying the hash algorithm.
  This enable us to use newer and more secure hash algorithms like SHA-384
  or SHA-512 (which are FIPS compliant).
- add a config parameter to fail if an hfile is encountered that uses a
  different hash algorithm than the one currently configured to ease validation after
  migrating key hash algorithms (def disabled)

Closes #2539

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Esteban Gutierrez <esteban@apache.org>
2020-11-09 13:53:50 -06:00
Duo Zhang 58c974888f
HBASE-25257 Remove MirroringTableStateManager (#2634)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-09 21:36:44 +08:00
gvprathyusha6 5c7432f4a9
HBASE-24667 Rename configs that support atypical DNS set ups to put them in hbase.unsafe
Closes #2542

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-09 12:27:34 +05:30
Duo Zhang c36ee4fac0
HBASE-25254 Rewrite TestMultiLogThreshold to remove the LogDelegate in RSRpcServices (#2631)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-08 21:47:18 +08:00
Duo Zhang 671129df56
HBASE-25252 Move HMaster inner classes out (#2628)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-07 20:05:04 +08:00
Michael Stack 23e656712b HBASE-25238 Upgrading HBase from 2.2.0 to 2.3.x fails because of “Message missing required fields: state” (#2625)
Make protobuf fields add post-2.0.0 release marked 'required' instead
be 'optional' so migrations from 2.0.x to 2.1+ or 2.2+ succeeds.

Signed-off-by: Viraj Jasani vjasani@apache.org
2020-11-05 08:49:47 -08:00
WenFeiYi 0356e8efd1
HBASE-25240 gson format of RpcServer.logResponse is abnormal
Closes #2623

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-05 19:58:05 +05:30
niuyulin 0e71d6192a
HBASE-25053 WAL replay should ignore 0-length files (#2437)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-04 08:21:31 -08:00
Duo Zhang 49774c7e18
HBASE-25216 The client zk syncer should deal with meta replica count change (#2614)
Signed-off-by: Yu Li <liyu@apache.org>
2020-11-04 17:54:18 +08:00
Duo Zhang f37cd05c32
HBASE-25235 Cleanup the deprecated methods in TimeRange (#2616)
Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-11-04 11:02:28 +08:00
Andrew Purtell c98e993b23
HBASE-25212 Optionally abort requests in progress after deciding a region should close (#2574)
If hbase.regionserver.close.wait.abort is set to true, interrupt RPC
handler threads holding the region close lock.

Until requests in progress can be aborted, wait on the region close lock for
a configurable interval (specified by hbase.regionserver.close.wait.time.ms,
default 60000 (1 minute)). If we have failed to acquire the close lock after
this interval elapses, if allowed (also specified by
hbase.regionserver.close.wait.abort), abort the regionserver.

We will attempt to interrupt any running handlers every
hbase.regionserver.close.wait.interval.ms (default 10000 (10 seconds)) until
either the close lock is acquired or we reach the maximum wait time.

Define a subset of region operations as interruptible. Track threads holding
the close lock transiting those operations. Set the thread interrupt status
of tracked threads when trying to close the region. Use the thread interrupt
status where safe to break out of request processing.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-03 15:20:27 -08:00
Nick Dimiduk d790bdedde
HBASE-25167 Normalizer support for hot config reloading (#2523)
Wire up the `ConfigurationObserver` chain for
`RegionNormalizerManager`. The following configuration keys support
hot-reloading:
 * hbase.normalizer.throughput.max_bytes_per_sec
 * hbase.normalizer.split.enabled
 * hbase.normalizer.merge.enabled
 * hbase.normalizer.min.region.count
 * hbase.normalizer.merge.min_region_age.days
 * hbase.normalizer.merge.min_region_size.mb

Note that support for `hbase.normalizer.period` is not provided
here. Support would need to be implemented generally for the `Chore`
subsystem.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Aman Poonia <aman.poonia.29@gmail.com>
2020-10-30 10:41:56 -07:00
Pankaj e3beccf1fc
HBASE-24977 Meta table shouldn't be modified as read only (#2537)
Signed-off-by: stack <stack@apache.org>
2020-10-29 16:34:23 -07:00
Nick Dimiduk 12d0397015 HBASE-24419 Normalizer merge plans should consider more than 2 regions when possible
The core change here is to the loop in
`SimpleRegionNormalizer#computeMergeNormalizationPlans`. It's a nested
loop that walks the table's region chain once, looking for contiguous
sequences of regions that meet the criteria for merge. The outer loop
tracks the starting point of the next sequence, the inner loop looks
for the end of that sequence. A single sequence becomes an instance of
`MergeNormalizationPlan`.

Signed-off-by: Huaxiang Sun <huaxiangsun@apache.org>
2020-10-29 13:37:23 -07:00
GeorryHuang 35b344c337 HBASE-25090 CompactionConfiguration logs unrealistic store file sizes (#2595)
Signed-off-by: stack <stack@apache.org>
2020-10-29 13:10:35 -07:00
Peter Somogyi bb4a9d335f
HBASE-25224 Maximize sleep for checking meta and namespace regions availability (#2593)
Signed-off-by: Michael Stack <stack@apache.org>
2020-10-28 18:08:05 +01:00
Minji Kim 735689d0f7
HBASE-25223 Use try-with-resources statement (#2592)
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-10-28 09:17:31 -07:00
niuyulin 17f9aded2c
HBASE-25201 YouAreDeadException should be moved to hbase-server module (#2581)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-28 14:58:28 +08:00
niuyulin 485da758bb
HBASE-25176 MasterStoppedException should be moved to hbase-client module (#2538)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-28 14:52:27 +08:00
WenFeiYi 4bbc772ffc
HBASE-25173 Remove owner related methods in TableDescriptor/TableDescriptorBuilder (#2541)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-27 22:01:57 +08:00
Duo Zhang 164cc5a3dc
HBASE-25215 TestClientOperationTimeout.testScanTimeout is flaky (#2583)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-10-26 10:03:43 +08:00
Reid Chan e5d4e2fc81
HBASE-25189 [Metrics] Add checkAndPut and checkAndDelete latency metrics at table level (#2549)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-10-25 17:46:14 +08:00
Toshihiro Suzuki 881c92b892
HBASE-25206 Data loss can happen if a cloned table loses original split region(delete table) (#2569)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-24 22:49:14 +08:00
Sandeep Pal 5e3ffb1db6
HBASE-25193: Add support for row prefix and type in the WAL Pretty Printer
Closes #2556

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-10-23 12:30:35 +05:30
Duo Zhang 74df1e1aba
HBASE-25207 Revisit the implementation and usage of RegionStates.include (#2571)
Remove the RegionStates.include method as its name is ambiguous.
Add more comments to describe the logic on why we filter region like
this.

Signed-off-by: Toshihiro Suzuki <brfrn169@gmail.com>
2020-10-22 21:50:43 +08:00
Jan Hentschel 2cd5f914af
HBASE-25197 Remove SingletonCoprocessorService interface (#2560)
Remove the SingletonCoprocessorService interface targeted for removal in
3.0.0.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-10-21 19:04:17 +02:00
Jan Hentschel 1b58af1841
HBASE-25198 Remove deprecated RpcSchedulerFactory#create (#2561)
Remove the deprecated RpcSchedulerFactory#create(Configuration,
PriorityFunction) method from the interface and in all implementing
classes.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-21 21:40:45 +08:00
Ankit Singhal 7d00e43dce
HBASE-25166 MobFileCompactionChore is closing the master's shared cluster connection (#2514) 2020-10-19 22:22:33 -07:00
shahrs87 e6c78f4037
HBASE-25179 : Fix Assert format in HFilePerformanceEvaluation class
Closes #2551

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-10-16 12:08:10 +05:30
Nick Dimiduk 70d2f37ee3
HBASE-24628 Region normalizer now respects a rate limit (HMaster chore shutdown NPE ADDENDUM) (#2540)
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-15 22:02:53 +08:00
Duo Zhang 38ac7e26ae
HBASE-25186 TestMasterRegionOnTwoFileSystems is failing after HBASE-25065 (#2544)
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
2020-10-15 10:22:14 +08:00
XinSun 78b7244091
HBASE-25117 ReplicationSourceShipper thread can not be finished (#2521)
Signed-off-by: Wellington Chevreuil <wellington.chevreuil@gmail.com>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-14 18:08:54 +01:00
Duo Zhang 92c3bcd9fb
HBASE-25164 Make ModifyTableProcedure support changing meta replica count (#2513)
Signed-off-by: Michael Stack <stack@apache.org>
2020-10-13 09:43:56 +08:00
stack c28220522d HBASE-25168 Unify WAL name timestamp parsers
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-10-12 07:52:26 -07:00
niuyulin 8eea052359
HBASE-25093 the RSGroupBasedLoadBalancer#retainAssignment throws NPE (#2450)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-12 11:02:37 +08:00
Nick Dimiduk c8c860c906
HBASE-25156 TestMasterFailover.testSimpleMasterFailover is flaky (ADDENDUM) (#2529)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-11 21:52:11 +08:00
bsglz 5a474449c1
HBASE-24840 Avoid shows closing region task when create table (#2226) 2020-10-11 19:12:56 +08:00
ramkrish86 accd9750aa
HBASE-25065 WAL archival to be done by a separate thread (#2501)
* HBASE-25065 WAL archival can be batched/throttled and also done by a separate thread

* Fix checkstyle issues

* Address review comments

* checkstyle comments

* Addressing final review comments
Signed-off-by: Michael Stack <stack@apache.org>
2020-10-11 10:46:06 +05:30
Michael Stack 665a8767a0
HBASE-22976 [HBCK2] Add RecoveredEditsPlayer (#2504)
Make it so WALPlayer can replay recovered.edits files.

hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/WALInputFormat.java
 Allow for WAL files that do NOT have a startime in their name.
 Use the 'generic' WAL-filename parser instead of the one that
 used be local here. Implement support for 'startTime' filter.
 Previous was just not implemented.

hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java
 Checkstyle.

hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractFSWALProvider.java
 Use the new general WAL name timestamp parser.

hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WAL.java
 Utility for parsing timestamp from WAL filename.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRecoveredEdits.java
 Export attributes about the local recovered.edits test file
 so other tests can play with it.

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-10-09 08:46:05 -07:00
Nick Dimiduk 78ae1f176d HBASE-24628 Region normalizer now respects a rate limit
Implement a rate limiter for the normalizer. Implemented in terms of
MB/sec of affacted region size (the same metrics used to make
normalization decisions). Uses Guava `RateLimiter` to perform the
resource accounting. `RateLimiter` works by blocking (uninterruptible
😖) the calling thread. Thus, the whole construction of the normalizer
subsystem needed refactoring. See the provided `package-info.java` for
an overview of this new structure.

Introduces a new configuration,
`hbase.normalizer.throughput.max_bytes_per_sec`, for specifying a
limit on the throughput of actions executed by the normalizer. Note
that while this configuration value is in bytes, the minimum honored
valued `1_000_000`. Supports values configured using the
human-readable suffixes honored by `Configuration.getLongBytes`

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
Signed-off-by: Michael Stack <stack@apache.org>
2020-10-08 15:45:08 -07:00
Nick Dimiduk b82d8a5517
HBASE-25156 TestMasterFailover.testSimpleMasterFailover is flaky (#2507)
Change the test to wait for evidence that the active master has seen
that the backup master killed by the test has gone away. This is done
before proceeding to validate that the dead backup is correctly
omitted from the ClusterStatus report.

Also, minor fixup to several assertions, using `assertEquals` instead
of `assertTrue(...equals(...))` and correcting expected vs. actual
ordering of assertion arguments.

Signed-off-by: Michael Stack <stack@apache.org>
2020-10-08 14:23:38 -07:00
Mohammad Arshad bdcafa895c
HBASE-24025: Improve performance of move_servers_rsgroup by using async region move API (#1549) 2020-10-08 10:30:16 -07:00
Michael Stack 8a995ae81b
HBASE-25165 Change 'State time' in UI so sorts (#2508)
Display startcode in iso8601.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-10-08 09:10:30 -07:00
Duo Zhang 0d63318f10
HBASE-25124 Support changing region replica count without disabling table (#2497)
Signed-off-by: stack <stack@apache.org>
2020-10-08 21:35:29 +08:00
Toshihiro Suzuki aff8bbf0cb
HBASE-25160 Refactor AccessController and VisibilityController (#2506)
Signed-off-by: stack <stack@apache.org>
2020-10-08 17:04:48 +09:00
Duo Zhang e88509841f Revert "HBASE-24813 ReplicationSource should clear buffer usage on Replicatio… (#2191)"
This reverts commit 687e53b7e4.
2020-10-06 21:09:00 +08:00
Andor Molnár 14b523ec98
HBASE-25146 Add extra logging at info level to HFileCorruptionChecker in order to report progress (#2503)
Signed-off-by: Wellington Chevreuil <wellington.chevreuil@gmail.com>
Signed-off-by: Mate Szalay-Beko <szalay.beko.mate@gmail.com>
Signed-off-by: Balazs Meszaros <meszibalu@apache.org>
2020-10-06 09:38:18 +02:00
Duo Zhang 9ba90e1679
HBASE-25121 Refactor MetaTableAccessor.addRegionsToMeta and its usage places (#2476)
Signed-off-by: stack <stack@apache.org>
2020-10-05 21:29:55 +08:00
Viraj Jasani 23ce91819a
HBASE-25147 : Serialize regionNames in ReopenTableRegionsProcedure
Closes #2494

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-10-05 15:49:26 +05:30
niuyulin 3226c1795a
HBASE-25115 HFilePrettyPrinter can't seek to the row which is the first row of a hfile
Closes #2473

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-10-04 16:07:28 +05:30
Duo Zhang b0170d0e24
HBASE-25080 Should not use AssignmentManager to test whether a table is enabled or not (#2436)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-10-03 20:48:38 +08:00
Michael Stack 3b91a15183
HBASE-25091 Move LogComparator from ReplicationSource to AbstractFSWALProvider#.WALsStartTimeComparator (#2449)
Give the comparator a more descriptive name, a better location,
and make it work even when passed hbase:meta WAL files.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-10-01 10:04:58 -07:00
ramkrish86 a0b3d6276f
HBASE-25135 Convert the internal seperator while emitting the memstore read metrics to # (#2486)
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2020-10-01 18:07:28 +05:30
Wellington Ramos Chevreuil 9fc29c4cbf
HBASE-24877 addendum: additional checks to avoid one extra possible race control in the initialize loop (#2400)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2020-09-29 10:00:57 +01:00
Duo Zhang b268b1f621
HBASE-25103 Remove ZNodePaths.metaReplicaZNodes (#2464)
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-09-29 15:48:57 +08:00
bsglz 5d926627ae
HBASE-24967 The table.jsp cost long time to load if the table include… (#2326)
* HBASE-24967 The table.jsp cost long time to load if the table include closed regions

* fix it  by another way

* fix review issue

* fix checkstyle warnings

* fix checkstyle warning
2020-09-29 15:36:03 +08:00
XinSun 68b56beab7
HBASE-25100 conf and conn are assigned twice in HBaseReplicationEndpoint and HBaseInterClusterReplicationEndpoint (#2463)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-29 08:27:37 +08:00
Viraj Jasani 80ffac29c1
HBASE-25070 : With new generic API getLogEntries, cleaning up unused RPC APIs
Closes #2426

Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-28 12:54:35 +05:30
Toshihiro Suzuki 86557edf2c
HBASE-25096 WAL size in RegionServer UI is wrong (#2456)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-28 14:47:18 +09:00
Mohammad Arshad b57bef5b7d
HBASE-25077: hbck.jsp page loading fails, logs NPE in master log. (#2433)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
2020-09-27 14:05:38 +08:00
XinSun 380585ef20
HBASE-25098 ReplicationStatisticsChore runs in wrong time unit (#2460)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-26 19:49:02 +08:00
meiyi 4f8aa75dca
HBASE-25097 Wrong RIT page number in Master UI (#2458)
Signed-off-by: Viraj Jasani<virajjasani@apache.org>
2020-09-25 18:36:41 +08:00
Wellington Ramos Chevreuil 687e53b7e4
HBASE-24813 ReplicationSource should clear buffer usage on Replicatio… (#2191)
Signed-off-by: Josh Elser <elserj@apache.org>
2020-09-24 10:44:36 +01:00
Guanghao Zhang 8828643bb2
HBASE-25086 Refactor Replication: move the default ReplicationSinkService implementation out (#2444)
Signed-off-by: meiyi <myimeiyi@gmail.com>
2020-09-24 17:25:34 +08:00
Bharath Vissapragada 56c7505f8f
HBASE-25082: Per table WAL metrics: appendCount and appendSize (#2440)
Signed-off-by: Geoffrey Jacoby <gjacoby@apache.org>
Signed-off-by: Ankit Jain <jain.ankit@salesforce.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-09-23 21:06:57 -07:00
Andrew Purtell 8bfa2cb2ee
HBASE-25079 Upgrade Bootstrap to 3.3.7 (#2442)
Signed-off-by: Viraj Jasani<virajjasani@apache.org>
2020-09-23 14:56:50 -07:00
stack e7797208d6 Revert "HBASE-25068 Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space"
This reverts commit 17ebf917ba.
2020-09-22 20:48:31 -07:00
Guanghao Zhang 7e910a573f
HBASE-25074 Refactor ReplicationSinkManager: reduce code and make it easy to understand (#2430)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-09-23 08:30:43 +08:00
stack 17ebf917ba HBASE-25068 Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space
Pass WALFactory to Replication instead of WALProvider. WALFactory has all
WALProviders in it, not just the user-space WALProvider. Do this so
ReplicationService has access to all WALProviders in the Server (To be
exploited by the follow-on patch in HBASE-25055)
2020-09-22 10:18:26 -07:00
Michael Stack 70a947dc6b
HBASE-25067 Edit of log messages around async WAL Replication; checkstyle fixes; and a bugfix (#2435)
Editing logging around region replicas: shortening and adding context.
Checkstyle fixes in edited files while I was in there.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-09-22 10:10:31 -07:00
niuyulin 8a6299bb38
HBASE-25076 fix typo in MasterRegion java doc (#2432)
Co-authored-by: niuyulin <niuyulin@xiaomi.com>

Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-09-21 19:00:49 +02:00
niuyulin 36e8b0beec
HBASE-25075 Fix typo in ReplicationProtobufUtil (#2431)
Co-authored-by: niuyulin <niuyulin@xiaomi.com>

Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-09-21 18:44:47 +02:00
stack 3e13b3f059
HBASE-24896 'Stuck' in static initialization creating RegionInfo instance
Closes #2422

Untangle RegionInfo, RegionInfoBuilder, and MutableRegionInfo static
initializations some. Move MutableRegionInfo from inner-class of
RegionInfoBuilder to be (package private) standalone. Undo static
initializing references from RI to RIB.

Co-authored-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-09-20 14:18:32 +05:30
Mohammad Arshad fd7260b506
HBASE-25069: Display region name instead of encoded region name in HBCK report page. (#2428)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-20 09:00:28 +08:00
Joseph295 58618e35ad
HBASE-24991 Replace MovedRegionsCleaner with guava cache (#2357)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-19 14:53:13 +08:00
Hyeran Lee 6371914adb
HBASE-25057: Fix typo memeber (#2414)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-09-19 14:36:06 +08:00
GeorryHuang 16831ec515
HBASE-24857:Fix several problems when starting webUI (#2245)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-09-19 14:34:30 +08:00
Joseph295 9fc586a7eb
HBASE-25047 WAL split edits number is negative in RegionServerUI (#2408)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-17 09:38:28 +08:00
Viraj Jasani 35d59dc570
HBASE-24528 : BalancerDecision queue implementation in HMaster with Admin API
* Admin API getLogEntries() for ring buffer use-cases: so far, provides balancerDecision and slowLogResponse
* Refactor RPC call for similar use-cases
* Single RPC API getLogEntries() for both Master.proto and Admin.proto

Closes #2261

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-09-16 20:56:54 +05:30
Duo Zhang 1bb19e0cdd
HBASE-25037 Lots of thread pool are changed to non daemon after HBASE-24750 which causes trouble when shutting down (#2407)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-09-16 21:11:47 +08:00
XinSun a918bf0d7b
HBASE-25012 HBASE-24359 causes replication missed log of some RemoteException (#2384)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-16 18:01:55 +08:00
bsglz c5ca191921
HBASE-24875 Remove the force param for unassign since it dose not take effect any more (#2254)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-09-15 22:05:03 -05:00
Mohammad Arshad 6d985dd7c6
HBASE-25009: Hbck chore logs wrong message when loading regions from RS (#2379)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-16 09:08:04 +08:00
Duo Zhang 724a0e5500
HBASE-25022 Remove 'hbase.testing.nocluster' config (#2394)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-14 20:15:43 +08:00
Wellington Ramos Chevreuil 2042523f43
HBASE-24877 Add option to avoid aborting RS process upon uncaught exc… (#2255)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2020-09-14 11:13:14 +01:00
Clara Xiong fe776306d0
HBASE-25006 Make the cost functions optional for StochastoicBalancer
Closes #2386

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-09-13 14:16:02 +05:30
ankitjain64 9c5dbb29c0
HBASE-24764: Add support of adding default peer configs via hbase-site.xml for all replication peers. (#2284)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-09-10 10:08:44 -07:00
WenFeiYi bbfbe33520
HBASE-24958 CompactingMemStore.timeOfOldestEdit error update (#2321) 2020-09-10 09:46:04 -07:00
Viraj Jasani ce59a2ba30
HBASE-25004 : Log RegionTooBusyException details (#2371)
Signed-off-by: mnpoonia <apoonia@salesforce.com>
Signed-off-by: stack <stack@apache.org>
2020-09-10 21:22:37 +05:30
Duo Zhang 23713f4693
HBASE-25005 Refactor CatalogJanitor (#2373)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-10 11:59:15 +08:00
Sandeep Pal 84a34be761
HBASE-24974: Provide a flexibility to print only row key and filter for multiple tables in the WALPrettyPrinter (#2345)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-09-09 10:16:45 -07:00
Duo Zhang 0511089066
HBASE-25000 Move delete region info related methods to RegionStateStore (#2366)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-09 23:11:28 +08:00
niuyulin 2e96a5b2d3
HBASE-24993 Remove OfflineMetaRebuildTestCore (#2359)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-09-09 09:47:15 +08:00
Javier Akira Luca de Tena 2e638de6a3
HBASE-24994 Add hedgedReadOpsInCurThread metric (#2365)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-09-09 09:45:46 +08:00
Mohammad Arshad 2250b51fe7
HBASE-24995: MetaFixer fails to fix overlaps when multiple tables have overlaps (#2361)
Signed-off-by: stack <stack@apache.org>
2020-09-08 12:00:44 -07:00
Viraj Jasani 0d95a8f91b
HBASE-24979 : Client operation timeout test for batch requests
Closes #2347

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-08 17:51:38 +05:30
Toshihiro Suzuki d48c732851
HBASE-24602 Add Increment and Append support to CheckAndMutate (#2228) 2020-09-08 15:05:47 +09:00
Duo Zhang be984cc8d4
HBASE-24986 Move ReplicationBarrier related methods to a separated class (#2354)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-07 20:35:27 +08:00
XinSun c2721c8235
HBASE-24759 Refuse to update configuration of default group (#2126)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-09-04 08:19:39 +08:00
Mohammad Arshad a352706700
HBASE-24940: runCatalogJanitor() API should return -1 to indicate already running status
Closes #2331

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-09-03 15:52:18 +05:30
Qi Yu 4dc08144f3
HBASE-24973 Remove read point parameter in method StoreFlush#performFlush and StoreFlush#createScanner (#2337)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-09-02 17:43:26 +08:00
Duo Zhang 85a174f5b3 HBASE-24964 addendum fix TestMetaTableMetrics (#2340)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-09-02 10:28:53 +08:00
Andrew Purtell 74807b951b
HBASE-24971 Upgrade JQuery to 3.5.1 (#2336)
- Update embedded minimized jquery to 3.5.1
- Update embedded minimized jquery.tablesorter to 2.31.3

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-09-01 10:39:12 -07:00
Zheng Wang ee450ae943
HBASE-24569 Get hostAndWeights in addition using localhost if it is null in local mode
Closes #1909

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-09-01 18:16:10 +05:30
XinSun 25fcc40f6a
HBASE-24913 Refactor TestJMXConnectorServer (#2286)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-08-31 10:23:59 +08:00
Mohammad Arshad bb64070bd3
HBASE-24916: Region hole contains wrong regions pair when hole is cre… (#2304)
Signed-off-by: stack <stack@apache.org>
2020-08-30 10:04:40 -07:00
Viraj Jasani 1c568ec95d
HBASE-24968 : Move META_COMPARATOR to subclass MetaCellComparator
* Break subclass referencing of MetaCellComparator from superclass CellComparatorImpl
  static initializer by moving META_COMPARATOR to subclass MetaCellComparator

Closes #2329

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-08-30 22:03:57 +05:30
Zheng Wang 2eb2c96960
HBASE-24894 [Flakey Test] TestStochasticLoadBalancer.testMoveCostMultiplier
Closes #2270

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-30 20:12:16 +05:30
Duo Zhang 2c4204ad75
HBASE-24964 Remove MetaTableAccessor.tableExists (#2330)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-30 17:31:26 +08:00
Duo Zhang ef5da4a61d
HBASE-24949 Optimize FSTableDescriptors.get to not always go to fs when cache miss (#2317)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-08-29 22:26:36 +08:00
XinSun 7909e29de5
HBASE-24760 Add a config hbase.rsgroup.fallback.enable for RSGroup fallback feature (#2149)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-08-28 16:52:45 +08:00
bsglz 047e0618d2
HBASE-24898 Addendum to deal with Daylight Saving Time (#2310)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-08-27 11:17:23 +08:00
WenFeiYi f9ce1df3f2
HBASE-24892 config 'hbase.hregion.memstore.mslab.indexchunksize' not be used (#2265)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-08-27 08:24:40 +08:00
Bharath Vissapragada 01cf60067c
HBASE-24765: Dynamic master discovery (#2130)
This patch adds the ability to discover newly added masters
dynamically on the master registry side. The trigger for the
re-fetch is either periodic (5 mins) or any registry RPC failure.
Master server information is cached in masters to avoid repeated
ZK lookups.

Updates the client side connection metrics to maintain a counter
per RPC type so that clients have visibility into counts grouped
by RPC method name.

I didn't add the method to ZK registry interface since there
is a design discussion going on in splittable meta doc. We can
add it later if needed.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-08-25 15:09:03 -07:00
Duo Zhang 227084c41f
HBASE-24945 Remove MetaTableAccessor.getRegionCount (#2303)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-25 21:15:22 +08:00
Duo Zhang 1220a8775c
HBASE-24942 MergeTableRegionsProcedure should not call clean merge region (#2301)
Signed-off-by: stack <stack@apache.org>
2020-08-25 12:19:19 +08:00
Duo Zhang 57e49b3959
HBASE-23834 HBase fails to run on Hadoop 3.3.0/3.2.2/3.1.4 due to jetty version mismatch (#2222)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-08-25 12:05:52 +08:00
Michael Stack 6ad73b9668 HBASE-24885 STUCK RIT by hbck2 assigns (#2283)
Adds region state check on hbck2 assigns/unassigns. Returns pid of -1
if in inappropriate state with logging explaination which suggests
passing override if operator wants to assign/unassign anyways. Here
is an example of what happens now if hbck2 tries an unassign and
Region already unassigned:

  2020-08-19 11:22:06,926 INFO  [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=50086] assignment.AssignmentManager(820): Failed {ENCODED => d1112e553991e938b6852f87774c91ee, NAME => 'TestHbck,zzzzz,1597861310769.d1112e553991e938b6852f87774c91ee.', STARTKEY => 'zzzzz', ENDKEY => ''} unassign, override=false; set override to by-pass state checks.
  org.apache.hadoop.hbase.client.DoNotRetryRegionException: Unexpected state for state=CLOSED, location=null, table=TestHbck, region=d1112e553991e938b6852f87774c91ee
          at org.apache.hadoop.hbase.master.assignment.AssignmentManager.preTransitCheck(AssignmentManager.java:583)
          at org.apache.hadoop.hbase.master.assignment.AssignmentManager.createOneUnassignProcedure(AssignmentManager.java:812)
          at org.apache.hadoop.hbase.master.MasterRpcServices.unassigns(MasterRpcServices.java:2616)
          at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$HbckService$2.callBlockingMethod(MasterProtos.java)
          at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:397)
          at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
          at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
          at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)

Previous it would just create the unassign anyways. Now must pass override
to queue the procedure regardless. Safer.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
 javadoc on assigns/unassigns. Minor refactor in assigns/unassigns to cater to
 case where procedure may come back null (if override not set and fails state checks).

hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
 checkstyle cleanups.
 Clarifying javadoc on how there is no state checking when bulk assigns creating/enabling
 tables.

 createOneAssignProcedure and createOneUnassignProcedure now handle exceptions which now
 can be thrown if no override and region state is not appropriate.

 Aggregation of createAssignProcedure and createUnassignProcedure instances adding in
 region state check invoked if override is NOT set.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateNode.java
 Change to setProcedure so it returns passed proc as result instead of void

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-08-24 09:22:16 -07:00
XinSun e0c9f911ad
HBASE-24871 Replication may loss data when refresh recovered replication sources (#2249)
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-08-24 21:43:15 +08:00
niuyulin 2874f00a2f
HBASE-24928 balanceRSGroup should skip generating balance plan for disabled table and splitParent region (#2292)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-08-24 11:52:21 +08:00
Yechao Chen 9f62a82334
HBASE-24811 Use class access static field or method (#2189)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2020-08-23 11:24:44 +08:00
Zheng Wang 0fe97947de
HBASE-24898 Use EnvironmentEdge.currentTime() instead of System.currentTimeMillis() in CurrentHourProvider
Closes #2272

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-22 15:30:09 +05:30
1996fanrui 5b515de792
HBASE-24915 Improve BlockCache read performance by specifying BlockType (#2288)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-08-21 09:31:39 -07:00
Duo Zhang 5c298790f2
HBASE-24926 Should call setFailure in MergeTableRegionsProcedure when isMergeable returns false (#2290)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-21 22:01:33 +08:00
Guanghao Zhang 54fe81eb56
HBASE-24912 Enlarge MemstoreFlusherChore/CompactionChecker period for unit test (#2285)
Signed-off-by: stack <stack@apache.org>
2020-08-21 12:41:20 +08:00
XinSun 65d28da7c2
HBASE-24914 Remove duplicate code appearing continuously in method ReplicationPeerManager.updatePeerConfig (#2287)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-08-20 14:44:56 -07:00
Toshihiro Suzuki 1164531d5a
HBASE-24884 BulkLoadHFilesTool/LoadIncrementalHFiles should accept -D options from command line parameters (#2260)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-08-19 13:36:17 +09:00
Nick Dimiduk ecc0c67077 HBASE-24627 Normalize one table at a time
Introduce an additional method to our Admin interface that allow an
operator to selectivly run the normalizer. The IPC protocol supports
general table name select via compound filter.

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-18 09:46:29 -07:00
niuyulin ea26463a33
HBASE-24880 Remove ReplicationPeerConfigUpgrader
Closes #2266

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-17 19:20:53 +05:30
Nick Dimiduk 2e8120a63f HBASE-24583 Normalizer can't actually merge empty regions...
when neighbor is larger than average size

* add `testMergeEmptyRegions` to explicitly cover different
  interleaving of 0-sized regions.
* fix bug where merging a 0-size region is skipped due to large
  neighbor.
* remove unused `splitPoint` from `SplitNormalizationPlan`.
* generate `toString`, `hashCode`, and `equals` methods from Apache
  Commons Lang3 template on `SplitNormalizationPlan` and
  `MergeNormalizationPlan`.
* simplify test to use equality matching over `*NormalizationPlan`
  instances as plain pojos.
* test make use of this handy `TableNameTestRule`.
* fix line-length issues in `TestSimpleRegionNormalizer`

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
Signed-off-by: Aman Poonia <aman.poonia.29@gmail.com>
2020-08-12 10:04:45 -07:00
Duo Zhang a535c7b1af
HBASE-24856 Fix error prone error in FlushTableSubprocedure (#2242)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-08-12 09:51:12 +08:00
Viraj Jasani ea130249ae
HBASE-24750 : Adding default UncaughtExceptionHandler for Thread factories (ADDENDUM)
Closes #2231
2020-08-11 17:18:47 +05:30
Bharath Vissapragada 1d65ad464f
HBASE-24846: Address compaction races in TestFIFOCompactionPolicy (#2203)
Looped through the test 100 times and it passes. Without the patch it fails
every ~10 runs or so.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2020-08-10 16:53:14 -07:00
Bharath Vissapragada 8ad59775c9
Revert "HBASE-21905 (addendum): Address compaction races in TestFIFOCompactionPolicy (#2203)"
This reverts commit 8003a15835.

Reverting to apply again as HBASE-24846.
2020-08-10 16:52:36 -07:00
Nick Dimiduk c1c2e160ec HBASE-24835 Normalizer should log a successful run at INFO level
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-10 09:57:48 -07:00
chenglei 7b099eaa75
HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length(addendum) (#2055)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-08-10 09:57:26 +08:00
Duo Zhang 726756a8b9
HBASE-23157 WAL unflushed seqId tracking may wrong when Durability.ASYNC_WAL is used (#762)
Signed-off-by: stack <stack@apache.org>
2020-08-10 09:11:31 +08:00
stack 11ac0a1ba6 HBASE-24830 Some tests involving RS crash fail with NullPointerException after HBASE-24632 in branch-2 2020-08-09 11:22:19 -07:00
Bo Cui d492aea3f8
HBASE-21721 FSHLog : reduce write#syncs() times (#2217)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Anoop <anoopsamjohn@apache.org>
2020-08-08 09:39:48 +05:30
Josh Elser 124af6392c HBASE-24779 Report on the WAL edit buffer usage/limit for replication
Closes #2193

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-08-07 12:59:17 -04:00
Viraj Jasani 0b604d921a
HBASE-24750 : All ExecutorService should use guava ThreadFactoryBuilder
Closes #2196

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Ted Yu <tyu@apache.org>
2020-08-07 20:24:36 +05:30
bsglz 485e0d2fa4
HBASE-24694 Support flush a single column family of table (#2179)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-08-07 11:28:53 +01:00
niuyulin f710d2d654
HBASE-24659 Calculate FIXED_OVERHEAD automatically (#2018)
Co-authored-by: niuyulin <niuyulin@xiaomi.com>

SIgned-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-08-06 14:50:32 -07:00
Michael Stack 0ae125ac6f HBASE-24817 Allow configuring WALEntry filters on ReplicationSource (#2198)
Allow specifying base WALEntry filter on construction of
ReplicationSource. Add means of being able to filter WALs by name.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 Add constructor that allows passing a predicate for filtering *in* WALs
 and a list of filters for filtering *out* WALEntries. The latter was
 hardcoded to filter out system-table WALEntries. The former did not
 exist but we'll need it if Replication takes in more than just the
 default Provider.

Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-06 12:35:42 -07:00
Bharath Vissapragada 8003a15835
HBASE-21905 (addendum): Address compaction races in TestFIFOCompactionPolicy (#2203)
Looped through the test 100 times and it passes. Without the patch it fails
every ~10 runs or so.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2020-08-06 09:33:06 -07:00
sguggilam f8b887b327
HBASE-24823 Port HBASE-22762 Print the delta between phases in the split/merge/compact/flush transaction journals
Closes #2204

Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-06 18:50:12 +05:30
Zheng Wang c39cad220d
HBASE-24821 simplify the logic of getRegionInfo in TestFlushFromClient to reduce redundancy code
Closes #2200

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-05 19:59:50 +05:30
Duo Zhang d2f5a5f27b
HBAE-24507 Remove HTableDescriptor and HColumnDescriptor (#2186)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: tedyu <yuzhihong@gmail.com>
2020-08-04 10:31:42 +08:00
Michael Stack 148c185486 HBASE-24808 skip empty log cleaner delegate class names (WAS => cleaner.CleanerChore: Can NOT create CleanerDelegate= ClassNotFoundException) (#2181)
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-08-03 13:16:27 -07:00
Michael Stack 21a0b8ea11 HBASE-24767 Change default to false for HBASE-15519 per-user metrics Set hbase.regionserver.user.metrics.enabled default to false; i.e. off. (#2132)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-08-03 11:37:54 -07:00
Viraj Jasani 9ec484a031
HBASE-24795 : TestRegionMover2#testWithSplitRegions fix (ADDENDUM) 2020-08-03 20:26:46 +05:30
Viraj Jasani ad7caf754a
HBASE-24795 : RegionMover to deal with unknown region while (un)loading
* RegionMover to ignore move failures for split/merged regions with ack mode
* Refactor MoveWithAck and MoveWithoutAck as high level classes
* UT for RegionMover gracefully handling split/merged regions while loading regions and throwing failure while loading offline regions

Closes #2172

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Ted Yu <tyu@apache.org>
2020-08-03 17:25:52 +05:30
gkanade 9a1bad84bf
HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751 (#2125)
* null check for writer if not initialized yet during syncrunner run

* Revert "null check for writer if not initialized yet during syncrunner run"

This reverts commit 72932ad0df.

* Writer check while trying to attain safe point
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
2020-08-03 10:04:55 +05:30
Zheng Wang 492cf108a9
HBASE-24704 Make Table Schema easier to view with multiple families
Closes #2042

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-02 23:26:04 +05:30
Sean Busbey 86ebbdd8a2
HBASE-24805 HBaseTestingUtility.getConnection should be threadsafe
* refactor how we use connection and async connection to rely on their access methods
* refactor initialization and cleanup of the shared connection
* incompatibly change HCTU's Configuration member variable to be final so it can be safely accessed from multiple threads.

Closes #2180

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-01 23:19:07 -05:00
Toshihiro Suzuki e22a2d2700
HBASE-24680 Refactor the checkAndMutate code on the server side (#2094)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2020-08-01 21:02:17 +09:00
Anoop Sam John a3f623eea7
HBASE-24695 FSHLog - close the current WAL file in a background thread. (#2168)
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-08-01 17:27:20 +05:30
Sean Busbey 840a55761b
HBASE-24794 hbase.rowlock.wait.duration should not be <= 0 (#2174)
if hbase.rowlock.wait.duration is <=0 then log a message and treat it as a value of 1ms.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-30 12:26:12 -05:00
ddupg 1b9269de4d
HBASE-24797 Move log code out of loop
Closes #2175

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-30 15:15:52 +05:30
Bharath Vissapragada f07f30ae24
HBASE-20226: Parallelize region manifest deletes (#2159)
We observed this delete call to be a bottleneck for table with lots of
regions. Patch attempts to parallelize them.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-07-29 10:59:23 -07:00
Sandeep Pal d65fb87a2e
HBASE-24788: Fix the connection leaks on getting hbase admin from unclosed connection (#2162)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-29 09:09:07 -07:00
WenFeiYi 5f27a009ab
HBASE-24752 NPE/500 accessing webui on master startup
Closes #2148

Signed-off-by: Ted Yu <yuzhihong@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-29 20:19:29 +05:30
Michael Stack 652f1e664b HBASE-24766 Document Remote Procedure Execution (#2131)
Add outline of how the mechanism works generally.

Signed-off-by: Anoop Sam John anoopsamjohn@apache.org
Signed-off-by: Viraj Jasani vjasani@apache.org
Signed-off-by: Jan Hentschel jan.hentschel@ultratendency.com
2020-07-29 07:44:51 -07:00
Michael Stack 345b77a4ca
HBASE-24632 Enable procedure-based log splitting as default in hbase3 Add deprecation of 'classic' zk-based WAL splitter. (#2156)
Also fix three bugs:

 * We were trying to delete non-empty directory; weren't doing
 accounting for meta WALs where meta had moved off the server
 (successfully)
 * We were deleting split WALs rather than archiving them.
 * We were not handling corrupt files.

Deprecations and removal of tests of old system.

Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2020-07-29 07:37:28 -07:00
YeChao Chen 5480f350c2
HBASE-24790 Remove unused counter from SplitLogCounters (#2164)
Signed-off-by: Anoop <anoopsamjohn@apache.org>
Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-29 17:22:17 +05:30
Duo Zhang 7e6e7a7051
HBASE-24770 Reimplement the Constraints API and revisit the IA annotations on related classes (#2140)
Signed-off-by: stack <stack@apache.org>
2020-07-28 09:24:55 +08:00
Wellington Ramos Chevreuil 8c0d7fa5b8
HBASE-24758 Avoid flooding replication source RSes logs when no sinks… (#2118)
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-27 10:08:13 +01:00
WenFeiYi 975cdf7b88
HBASE-24665 all wal of RegionGroupingProvider together roll (#2021)
Co-authored-by: wen_yi <liu.wenwen@immomo.com>
Signed-off-by: Anoop <anoopsamjohn@apache.org>
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-24 22:54:25 +05:30
Viraj Jasani 09e7ccd42d
HBASE-24757 : ReplicationSink should limit row count in batch mutation based on hbase.rpc.rows.warning.threshold
Closes #2127

Signed-off-by: stack <stack@apache.org>
2020-07-24 13:16:02 +05:30
Duo Zhang 0b85729da4
HBASE-24762 Purge protobuf java 2.5.0 dependency (#2128)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-07-24 11:48:35 +08:00
Guanghao Zhang 32c7012ad7
HBASE-24743 Reject to add a peer which replicate to itself earlier (#2122)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-07-24 08:08:20 +08:00
surbhi 8191fbdd7d HBASE-22146 Removing a namespace-level space quota does not remove policies against contained tables
Closes #1935

Signed-off-by: Josh Elser <elserj@apache.org>
2020-07-21 20:12:46 -04:00
Viraj Jasani f35c5eaadd
HBASE-24718 : Generic NamedQueue framework for multiple use-cases (Refactor SlowLog responses) (#2109)
Closes #2052

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-07-21 20:24:13 +05:30
Viraj Jasani 32796ad24e
Revert "HBASE-24743 Reject to add a peer which replicate to itself earlier (#2071)"
This reverts commit 6cf013ddc4.

TestReplicationAdmin and TestReplicationShell are broken on branch-2 and master respectively
2020-07-21 20:17:27 +05:30
Mingliang Liu 3d270ba6f9
HBASE-24696 Include JVM information on Web UI under "Software Attributes"
Closes #2087

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-07-21 16:53:58 +05:30
Viraj Jasani f2c087caeb
Revert "HBASE-24718 : Generic NamedQueue framework for multiple use-cases (Refactor SlowLog responses)"
This reverts commit 16f306b4a1.

TestAdminShell and TestThriftHBaseServiceHandler are failing consistently
2020-07-21 13:50:27 +05:30
sguggilam 430602aed8
HBASE-24740 Enable journal logging for HBase snapshot operation (#2104)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-07-20 18:11:41 -07:00
Guanghao Zhang 6cf013ddc4
HBASE-24743 Reject to add a peer which replicate to itself earlier (#2071)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-07-21 08:37:56 +08:00
Sanjeet Nishad a857c6d90c
HBASE-24714 Error message is displayed in UI of table's compaction state if any region of that table is not open
Closes #2047

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-20 21:40:23 +05:30
Zheng Wang 9b32943697
HBASE-24709 Support MoveCostFunction use a lower multiplier in offpeak hours
Closes #2099

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-07-19 20:09:12 +05:30
Michael Stack 9b02a26a1d HBASE-24698 Turn OFF Canary WebUI as default (#2037)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-07-17 15:52:52 -07:00
Wellington Ramos Chevreuil bd42c75cac
HBASE-24747 Log an ERROR if HBaseSaslRpcServer initialisation fails with an uncaught exception (#2079)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2020-07-17 18:50:03 +01:00
Mohammad Arshad 1781185b7b
HBASE-24675: On Master restart all servers are assigned to default rsgroup
Closes #2053

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-17 22:38:19 +05:30
Zheng Wang 6ae2fdd6b2
HBASE-24746 The sort icons overlap the col name in master UI
Closes #2078

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-17 21:47:17 +05:30
Viraj Jasani 16f306b4a1
HBASE-24718 : Generic NamedQueue framework for multiple use-cases (Refactor SlowLog responses)
Closes #2052

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-07-17 19:16:22 +05:30
Lars a4e5ccc798 HBASE-24742 Improve performance of SKIP vs SEEK logic. 2020-07-16 13:17:05 -07:00
huaxiangsun 2505c7760d
HBASE-24581 Skip compaction request/check for replica regions at the early stage. (#1986)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-07-14 15:49:20 -07:00
huaxiangsun 1360bee7f9
HBASE-24705 MetaFixer#fixHoles() does not include the case for read replicas (i.e, replica regions are not created) (#2062)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-14 15:48:19 -07:00
BukrosSzabolcs 3e709c6f53
HBASE-24720: Meta replicas not cleaned when disabled (#2057)
- make sure to always clean up excess meta replicas not just when their
number get decreased
- make sure NotServingRegionException is handled properly even when
wrapped
- add test

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-07-14 18:24:55 +02:00
Mohammad Arshad 70cfe2525e
HBASE-24721: rename_rsgroup overwriting the existing rsgroup
Closes #2059

Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-14 21:30:23 +05:30
bsglz 3bd54217a0
HBASE-24664 Some changing of split region by overall region size rath… (#2011)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-07-13 10:00:48 +01:00
Geoffrey Jacoby af1cc2fc44
HBASE-23744 - FastPathBalancedQueueRpcExecutor should enforce queue length of 0
Closes #1094

Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-13 12:15:10 +05:30
bsglz 724f0478ed
HBASE-24404 Support flush a single column family of region (#2032)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-07-10 18:13:50 +01:00
Zheng Wang b2321b3583
HBASE-24586 Add table level locality in table.jsp
Closes #1926

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-10 21:59:59 +05:30
Viraj Jasani f0bc33122e
HBASE-24712 : Remove flakes from TestMasterNoCluster#testStopDuringStart (#2045)
Forward-port from 91da188d78
2020-07-10 20:41:03 +05:30
Joseph295 67dfbe0406
HBASE-24663 Add procedure process time statistics UI (#2035)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-07-10 15:03:52 +08:00
niuyulin 5de1dff8f6
HBASE-24653 Show snapshot owner on Master WebUI (#2016)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-07-10 15:03:05 +08:00
XinSun 1d0e285a6f
HBASE-24489 Rewrite TestClusterRestartFailover.test since namespace table is gone on on master (#2025)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-07-10 10:03:57 +08:00
Wellington Chevreuil 7b92510bb4 Revert "HBASE-21596 Delete for a specific cell version can bring back version… (#2009)"
This reverts commit 1db89773e6.
2020-07-09 15:55:57 +01:00
Wellington Ramos Chevreuil 1db89773e6
HBASE-21596 Delete for a specific cell version can bring back version… (#2009)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-08 15:17:14 +01:00
huaxiangsun 90f4ff7d7c
HBASE-24688 AssignRegionHandler uses EventType.M_RS_CLOSE_META instead of EventType.M_RS_OPEN_META for meta region (#2029)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-07 09:58:25 -07:00
Toshihiro Suzuki 34e807a8b5
HBASE-24650 Change the return types of the new CheckAndMutate methods introduced in HBASE-8458 (#1991)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-07-07 12:49:20 +09:00
WenFeiYi 287f29818f
HBASE-24546 CloneSnapshotProcedure unlimited retry (#1900)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-07-06 13:45:19 +08:00
WenFeiYi 44691ce23c
HBASE-24578 [WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count (#1923)
Signed-off-by: Reid Chan <reidchan@apache.org>
2020-07-06 11:24:09 +08:00
chenglei a7a0e1a596
HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length. (#1970)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-07-04 21:00:35 +08:00
Baiqiang Zhao afe2eac7b6
HBASE-24560 Add a new option of designatedfile in RegionMover
Closes #1901

Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-07-01 20:26:51 +05:30
xincunSong 9ad16aa376
HBASE-24624 Optimize table.jsp code (#1963)
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
2020-07-01 08:49:51 +08:00
stack 2c43f6cde7 HBASE-24661 TestHeapSize.testSizes failure
Revert "HBASE-24648 Remove the legacy 'forceSplit' related code at region server side (#1990)"

This reverts commit bfec964aef.
2020-06-30 15:57:49 -07:00
pengmq1 be57e40f36
HBASE-24289 Heterogeneous Storage for Date Tiered Compaction (#1730)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-06-30 15:10:04 +08:00
stack bfec964aef HBASE-24648 Remove the legacy 'forceSplit' related code at region server side (#1990)
Addendum to fix TestHeapSize
2020-06-29 14:20:39 -07:00
Duo Zhang 37b863bd0b
HBASE-24648 Remove the legacy 'forceSplit' related code at region server side (#1990)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-06-29 22:50:42 +08:00
Duo Zhang 46bf8944fd
HBASE-24646 Set the log level for ScheduledChore to INFO in HBTU (#1987)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-06-28 23:26:05 +08:00
Viraj Jasani 47742b2083
HBASE-24588 : createTableBegsSplit should consider tuning mergeEnabled for table (ADDENDUM) (#1988) 2020-06-27 19:09:19 +05:30
Duo Zhang af49b87367 HBASE-24635 Addendum make test more robust 2020-06-27 11:29:43 +08:00
Duo Zhang 16a25b74db
HBASE-24635 Split TestMetaWithReplicas (#1980)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-06-27 10:36:07 +08:00
Viraj Jasani 51ad588565
HBASE-24588 : normalization log format (ADDENDUM) 2020-06-26 23:09:37 +05:30
Viraj Jasani 71af97cd05
HBASE-24588 : Submit task for NormalizationPlan (#1933)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-06-26 22:08:52 +05:30
bsglz c0461207ee
HBASE-24382 Flush partial stores of region filtered by seqId when arc… (#1737)
* HBASE-24382 Flush partial stores of region filtered by seqId when archive wal due to too many wals

* fix checkstyle and javadoc issue

* fix javadoc issues

* move the geting of stores to HRegion, since it should not be part of FlushPolicy, and comment fix

* fix checkstyle issue

* add some comment

* remove the forceFlushAllStores since we can use families to determine how to select stores to flush

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@duboce.net>
2020-06-25 22:03:18 -07:00
Bharath Vissapragada 84e246f9b1
HBASE-24603: Make Zookeeper sync() call synchronous (#1945)
Writing a test for this is tricky. There is enough coverage for
functional tests. Only concern is performance, but there is enough
logging for it to detect timed out/badly performing sync calls.

Additionally, this patch decouples the ZK event processing into it's
own thread rather than doing it in the EventThread's context. That
avoids deadlocks and stalls of the event thread.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-06-25 13:27:38 -07:00
Michael Stack e54e3afc5c HBASE-24616 Remove BoundedRecoveredHFilesOutputSink dependency on a TableDescriptor (#1955)
Purge query Master for table descriptors; make do w/ generic options.

Logging cleanup.

hbase-server/src/main/java/org/apache/hadoop/hbase/wal/BoundedRecoveredHFilesOutputSink.java
 Undo fetching Table Descriptor. Not reliably available at recovery time.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-06-25 12:10:17 -07:00
BukrosSzabolcs 8cdb2cca44
HBASE-24562: Stabilize master startup with meta replicas enabled (#1903)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-06-24 18:38:36 +01:00
huaxiangsun 82fad5d789
HBASE-24552 Replica region needs to check if primary region directory exists at file system in TransitRegionStateProcedure (#1924)
Signed-off-by:  stack <stack@apache.org>
2020-06-24 09:43:12 -07:00
Duo Zhang 22103067b6
HBASE-24609 Move MetaTableAccessor out of hbase-client (#1943)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-06-24 21:20:35 +08:00
XinSun 4b89ee24f3
HBASE-24431 RSGroupInfo add configuration map to store something extra (#1782)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-06-24 18:46:09 +08:00
xincunSong d46adc7cc8
HBASE-24605 Break long region names in the web UI (#1942)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
2020-06-23 10:03:18 +08:00
Viraj Jasani 5a79a1c83c
HBASE-24102 : Undo visibility change for RegionMover fields (ADDENDUM) 2020-06-22 22:44:05 +05:30
Duo Zhang 8f1353b447 Revert "HBASE-24403 FsDelegationToken Should Cache Token After Acquired A New One (#1743)"
This reverts commit 7b396e9b8c.
2020-06-21 23:43:38 +08:00
Viraj Jasani 73ab216414
HBASE-24594 : Separate resource allocation for TestReplicationStatusSink (#1934)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-06-21 16:11:32 +05:30
XinSun dfaba965c9
HBASE-24591 get_table_rsgroup ignored the existence of rsgroup config for namespace (#1925)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-06-21 17:54:39 +08:00
Viraj Jasani 3c31981179
HBASE-24380 : Provide WAL splitting journal logging (#1860)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-06-19 16:25:03 -07:00
stack 537a3caccd HBASE-24574 Procedure V2 - Distributed WAL Splitting => LOGGING (#1912)
Addendum 2 fix compile error.
2020-06-18 08:49:39 -07:00
stack 489f07d945 HBASE-24574 Procedure V2 - Distributed WAL Splitting => LOGGING (#1912)
Addendum; minor log edits
2020-06-18 08:40:11 -07:00
Michael Stack aadd2bb1a1 HBASE-24574 Procedure V2 - Distributed WAL Splitting => LOGGING (#1912)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
 Don't register as a chore on construction if no coordination state
 manager instance (there is no instance when procv2 WAL splitter).

hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitWALManager.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
 Edit logs.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionRemoteProcedureBase.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/SplitWALProcedure.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/SplitWALRemoteProcedure.java
 Add proc name rather than rely on default behavior. Add detail to the
 toString.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 Factoring

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
 Print the maxLogs... we don't do it any where.

hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WAL.java
 Utility method to strip prefix from wal path.

Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: tianjingyun <tianjy@apache.org>
2020-06-18 08:38:59 -07:00