Commit Graph

17450 Commits

Author SHA1 Message Date
Andrew Purtell 282ab70012 Preparing hbase release 2.4.0RC1; tagging and updates to CHANGES.md and RELEASENOTES.md 2020-12-03 09:51:41 -08:00
Andrew Purtell f7bc7be1eb Revert "Preparing development version 2.4.1-SNAPSHOT"
This reverts commit 4127a3940a.
2020-12-03 09:37:38 -08:00
Viraj Jasani 75421af689
Revert "HBASE-25246 Backup/Restore hbase cell tags"
This reverts commit 34721c42ec.
2020-12-03 21:18:03 +05:30
Laxman Goswami 9a2994a5a3
HBASE-25230 Embedded zookeeper server not clean up the old data
Closes #2732

Signed-off-by: maoling <maoling199210191@sina.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-03 14:46:25 +05:30
Andrew Purtell 4127a3940a Preparing development version 2.4.1-SNAPSHOT 2020-12-02 14:04:53 -08:00
Andrew Purtell 108c2ecb52 Preparing hbase release 2.4.0RC0; tagging and updates to CHANGES.md and RELEASENOTES.md 2020-12-02 14:04:49 -08:00
Andrew Purtell 048cee6e47 Rebase RELEASENOTES.md on rel/2.3.0 to bring up to date 2020-12-02 12:53:35 -08:00
Andrew Purtell ffeed7c664 Refer to earlier releases in CHANGES.md with hyperlinks 2020-12-02 11:13:28 -08:00
Michael Stack 9cc2f76248
HBASE-25349 [Flakey Tests] branch-2 TestRefreshRecoveredReplication.testReplicationRefreshSource:141 Waiting timed out after [60,000] msec (#2731)
Start the check for recovered queue presence earlier.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-12-02 09:55:24 -08:00
shahrs87 34721c42ec
HBASE-25246 Backup/Restore hbase cell tags
Closes #2706

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-02 21:30:46 +05:30
Duo Zhang 90e96d1a6e HBASE-25342 Upgrade error prone to 2.4.0 (#2725)
Have to disable MutablePublicArray because of a bug in error prone
https://github.com/google/error-prone/issues/1645

Signed-off-by: stack <stack@apache.org>
2020-12-02 23:38:05 +08:00
stack 745a3a9ab7 Revert "Revert "HBASE-25320 Upgrade hbase-thirdparty dependency to 3.4.1 (#2693)""
This reverts commit 78f30ff496.

This is a revert of a revert; i.e. a repplication -- original patch was
good.
2020-12-01 17:35:43 -08:00
stack 78f30ff496 Revert "HBASE-25320 Upgrade hbase-thirdparty dependency to 3.4.1 (#2693)"
This reverts commit 0851e3273a.
2020-12-01 09:35:51 -08:00
Duo Zhang 0851e3273a HBASE-25320 Upgrade hbase-thirdparty dependency to 3.4.1 (#2693)
Signed-off-by: stack <stack@apache.org>
2020-12-01 08:17:17 -08:00
lujiefsi a1db679e4d
HBASE-25332:fix One pontential NPE
Closes #2715

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-12-01 19:47:09 +05:30
clarax 4c8fa47b28
HBASE-25127 Enhance PerformanceEvaluation to profile meta replica performance. (#2655)
* HBASE-25127 Enhance PerformanceEvaluation to profile meta replica performance. (#2644)


Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
Signed-off-by: stack <stack@apache.com>
2020-11-30 22:51:57 -08:00
Michael Stack f0a4aefcdf
HBASE-25345 [Flakey Tests] branch-2 TestReadReplicas#testVerifySecondaryAbilityToReadWithOnFiles (#2727)
Check TEST_SKIP_REPORTING_TRANSITION and if true, skip trying to talk to
update master on state transition -- i.e. reportFileArchivalForQuotas --
as we allow for reportRegionStateTransition (For tests only)

Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-11-30 21:02:45 -08:00
stack 01337e4104 Revert "HBASE-25127 Enhance PerformanceEvaluation to profile meta replica performance. (#2644) (#2682)"
This reverts commit 5fcbf1f9f2.
2020-11-30 16:45:59 -08:00
Michael Stack 5fcbf1f9f2 HBASE-25127 Enhance PerformanceEvaluation to profile meta replica performance. (#2644) (#2682)
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
Signed-off-by: stack <stack@apache.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-30 10:29:29 -08:00
GeorryHuang 433235e98c
HBASE-25339 Method parameter and member variable are duplicated in checkSplittable() of SplitTableRegionProcedure
Closes #2717

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-30 19:50:54 +05:30
Viraj Jasani 904b8a814b
Revert "HBASE-25339 Method parameter and member variable are duplicated in checkSplittable() of SplitTableRegionProcedure"
This reverts commit 861c5509ad.
2020-11-30 19:50:29 +05:30
GeorryHuang 861c5509ad
HBASE-25339 Method parameter and member variable are duplicated in checkSplittable() of SplitTableRegionProcedure
Closes #2717

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-30 18:01:05 +05:30
Balazs Meszaros b75b772bc6
HBASE-25307 ThreadLocal pooling leads to NullPointerException (#2709)
* PoolMap does not discard any elements anymore. If an element is put,
it always stores it. The reason: it stores expensive resources (rpc
connections) which would lead to resource leak if we simple discard it.
RpcClients can reference netty ByteBufs which are reference counted.
Resource cleanup is done by AbstractRpcClient.cleanupIdleConnections().
* PoolMap does not implement Map interface anymore, so ensuring
thread-safety has become easier. Put method is replaced with getOrCreate().
* ThreadLocalPool doesn't use ThreadLocal class anymore. It stores
resources on thread basis, but it doesn't remove values when a thread
exits. Again, proper cleanup is done by cleanupIdleConnections().

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Wellington Chevreuil <wellington.chevreuil@gmail.com>
2020-11-30 12:36:03 +01:00
Duo Zhang d6d0a20247 HBASE-25341 Fix ErrorProne error which causes nightly to fail (#2718)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-30 14:48:41 +08:00
Baiqiang Zhao e927fe88fb
HBASE-25330 RSGroupInfoManagerImpl#moveServers return is not set of servers moved (#2719)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-30 11:40:51 +08:00
Akshay Sudheer 85bf542a44 HBASE-25321 The sort icons not shown after Upgrade JQuery to 3.5.1 (#2694)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-29 12:13:16 +08:00
Lokesh Khurana 8472c68720
HBASE-25237 : 'hbase master stop' shuts down the cluster, not the master only
Closes #2705

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-27 16:05:07 +05:30
Qi Yu e81f49c918 HBASE-25324 Remove unnecessary array to list conversion in SplitLogManager (#2703)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.com>
2020-11-27 08:21:13 +08:00
Toshihiro Suzuki 3775464981
HBASE-25242 Add Increment/Append support to RowMutations (#2711)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-11-27 03:53:19 +09:00
Mate Szalay-Beko 1dbfe96d69 HBASE-25263 Various improvements of column family encryption
This PR is a follow-up of HBASE-25181 (#2539), where several issues were
discussed on the PR:

1. Currently we use PBKDF2WithHmacSHA1 key generation algorithm to generate a
secret key for HFile / WalFile encryption, when the user is defining a string
encryption key in the hbase shell. This algorithm is not secure enough and
not allowed in certain environments (e.g. on FIPS compliant clusters). We are
changing it to PBKDF2WithHmacSHA384. It will not break backward-compatibility,
as even the tables created by the shell using the new algorithm will be able
to load (e.g. during bulkload / replication) the HFiles serialized with the
key generated by an old algorithm, as the HFiles themselves already contain
the key necessary for their decryption.

Smaller issues fixed by this commit:

2. Improve the documentation e.g. with the changes introduced by HBASE-25181
and also by some points discussed on the Jira ticket of HBASE-25263.

3. In EncryptionUtil.createEncryptionContext the various encryption config
checks should throw IllegalStateExceptions instead of RuntimeExceptions.

4. Test cases in TestEncryptionTest.java should be broken down into smaller
tests.

5. TestEncryptionDisabled.java should use ExpectedException JUnit rule to
validate exceptions.

closes #2676

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-11-26 16:57:12 +01:00
niuyulin 751b4f040e HBASE-24872 refactor valueOf PoolType (#2250)
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-11-26 12:50:59 +01:00
Jonathan M Hsieh 35a2875dc1 HBASE-24827 BackPort HBASE-11554 Remove Reusable poolmap Rpc client type. (#2208)
Co-authored-by: niuyulin <niuyulin@xiaomi.com>
Signed-off-by: stack <stack@apache.org>
2020-11-26 11:22:40 +01:00
Andrew Purtell 1b6399a480
HBASE-24640 [branch-2] Purge use of VisibleForTesting (#2696)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-11-25 14:15:32 -08:00
Qi Yu 3dd425abfa HBASE-25323 Fix potential NPE when the zookeeper path of RegionServerTracker does not exist when start (#2702)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-25 15:28:32 +08:00
Qi Yu 6f07efb971 HBASE-25325 Remove unused class ClusterSchemaException (#2704)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-25 15:28:32 +08:00
niuyulin 8e3727ea06
HBASE-25281 Bulkload split hfile too many times due to unreasonable split point (#2692)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-25 15:27:09 +08:00
niuyulin d523d758af HBASE-25213 Should request Compaction when bulkLoadHFiles is done (addendum) (#2701)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-25 08:00:28 +08:00
Richard Antal 5ce3e3e12c HBASE-24268 REST and Thrift server do not handle the "doAs" parameter case insensitively
Closes #1843

Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-11-24 12:25:46 -05:00
Josh Elser 31917b0a8a HBASE-25278 Add CACHE_BLOCKS option to count shell command
Expose an argument on the `count` command which is passed to the
`setCacheBlocks` method on the Scan which the count command uses.

This is a quick and dirty approach to read all of the blocks for a table
into the block cache.

* Raise an error when the value isn't a boolean or the expected string

Closes #2650

Signed-off-by: Zach York <zyork@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-11-24 10:05:54 -05:00
ramkrish86 988be9321d HBASE-25050 - We initialize Filesystems more than once. (#2419)
* HBASE-25050 - We initialize Filesystems more than once.

* Ensuring that calling the FS#get() will only ensure FS init.

* Fix for testfailures. We should pass the entire path and no the scheme
alone

* Cases where we don't have a scheme for the URI

* Address review comments

* Add some comments on why FS#get(URI, conf) is getting used

* Adding the comment as per Sean's review

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2020-11-24 15:31:55 +05:30
ramkrish86 a8e02c97bc HBASE-25187 Improve SizeCachedKV variants initialization (#2582)
* HBASE-25187 Improve SizeCachedKV variants initialization

* HBASE-25187 Improve SizeCachedKV variants initialization

* The BBKeyValue also can be optimized

* Change for SizeCachedKeyValue

* Addressing revew comments

* Fixing checkstyle and spot bugs comments

* Spot bug fix for hashCode

* Minor updates make the rowLen as short and some consturctor formatting

* Change two more places where there was a cast
2020-11-24 15:10:44 +05:30
Bo Cui 0a1decdf18 HBASE-25311 UI throws NPE (#2688)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Wellington Chevreuil <wellington.chevreuil@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-11-23 12:08:14 +08:00
niuyulin 8976781302
HBASE-25213 Should request Compaction after bulkLoadHFiles is done (#2684)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-11-23 08:33:41 +08:00
Michael Stack febd832f2f HBASE-25284 Check-in "Enable memstore replication..." design (#2680)
Update the design document to the state as of not. Adds a paragraph
on the preamble. See discussion on end of HBASE-25284.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-11-21 08:53:28 -08:00
stack 73086f25e6 HBASE-25291 Document how to enable the meta replica load balance mode for the client and clean up around hbase:meta read replicas
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSelector.java
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSelectorFactory.java
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaLoadBalanceSimpleSelector.java
hbase-client/src/main/java/org/apache/hadoop/hbase/client/CatalogReplicaMode.java
 Make the leading license comment opener align on all these new
 files.... one star rather than two.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java
 Add some comment better situating the catalogreplicationsource
 specialization.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
 Add comment how this differs from replicationsource factory and why no
 need to keep offset up in zk.

src/main/asciidoc/_chapters/architecture.adoc
 Clean up the read replica of META doc. Introduce the new read replica
 liveness on META and the new 'LoadBalance' feature (with warnings).
 General edit.
2020-11-20 15:49:00 -08:00
Huaxiang Sun 920cf9faf4 HBASE-25126 Add load balance logic in hbase-client to distribute read load over meta replica regions
It adds load balance support for meta lookup in AsyncTableRegionLocator.
The existing meta replica mode is renamed as "HedgedRead", client sends scan request to the primary meta replica region first,
if response is not back within a configured amount of time, it will send scan requests to all meta replica regions and
take the first response. On top of the existing mode, a new mode "LoadBalance" is introduced. In this mode, client first
choose a meta replica region to send scan request. If the response is stale, it may send the request to another meta replica region or
the primary region. In this mode, meta scan requests are load balanced across all replica regions with the primary mode as
the ultimate source of truth.

Two new config knobs are added:

1. hbase.locator.meta.replicas.mode
   Valid options are "None", "HedgedRead" and "LoadBalance", they are case insensitive. The default mode is "None".

2. hbase.locator.meta.replicas.mode.loadbalance.selector
   The load balance alogrithm to select a meta replica to send the requests.
   Only org.apache.hadoop.hbase.client.CatalogReplicaLoadBalanceReplicaSimpleSelector.class
   is supported for now, which is the default as well. The algorithm works as follows:
      a. Clients select a randome meta replica region to send the requests if there is no entry for the location in the stale
         location cache.
      b. If the location from one meta replica region is stale, a stale entry will be created in the statle location cache
         for the region.
      c. Clients select the primary meta region if the location is in the stale location cache.
      d. The stale location cache entries time out in 3 seconds.

If there is no "hbase.locator.meta.replicas.mode" configured, it will check the config knob "hbase.meta.replicas.use".
If "hbase.meta.replicas.use" is configured, the mode will be set to "HedgedRead".

For branch-2 support, it introduces support for ReversedScan over a specific non-default replica region, this is mainly
for load balance meta scan among all its replica regions.

Remove ConnectionImplementation#setUseMetaReplicas()
2020-11-20 10:45:24 -08:00
Michael Stack 47ad207664 HBASE-25280 [meta replicas] ArrayIndexOutOfBoundsException in ZKConnectionRegistry (#2652)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-11-20 00:31:43 -08:00
stack 1f6779b2e0 HBASE-25151 warmupRegion frustrates registering WALs on the catalog replicationsource
warmupRegion called by Master on Region move will instatiate
the meta WALProvider as part of its action making it so
it is already created by the time we go to open the
hbsae:meta Region. Accommodate meta walProvider
being already up.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
 Pass regionInfo. Needed internally.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
 Add handling if meta wal provider already instantiated when
 addCatalogReplicationSource runs.

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java
 Add exercising moving meta around between servers. Test replication
 keeps working.
2020-11-20 00:31:43 -08:00
Michael Stack de1843ac94 HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable w… (#2451)
* HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable when hbase:meta assigned to RS

Fill in gap left by HBASE-11183 'Timeline Consistent region replicas - Phase 2 design'.
HBASE-11183 left off implementing 'async WAL Replication' on the hbase:meta
Table; hbase:meta Table could only do Phase 1 Region Replicas reading
the primary Regions' hfiles. Here we add 'async WAL Replication' to
hbase:meta so Replicas can be more current with the primary's changes.

Adds a 'special' ReplicationSource that reads hbase:meta WAL files and replicates
all edits to the configured in-cluster endpoint (Defaults to the
RegionReadReplicaEndpoint.class -- set hbase.region.replica.catalog.replication to
target a different endpoint implementation).

Set hbase.region.replica.replication.catalog.enabled to enable async WAL
Replication for hbase:meta region replicas. Its off by default.

The CatalogReplicationSource for async WAL Replication of hbase:meta does
NOT need to keep up WAL offset or a queue of WALs-to-replicate in the
replication queue store as is done in other ReplicationSource implementations;
the CatalogReplicationSource is for Region Replicas only. General
Replication does not replicate hbase:meta. hbase:meta Region Replicas reset
on crash of the primary replica so there is no need to 'recover'
replication that was running on the crashed server.

Because it so different in operation, the CatalogReplicationSource is bolted
on to the side of the ReplicationSourceManager. It is lazily
instantiated to match the lazy instantiation of the hbase:meta
WALProvider, created and started on the open of the first Region of an
hbase:meta table. Thereafter it stays up till the process dies, even if
all hbase:meta Regions have moved off the server, in case a hbase:meta
Region is moved back (Doing this latter simplifies the implementation)

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
  Read configuration to see if we need to wait on setting a Region read-enabled
  (if so, replicas will only flip to enable read after confirming a
  flush of the primary so they for sure are a replica of a known point)

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java
 If configured, on open of hbase:meta, ask the ReplicationSourceManager
 to add a ReplicationSource (if it hasn't already).

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java
 Edit log message.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java
 If configured, on close of hbase:meta, update ReplicationSourceManager
 that a source Region has closed.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java
 javadoc and make constructor private.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java
 Add logPositionAndCleanOldLogs w/ default of the old behavior so
 CatalogReplicationSource can bypass updating store with WAL position,
 etc.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
 Add creation and start of an CatalogReplicationSource.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java
 Go via ReplicationSource when calling logPostionAndCleanOldLogs so new RS can intercept.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java
 Javadoc.

hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java
 Add utility for reading configurations for hbase:meta region replicas.

hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java
 Javadoc.

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
 Use define.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java
 Specical version of ReplicationSource for Region Replicas on hbase:meta.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSourcePeer.java
 Needs a special peer too (peers are baked into replication though we don't use 'peers' here)

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALProvider.java
 Tests.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
2020-11-20 00:31:43 -08:00
stack 2b1e8b306f HBASE-25068 Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space
Pass WALFactory to Replication instead of WALProvider. WALFactory has all
WALProviders in it, not just the user-space WALProvider. Do this so
ReplicationService has access to all WALProviders in the Server (To be
exploited by the follow-on patch in HBASE-25055)
2020-11-20 00:31:43 -08:00