Modified compared to main branch to deprecate obviated MasterObserver interface methods instead of remove them.
Signed-off-by: Sean Busbey <busbey@apache.org>
(cherry picked from commit c5ca191921)
Conflicts:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AdminOverAsyncAdmin.java
hbase-server/src/test/java/org/apache/hadoop/hbase/rsgroup/VerifyingRSGroupAdmin.java
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestWithDisabledAuthorization.java
* HBASE-25065 - WAL archival to be done by a separate thread
* Fix checkstyle comments
* Fix compile issue
* Fix checkstyle and make the failing test more reliable
* Remove unused import
Implement a rate limiter for the normalizer. Implemented in terms of
MB/sec of affacted region size (the same metrics used to make
normalization decisions). Uses Guava `RateLimiter` to perform the
resource accounting. `RateLimiter` works by blocking (uninterruptible
😖) the calling thread. Thus, the whole construction of the normalizer
subsystem needed refactoring. See the provided `package-info.java` for
an overview of this new structure.
Introduces a new configuration,
`hbase.normalizer.throughput.max_bytes_per_sec`, for specifying a
limit on the throughput of actions executed by the normalizer. Note
that while this configuration value is in bytes, the minimum honored
valued `1_000_000`. Supports values configured using the
human-readable suffixes honored by `Configuration.getLongBytes`
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
Signed-off-by: Michael Stack <stack@apache.org>
Make it so WALPlayer can replay recovered.edits files.
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/WALInputFormat.java
Allow for WAL files that do NOT have a startime in their name.
Use the 'generic' WAL-filename parser instead of the one that
used be local here. Implement support for 'startTime' filter.
Previous was just not implemented.
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java
Checkstyle.
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/AbstractFSWALProvider.java
Use the new general WAL name timestamp parser.
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WAL.java
Utility for parsing timestamp from WAL filename.
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRecoveredEdits.java
Export attributes about the local recovered.edits test file
so other tests can play with it.
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Change the test to wait for evidence that the active master has seen
that the backup master killed by the test has gone away. This is done
before proceeding to validate that the dead backup is correctly
omitted from the ClusterStatus report.
Also, minor fixup to several assertions, using `assertEquals` instead
of `assertTrue(...equals(...))` and correcting expected vs. actual
ordering of assertion arguments.
Signed-off-by: Michael Stack <stack@apache.org>
Give the comparator a more descriptive name, a better location,
and make it work even when passed hbase:meta WAL files.
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Pass WALFactory to Replication instead of WALProvider. WALFactory has all
WALProviders in it, not just the user-space WALProvider. Do this so
ReplicationService has access to all WALProviders in the Server (To be
exploited by the follow-on patch in HBASE-25055)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Editing logging around region replicas: shortening and adding context.
Checkstyle fixes in edited files while I was in there.
Bug fix in AssignRegionHandler -- was using M_RS_CLOSE_META to open
a Region instead of a M_RS_OPEN_META.
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Closes#2423
Untangle RegionInfo, RegionInfoBuilder, and MutableRegionInfo static
initializations some. Move MutableRegionInfo from inner-class of
RegionInfoBuilder to be (package private) standalone. Undo static
initializing references from RI to RIB.
Co-authored-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
* Admin API getLogEntries() for ring buffer use-cases: so far, provides balancerDecision and slowLogResponse
* Refactor RPC call for similar use-cases
* Single RPC API getLogEntries() for both Master.proto and Admin.proto
Closes#2261
Signed-off-by: Andrew Purtell <apurtell@apache.org>
* Break subclass referencing of MetaCellComparator from superclass CellComparatorImpl
static initializer by moving META_COMPARATOR to subclass MetaCellComparator
Closes#2329
Signed-off-by: Duo Zhang <zhangduo@apache.org>
This patch adds the ability to discover newly added masters
dynamically on the master registry side. The trigger for the
re-fetch is either periodic (5 mins) or any registry RPC failure.
Master server information is cached in masters to avoid repeated
ZK lookups.
Updates the client side connection metrics to maintain a counter
per RPC type so that clients have visibility into counts grouped
by RPC method name.
I didn't add the method to ZK registry interface since there
is a design discussion going on in splittable meta doc. We can
add it later if needed.
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 275a38e153)
Adds region state check on hbck2 assigns/unassigns. Returns pid of -1
if in inappropriate state with logging explaination which suggests
passing override if operator wants to assign/unassign anyways. Here
is an example of what happens now if hbck2 tries an unassign and
Region already unassigned:
2020-08-19 11:22:06,926 INFO [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=50086] assignment.AssignmentManager(820): Failed {ENCODED => d1112e553991e938b6852f87774c91ee, NAME => 'TestHbck,zzzzz,1597861310769.d1112e553991e938b6852f87774c91ee.', STARTKEY => 'zzzzz', ENDKEY => ''} unassign, override=false; set override to by-pass state checks.
org.apache.hadoop.hbase.client.DoNotRetryRegionException: Unexpected state for state=CLOSED, location=null, table=TestHbck, region=d1112e553991e938b6852f87774c91ee
at org.apache.hadoop.hbase.master.assignment.AssignmentManager.preTransitCheck(AssignmentManager.java:583)
at org.apache.hadoop.hbase.master.assignment.AssignmentManager.createOneUnassignProcedure(AssignmentManager.java:812)
at org.apache.hadoop.hbase.master.MasterRpcServices.unassigns(MasterRpcServices.java:2616)
at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$HbckService$2.callBlockingMethod(MasterProtos.java)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:397)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
Previous it would just create the unassign anyways. Now must pass override
to queue the procedure regardless. Safer.
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
javadoc on assigns/unassigns. Minor refactor in assigns/unassigns to cater to
case where procedure may come back null (if override not set and fails state checks).
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
checkstyle cleanups.
Clarifying javadoc on how there is no state checking when bulk assigns creating/enabling
tables.
createOneAssignProcedure and createOneUnassignProcedure now handle exceptions which now
can be thrown if no override and region state is not appropriate.
Aggregation of createAssignProcedure and createUnassignProcedure instances adding in
region state check invoked if override is NOT set.
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateNode.java
Change to setProcedure so it returns passed proc as result instead of void
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Introduce an additional method to our Admin interface that allow an
operator to selectivly run the normalizer. The IPC protocol supports
general table name select via compound filter.
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
when neighbor is larger than average size
* add `testMergeEmptyRegions` to explicitly cover different
interleaving of 0-sized regions.
* fix bug where merging a 0-size region is skipped due to large
neighbor.
* remove unused `splitPoint` from `SplitNormalizationPlan`.
* generate `toString`, `hashCode`, and `equals` methods from Apache
Commons Lang3 template on `SplitNormalizationPlan` and
`MergeNormalizationPlan`.
* simplify test to use equality matching over `*NormalizationPlan`
instances as plain pojos.
* test make use of this handy `TableNameTestRule`.
* fix line-length issues in `TestSimpleRegionNormalizer`
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
Signed-off-by: Aman Poonia <aman.poonia.29@gmail.com>
Looped through the test 100 times and it passes. Without the patch it fails
every ~10 runs or so.
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
Looped through the test 100 times and it passes. Without the patch it fails
every ~10 runs or so.
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
Allow specifying base WALEntry filter on construction of
ReplicationSource. Add means of being able to filter WALs by name.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
Add constructor that allows passing a predicate for filtering *in* WALs
and a list of filters for filtering *out* WALEntries. The latter was
hardcoded to filter out system-table WALEntries. The former did not
exist but we'll need it if Replication takes in more than just the
default Provider.
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
writer null check if not initialized yet during syncrunner run (#2201)
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: AnoopSamJohn<anoopsamjohn@apache.org>
(cherry picked from commit b0863c5832)
Wellington Ramos Chevreuil
bsglz
niuyulin
Duo Zhang
ramkrish86
Nick Dimiduk
Michael Stack
Toshihiro Suzuki
Andor Molnár
Viraj Jasani
Mohammad Arshad
Joseph295
meiyi
Bharath Vissapragada
Andrew Purtell
stack
Hyeran Lee
shahrs87
XinSun
Clara Xiong
Sandeep Pal
Javier Akira Luca de Tena
ankitjain64
WenFeiYi
GeorryHuang
Qi Yu
Yechao Chen
Guanghao Zhang
chenglei
Bo Cui
Josh Elser
sguggilam
gkanade
Sean Busbey