Closes#2423
Untangle RegionInfo, RegionInfoBuilder, and MutableRegionInfo static
initializations some. Move MutableRegionInfo from inner-class of
RegionInfoBuilder to be (package private) standalone. Undo static
initializing references from RI to RIB.
Co-authored-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
* Admin API getLogEntries() for ring buffer use-cases: so far, provides balancerDecision and slowLogResponse
* Refactor RPC call for similar use-cases
* Single RPC API getLogEntries() for both Master.proto and Admin.proto
Closes#2261
Signed-off-by: Andrew Purtell <apurtell@apache.org>
* Break subclass referencing of MetaCellComparator from superclass CellComparatorImpl
static initializer by moving META_COMPARATOR to subclass MetaCellComparator
Closes#2329
Signed-off-by: Duo Zhang <zhangduo@apache.org>
This patch adds the ability to discover newly added masters
dynamically on the master registry side. The trigger for the
re-fetch is either periodic (5 mins) or any registry RPC failure.
Master server information is cached in masters to avoid repeated
ZK lookups.
Updates the client side connection metrics to maintain a counter
per RPC type so that clients have visibility into counts grouped
by RPC method name.
I didn't add the method to ZK registry interface since there
is a design discussion going on in splittable meta doc. We can
add it later if needed.
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 275a38e153)
Mark RegionInfo#UNDEFINED IA.Private and deprecated;
it is for internal use only and likely to be removed in hbase4.
Move MutableRegionInfo out of RegionInfoBuilder and have it as a
stanadlone task; a nice-to-have.
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Introduce an additional method to our Admin interface that allow an
operator to selectivly run the normalizer. The IPC protocol supports
general table name select via compound filter.
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Purge query Master for table descriptors; make do w/ generic options.
Logging cleanup.
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/BoundedRecoveredHFilesOutputSink.java
Undo fetching Table Descriptor. Not reliably available at recovery time.
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
It addresses couple issues:
1. Make sure deleteMergeQualifiers() does not delete the row if there is no columns with "merge" keyword.
2. GCMulitpleMergedRegionsProcedure now acquire an exclusive lock on the child region.
Signed-off-by: stack <stack@apache.org>
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
Fix weird brackets around each region name when logging.
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetaFixer.java
Log when we hit the max merge limit. Also up limit to 64.
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
Make logs make more sense to operator.
hbase-server/src/main/resources/hbase-webapps/master/hbck.jsp
Make RegionName show when you mouseover so long names don't mess up
display of holes and overlaps.
Address Mingliang Liu liuml07 feedback
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionInfo.java
Add new isOverlap method that takes list of RegionInfos checking that
current RegionInfo is overlapped by the passed in Regions.
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientAsyncPrefetchScanner.java
Refactor to avoid NPE timing issue referencing lock during Construction.
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java
Comment
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
Refactor. Catch NPE during startup and return it instead as failed initialization.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplit.java
Catch IndexOutOfBounds exception and convert to non-split request.
hbase-server/src/test/java/org/apache/hadoop/hbase/TestCachedClusterId.java
Make less furious. Make it less flakie.
hbase-server/src/test/java/org/apache/hadoop/hbase/TestServerSideScanMetricsFromClientSide.java
Debug. Catch exception to log, then rethrow.
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java
Guess that waiting longer on compaction to succeed may help make this
less flakey.
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java
Be explicit about timestamping to avoid concurrent edit landing
server-side and messing up test expectation.
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMasterRegistry.java
Add wait on meta before proceeding w/ test.
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannersFromClientSide.java
Be explicit that edits are distinct.
hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/bucket/TestBucketCacheRefCnt.java
Add @Ignore on RAM test... Fails sporadically.
hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestRegionMoveAndAbandon.java
Add wait for all RegionServers going down before proceeding; was
messing up RS accounting.
hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
Make balancer test sloppier; less restrictive; would fail on occasion
by being just outside test limits.
hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaObserverChoreRegionReports.java
Add wait on quota table coming up; helps make this less flakie.
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
Be explicity about timestamps; see if helps w/ flakie failure.
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
Catch and ignore if issue in shutdown; don't care if after test.
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerReportForDuty.java
Comment.
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java
Add retry to see if helps w/ odd failure; grant hasn't propagated?
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestCellACLWithMultipleVersions.java
Explicit w/ timestamps so no accidental overlap of puts.
hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftHttpServer.java
hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftServerCmdLine.java
Hack to deal w/ BindException on startup.
hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift2/TestThrift2ServerCmdLine.java
Use loopback.
hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift2/TestThriftHBaseServiceHandler.java
Disable flakie test.
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
hbase-rsgroup/src/test/java/org/apache/hadoop/hbase/rsgroup/TestRSGroupMajorCompactionTTL.java
Remove spurious assert. Just before this it waits an arbitrary 10
seconds. Compactions could have completed inside this time. The spirit
of the test remains.
hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java
Get log cleaner to go down promptly; its sticking around. See if this
helps with TestMasterShutdown
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
We get a rare NPE trying to sync. Make local copy of SyncFuture and see
if that helps.
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java
Compaction may have completed when not expected; allow for it.
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestBlockEvictionFromClient.java
Add wait before testing. Compaction may not have completed. Let
compaction complete before progressing and then test for empty cache.
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterShutdown.java
Less resources.
hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestDefaultLoadBalancer.java
Less resources.
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestEndToEndSplitTransaction.java
Wait till online before we try and do compaction (else request is
ignored)
hbase-server/src/test/java/org/apache/hadoop/hbase/tool/TestCanaryTool.java
Disable test that fails randomly w/ mockito complaint on some mac os
x's.
TestMasterShutdown... fix NPE in RSRpcDispatcher... catch it and covert
to false and have master check for successful startup.
Add being able to configure netty thread counts. Enable socket reuse
(should not have any impact).
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java
Rename the threads we create in here so they are NOT named same was
threads created by Hadoop RPC.
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java
Allow configuring eventloopgroup thread count (so can override for
tests)
hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java
Enable socket resuse.
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java
Enable socket resuse and config for how many threads to use.
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java
Thread name edit; drop the redundant 'Thread' suffix.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java
Make closeable and shutdown executor when called.
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
Call close on HFileReplicator
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
HDFS creates lots of threads. Use less of it so less threads overall.
hbase-server/src/test/resources/hbase-site.xml
hbase-server/src/test/resources/hdfs-site.xml
Constrain resources when running in test context.
hbase-server/src/test/resources/log4j.properties
Enable debug on netty to see netty configs in our log
pom.xml
Add system properties when we launch JVMs to constrain thread counts in
tests
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Restore behavior from before HBASE-21789 (hbase-2.2.0) where we convert
all exceptions to IOEs, even RuntimeExceptions. Actual fix is this change (in case
obscured by doc and lambda simplification):
} catch (Throwable e) {
- Throwables.propagateIfPossible(e, IOException.class);
+ // Throw if an IOE else wrap in an IOE EVEN IF IT IS a RuntimeException (e.g.
+ // a RejectedExecutionException because the hosting exception is shutting down.
+ // This is old behavior worth reexamining. Procedures doing merge or split
+ // currently don't handle RuntimeExceptions coming up out of meta table edits.
+ // Would have to work on this at least. See HBASE-23904.
+ Throwables.throwIfInstanceOf(e, IOException.class);
A miscellaney. Add extra logging to help w/ debug to a bunch of tests.
Fix some issues particular where we ran into mismatched filesystem
complaint. Some modernizations, removal of unnecessary deletes
(especially after seeing tests fail in table delete), and cleanup.
Recategorized one tests because it starts four clusters in the one
JVM from medium to large. Finally, zk standalone server won't come
on occasion; added debug and thread dumping to help figure why (
manifests as test failing in startup saying master didn't launch).
hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
Fixes occasional mismatched filesystems where the difference is file:// vs file:///
or we pick up hdfs schema when it a local fs test. Had to do this
vetting of how we do make qualified on a Path in a few places, not
just here as a few tests failed with this same issue. Code in here is
used by a lot of tests that each in turn suffered this mismatch.
Refactor for clarity
hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotV1NoCluster.java
Unused import.
hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/TestWALProcedureStore.java
This test fails if tmp dir is not where it expects because tries to
make rootdir there. Give it a rootdir under test data dir.
hbase-server/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java
This change is probably useless. I think the issue is actually
a problem addressed later where our test for zk server being
up gets stuck and never times out.
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSplitOrMergeStatus.java
Move off deprecated APIs.
hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
Log when we fail balance check for DEBUG Currently just says 'false'
hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestSplitWALProcedure.java
NPEs on way out if setup failed.
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
Add logging when assert fails to help w/ DEBUG
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerAbortTimeout.java
Don't bother removing stuff on teardown. All gets thrown away anyways.
Saw a few hangs in here in the teardown where hdfs was down before
expected messing up shutdown.
hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
Add timeout on socket; was seeing check for zk server getting stuck
and never timing out (test time out in startup)
hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotWithTemporaryDirectory.java
Write to test data dir instead.
Be careful about how we make qualified paths.
hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java
Remove snowflake configs.
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStatus.java
Add a hacky pause. Tried adding barriers but didn't work. Needs deep
dive.
hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
Remove code copied from zk and use zk methods directly instead.
A general problem is that zk cluster doesn't come up occasionally but
no clue why. Add thread dumping and state check.
Master rpc server end point doesn't bind to localhost's
IP address by default. Instead, it looks up the hostname and
binds to the endpoint to which it resolves. MasterRegistry should
do the same when building the default server end point to talk to.
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
(cherry picked from commit c34dbc3c29)
Implements a master based registry for clients.
- Supports hedged RPCs (fan out configured via configs).
- Parameterized existing client tests to run with multiple registry combinations.
- Added unit-test coverage for the new registry implementation.
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
(cherry picked from commit 62da419b23)
* HBASE-23648: Re-use underlying connection registry in RawAsyncHBaseAdmin
No need to create and close a new registry on demand. Audited other
usages of getRegistry() and the code looks fine.
* Fix checkstyle issues in RawAsyncHBaseAdmin
(cherry picked from commit 07c38260f5)
* HBASE-23604: Cleanup AsyncRegistry interface
- Cleans up the method names to make more sense and adds a little
more javadocs for context. In future patches we can revisit
the name of the actual class to make it more self explanatory.
- Does AsyncRegistry -> ConnectionRegistry rename.
"async" ness of the registry is kind of implicit based on
the interface contents and need not be reflected in the name.
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 12bb41eb2c)
* HBASE-23304: RPCs needed for client meta information lookup
This patch implements the RPCs needed for the meta information
lookup during connection init. New tests added to cover the RPC
code paths. HBASE-23305 builds on this to implement the client
side logic.
Fixed a bunch of checkstyle nits around the places the patch
touches.
Signed-off-by: Andrew Purtell <apurtell@apache.org>
(cherry picked from commit 4f8fbba0c0)
* HBASE-23281: Track meta region changes on masters
This patch adds a simple cache that tracks the meta region replica
locations. It keeps an eye on the region movements so that the
cached locations are not stale.
This information is used for servicing client RPCs for connections
that use master based registry (HBASE-18095). The RPC end points
will be added in a separate patch.
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
(cherry picked from commit 8571d389cf)
Sanjeet Nishad
Duo Zhang
Joseph295
王伟
niuyulin
Viraj Jasani
stack
shahrs87
ankitjain64
Toshihiro Suzuki
Mohammad Arshad
Bharath Vissapragada
Michael Stack
mokai
Guanghao Zhang
Nick Dimiduk
bsglz
surbhi
huaxiangsun
Bo Cui
QilinCao
BukrosSzabolcs
Ravi Kishore Valeti
Wellington Ramos Chevreuil
Mingliang Liu
xuqinya1