Commit Graph

8127 Commits

Author SHA1 Message Date
Bharath Vissapragada 5c88672d54 HBASE-23731: De-flake TestFromClientSide (#1091)
There were a couple of issues.

- There was a leak of a file descriptor for hbck lock file. This
was contributing to all the "ConnectionRefused" stack traces since
it was trying to renew lease for an already expired mini dfs cluster.
This issue was there for a while, just that we noticed it now.

- After upgrade to JUnit 4.13, it looks like the behavior for test
timeouts has changed. Earlier the timeout seems to have applied for
each parameterized run, but now it looks like it is applied across
all the runs.

This patch fixes both the issues.

Signed-off-by: Stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 229b8aaaf3 HBASE-23647: Make MasterRegistry the default impl. (#1039)
Signed-off-by: Stack <stack@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada d8b3f55518 HBASE-23330: Fix delegation token fetch with MasterRegistry (#1084)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 1aa6a4efb9 HBASE-23305: Master based registry implementation (#954)
Implements a master based registry for clients.

 - Supports hedged RPCs (fan out configured via configs).
 - Parameterized existing client tests to run with multiple registry combinations.
 - Added unit-test coverage for the new registry implementation.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 0a1c3b2055 HBASE-23604: Clarify AsyncRegistry usage in the code. (#957)
* HBASE-23604: Cleanup AsyncRegistry interface

- Cleans up the method names to make more sense and adds a little
more javadocs for context. In future patches we can revisit
the name of the actual class to make it more self explanatory.

- Does AsyncRegistry -> ConnectionRegistry rename.
"async" ness of the registry is kind of implicit based on
the interface contents and need not be reflected in the name.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 023078ffa4 HBASE-23304: RPCs needed for client meta information lookup (#904)
* HBASE-23304: RPCs needed for client meta information lookup

This patch implements the RPCs needed for the meta information
lookup during connection init. New tests added to cover the RPC
code paths. HBASE-23305 builds on this to implement the client
side logic.

Fixed a bunch of checkstyle nits around the places the patch
touches.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 8126566b27 HBASE-23281: Track meta region locations in masters (#830)
* HBASE-23281: Track meta region changes on masters

This patch adds a simple cache that tracks the meta region replica
locations. It keeps an eye on the region movements so that the
cached locations are not stale.

This information is used for servicing client RPCs for connections
that use master based registry (HBASE-18095). The RPC end points
will be added in a separate patch.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-02-19 20:10:17 -08:00
Bharath Vissapragada 3d298b087a HBASE-23275: Track active master's address in ActiveMasterManager (#812)
* HBASE-23275: Track active master's address in ActiveMasterManager

Currently we just track whether an active master exists.
It helps to also track the address of the active master in
all the masters to help serve the client RPC requests to
know which master is active.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-02-19 20:10:17 -08:00
stack 3168cf210c HBASE-23877 [Flakey Test] TestStochasticLoadBalancerRegionReplicaHighReplication "Two or more region replicas are hosted on the same host after balance" 2020-02-19 16:09:11 -08:00
stack 435b79335e HBASE-23872 [Flakey Test] TestGenerateDelegationToken; Master not initialized after 200000ms 2020-02-19 15:48:33 -08:00
Vladimir Rodionov b8194b4902 HBASE-22749 Distributed MOB compactions
- MOB compaction is now handled in-line with per-region compaction on region
  servers
- regions with mob data store per-hfile metadata about which mob hfiles are
  referenced
- admin requested major compaction will also rewrite MOB files; periodic RS
  initiated major compaction will not
- periodically a chore in the master will initiate a major compaction that
  will rewrite MOB values to ensure it happens. controlled by
  'hbase.mob.compaction.chore.period'. default is weekly
- control how many RS the chore requests major compaction on in parallel
  with 'hbase.mob.major.compaction.region.batch.size'. default is as
  parallel as possible.
- periodic chore in master will scan backing hfiles from regions to get the
  set of referenced mob hfiles and archive those that are no longer
  referenced. control period with 'hbase.master.mob.cleaner.period'
- Optionally, RS that are compacting mob files can limit write
  amplification by not rewriting values from mob hfiles over a certain size
  limit. opt-in by setting 'hbase.mob.compaction.type' to 'optimized'.
  control threshold by 'hbase.mob.compactions.max.file.size'.
  default is 1GiB
- Should smoothly integrate with existing MOB users via rolling upgrade.
  will delay old MOB file cleanup until per-region compaction has managed
  to compact each region at least once so that used mob hfile metadata can
  be gathered.
2020-02-19 16:06:38 -06:00
Guanghao Zhang 53afc1dc10
HBASE-23864 No need to submit SplitTableRegionProcedure/MergeTableRegionsProcedure when split/merge is disabled (#1182)
Signed-off-by: binlijin <binlijin@gmail.com>
2020-02-19 18:20:12 +08:00
stack 156ababa4e HBASE-23867 [Flakey Test] TestStochasticLoadBalancerRegionReplicaSameHosts#testRegionReplicationOnMidClusterSameHosts 2020-02-18 22:40:34 -08:00
stack be4b6d6824 HBASE-23866 More test classifications
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncMetaRegionLocator.java
 Add wait on no region in transition in setup before proceeding.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncResultScannerCursor.java
 Make test large. Takes longer than 50seconds.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableRSCrashPublish.java
 Make large because it fails when networking is weird... when VPN or
 something is up. See HBASE-23850. When large, doesn't get run
 when you do mvn test.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableScanAll.java
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannersFromClientSide2.java
hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestMultiRowRangeFilter.java
hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/TestStochasticLoadBalancerRegionReplica.java
 Make it large. Takes too long.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSeparateClientZKCluster.java
 Use data dir instead of /tmp.

M hbase-server/src/test/java/org/apache/hadoop/hbase/http/TestInfoServersACL.java
 Wait till mbean is up before proceeding w/ test.

hbase-server/src/test/java/org/apache/hadoop/hbase/security/token/SecureTestCluster.java
 Test for null CLUSTER in tearDown.
2020-02-18 21:11:21 -08:00
GeorryHuang c4fb1cd234
HBASE-23859 Modify "Block locality" of RegionServer Web UI to human readable percentage (#1176)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-02-18 10:06:47 +08:00
GeorryHuang e11dbf36a5
HBASE-23855: Change bytes size to human readable size for Server Metrics of RegionServer Web UI (#1175)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-02-17 09:14:53 +08:00
Jan Hentschel ae781e569c
HBASE-23837 Removed deprecated isLegalFamilyName(byte[]) from HColumnDescriptor
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-15 21:19:51 +01:00
Nick Dimiduk 533302adde
HBASE-23808 [Flakey Test] TestMasterShutdown#testMasterShutdownBefore… (#1141)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-02-13 14:02:15 -08:00
stack b7ef225609 HBASE-23838 Adding debug logging to a few ExportSnapshot tests
* hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java
 Add logging of temp output dir and if tmp dir is being used at all.

* hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
 Don't create new Configurations. Use current. Set it into the launched
 ExportSnapshot too. Log as we make progress through stages so easier to
 find location of exception source.

* hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotV1NoCluster.java
 Add makeQualified though looks to be redundant.

* hbase-server/src/test/java/org/apache/hadoop/hbase/http/TestInfoServersACL.java
 Might fix a local fail where hbase doesn't seem to be up when we query
 jmx. Wait on meta being online.

M hbase-server/src/test/java/org/apache/hadoop/hbase/master/assignment/TestRegionStateStore.java
 Run all on the one cluster instance. See if helps w/ odd failure when
 run locally.
2020-02-13 13:50:43 -08:00
maoling 12f98bc50e
HBASE-20623: Introduce the helper method "getCellBuilder()" to Mutation (#1121)
Signed-off-by: stack <stack@apache.org>
2020-02-10 10:03:35 -08:00
stack a6774ca7f7 HBASE-23812 [Flakey Test] TestReplicator#testReplicatorWithErrors: AssertionError: We did not replicate enough rows expected:<10> but was:<7> 2020-02-10 09:45:43 -08:00
meiyi 29a36568b1
HBASE-23824 TestSnapshotScannerHDFSAclController is flakey (#1157)
Signed-off-by: stack <stack@apache.org>
2020-02-10 09:40:05 -08:00
stack 93c04e4f60 HBASE-23815 [Flakey Test] AbstractTestAsyncTableRegionReplicasRead family of tests fails with no breadcrumbs on why 2020-02-10 09:04:36 -08:00
stack a4aa183f45 HBASE-23814 Add null checks and logging to misc set of tests
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Zach York <zyork@apache.org>

* TestFullLogReconstruction log the server we've chosen to expire and then note where we starting counting rows
* TestAsyncTableScanException use a define for row counts
* TestRawAsyncTableLimitedScanWithFilter check connection was made before closing it in tearDown
* TestLogsCleaner use single mod time. Make it for sure less than now in case test runs all in the same millisecond (would cause test fail)
* TestReplicationBase test table is non-null before closing in tearDown
2020-02-10 08:51:56 -08:00
GeorryHuang 35989c1968
HBASE-23822:Fix typo in procedures.jsp (#1155)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-02-10 09:48:01 +08:00
binlijin 24823ecfc9
HBASE-23682 Fix NPE when disable DeadServerMetricRegionChore (#1026)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-02-08 15:20:23 -08:00
stack 8654597be0 HBASE-23805 [Flakey Test] TestRaceBetweenSCPAndDTP 2020-02-07 08:35:43 -08:00
Jan Hentschel 93c5e7691f
HBASE-23775 Removed deprecated method createLocalHTU(Configuration) from HBaseTestingUtility
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-02-07 09:51:04 +01:00
Nick Dimiduk 5b4545de5e
HBASE-23802 Remove unnecessary Configuration instantiation in LossyAccounting (#1127)
Signed-off-by: stack <stack@apache.org>
2020-02-05 14:41:50 -08:00
stack 3a1a39d40d HBASE-23789 [Flakey Tests] ERROR [Time-limited test] balancer.HeterogeneousRegionCountCostFunction(199): cannot read rules file located at ' /tmp/hbase-balancer.rules '; ADDENDUM Missed adding these files. 2020-02-04 16:40:53 -08:00
Mark Robert Miller 299b6bebc5
HBASE-23783: Address tests writing and reading SSL/Security files in a common location. (#1116)
This is causing me issues with parallel test runs.

Also allow setting the surefire reports and temp directories via command line.

Signed-off-by: stack <stack@apache.org>
2020-02-04 14:38:56 -08:00
Duo Zhang bb14bdad62 HBASE-23782 Addendum fix error prone warning 2020-02-04 10:14:27 +08:00
stack d797ed04e5 HBASE-23786 [Flakey Test] TestMasterNotCarryTable.testMasterMemStoreLAB 2020-02-03 17:34:22 -08:00
Jan Hentschel b591a3d205
HBASE-23772 Removed deprecated getTimeStampOfLastShippedOp from MetricsSource
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-04 00:25:58 +01:00
Michael Stack 661abeb730 HBASE-23780 Edit of test classifications (#1109)
These classifications come of running at various fork counts.. A test
may complete quick if low fork count but if it is accessing disk, it
will run much slower if fork count is high. This edit accommodates
some of this phenomenon.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-02-03 10:28:19 -08:00
Duo Zhang ab0039974d
HBASE-23782 We still reference the hard coded meta descriptor in some places when listing table descriptors (#1115)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-03 14:15:57 +08:00
Jan Hentschel 12f1c7c1ce
HBASE-23777 Removed deprecated createTableDescriptor(String, int, int, int, KeepDeletedCells) from HBaseTestingUtility
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-02-03 00:24:19 +01:00
Jan Hentschel 5a3ad6f08f
HBASE-23776 Removed deprecated method createLocalHTU() from HBaseTestingUtility
Signed-off-by: stack <stack@apache.org>
2020-02-02 00:28:25 +01:00
Bharath Vissapragada d9453203ae HBASE-23764: Switch to IP address for ZK ensemble
Tests showed that using IP address is much faster for mini cluster
and a lot less flakier. See the jira for more details.
2020-02-01 14:57:51 -08:00
stack ba9f026d69 HBASE-23770 [Flakey Tests] TestRegionReplicasWithRestartScenarios#testWhenRestart 2020-01-31 08:39:46 -08:00
stack 1a6bdc7c50 HBASE-23771 [Flakey Tests] Test TestSplitTransactionOnCluster Again 2020-01-31 08:35:58 -08:00
abhinaba.sarkar 77229c79e3 HBASE-23350 Make compaction files cacheonWrite configurable based on threshold
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
2020-01-31 11:35:11 +05:30
Josh Elser 8b00f9f0b1 HBASE-17115 Define UI admins via an ACL
The Hadoop AccessControlList allows us to specify admins of the webUI
via a list of users and/or groups. Admins of the WebUI can mutate the
system, potentially seeing sensitive data or modifying the system.

hbase.security.authentication.spnego.admin.users is a comma-separated
list of users who are admins.
hbase.security.authentication.spnego.admin.groups is a comma-separated
list of groups whose membership are admins. Either of these
configuration properties may also contain an asterisk (*) which denotes
"any entity" (e.g user, group).

Previously, when a user was denied from some endpoint that was
designated for admins, they received an HTTP/401. In this case, it is
more correct to return HTTP/403 as they were correctly authenticated,
but they were disallowed from fetching the given resource. This commit
incorporates this change.

hbase.security.authentication.ui.config.protected also exists for users
who have sensitive information stored in the Hadoop service
configuration and want to limit access to this endpoint. By default,
the Hadoop configuration endpoint is not protected and any
authenticated user can access it.

The test is based off of work by Nihal Jain in HBASE-20472.

Co-authored-by: Nihal Jain <nihaljain.cs@gmail.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-01-29 16:36:55 -05:00
Viraj Jasani 66d198d35f
HBASE-23749 : TestHFileWriterV3 for all DataBlock Encoders
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
2020-01-29 23:54:18 +05:30
Jan Hentschel 60b80b6298
HBASE-23736 Removed deprecated getTimeStampOfLastAppliedOp from MetricsSink
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Xu Cang <xucang@apache.org>
2020-01-29 17:02:09 +01:00
stack 98cff8a26a HBASE-23753 Update of errorprone generated failures
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>

hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
hbase-server/src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java
 Complains about mismatch in types when Compare. Implement Compare in
 base Interface.

hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
 Complains pbs never return null.

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSinkManager.java
 Needed redo because errorprone complains can't mock Service from guava.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicasWithRestartScenarios.java
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController.java
 Unrelated...adding one-liner debug statements chasing other test
 failures.
2020-01-28 18:54:35 -08:00
stack 92a4a1d7b7 HBASE-23746 [Flakey Tests] Caused by: org.apache.hadoop.hbase.util.CommonFSUtils$StreamLacksCapabilityException: hflush and hsync 2020-01-28 09:45:08 -08:00
Jan Hentschel e836f5684f
HBASE-23738 Removed deprecated createLocalHRegion from HBaseTestingUtility
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-01-28 12:40:00 +01:00
stack 375e01dac4 HBASE-23737 [Flakey Tests] TestFavoredNodeTableImport fails 30% of the time
Second part of this issue -- changes to the test.
2020-01-26 10:41:42 -08:00
stack d82c3a5ebd HBASE-23737 [Flakey Tests] TestFavoredNodeTableImport fails 30% of the time; AMENDMENT
This is actual fix; previous added debug to test.
2020-01-26 10:31:59 -08:00
stack 1690414263 HBASE-23735 [Flakey Tests] TestClusterRestartFailover & TestClusterRestartFailoverSplitWithoutZk 2020-01-24 18:21:33 -08:00
Michael Stack 753cc99814 HBASE-23726 Forward-port HBASE-21345 to branch-2.2, 2.3 & master as well.
HBASE-21345 - [hbck2] Allow version check to proceed even though master is 'initializing'.
Just remove the check state from the getClusterStatus call.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Sakthi <sakthi@apache.org>
(cherry picked from commit dd8496a546)
2020-01-24 12:00:20 -08:00
stack 3c1bccb0f8 HBASE-23733 [Flakey Tests] TestSplitTransactionOnCluster 2020-01-24 11:25:02 -08:00
Geoffrey Jacoby 2d6bb81dff HBASE-23710 - Priority configuration for system coprocessors (#1077)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-01-23 16:23:28 -08:00
Michael Stack 373857854a
HBASE-21065 Try ROW_INDEX_V1 encoding on meta table (fix bloomfilters… (#1012)
Set encoding and blooms on meta as default. Also shutdown access to the
initial meta schema creating method; get from TableDescriptors if you
need access to schema or edit it as you would any other table if you
want to edit it.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-01-23 09:34:54 -08:00
Wellington Ramos Chevreuil d6ac8b34eb
HBASE-23715 MasterFileSystem should not create MasterProcWALs dir on … (#1078)
Signed-off-by: Josh Elser <elserj@apache.org>
2020-01-23 09:25:31 +00:00
Michael Stack 6cdc4b1f05
HBASE-23705 Add CellComparator to HFileContext (#1062)
Codecs don't have access to what CellComparator to use.  Backfill.

M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java
 Adds a new compareRows with default implementation that takes a ByteBuffer.
 Needed by the index in a block encoder implementation.

M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java
 Adds implementation for meta of new compareRows method. Adds utility
 method for figuring comparator based off tablename.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/AbstractDataBlockEncoder.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexCodecV1.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexSeekerV1.java
 Comparator is in context. Remove redundant handling.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
 Comparator is in context. Remove redundant handling. Clean javadoc.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDecodingContext.java
 Clean javadoc.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexEncoderV1.java
 Cache context so can use it to get comparator to use later.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java
 Cache cellcomparator to use. Javdoc on diff between HFileContext and
 HFileInfo.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java
 Add CellComparator

M hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java
 Remove comparator caching. Get from context instead.

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
 Skip a reflection if we can.

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileInfo.java
 Javadoc. Removed unused filed.

Signed-off-by: Anoop Sam John <anoopsamjohn@apacher.org>
Signed-off-by: Ramkrishna <ramkrishna.s.vasudevan@intel.com>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-01-22 20:00:21 -08:00
Geoffrey Jacoby bb56dfa974 HBASE-23711 - Add test for MinVersions and KeepDeletedCells TTL (#1079)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-01-22 00:07:15 -08:00
Michael Stack ba3463d9de HBASE-23055 Alter hbase:meta (#1043)
Make hbase:meta region schema dynamic.

Patch has been under development a good while and its focus has changed
a few times so its bloated with fixup from older versions.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
 Shut down access to internals and removed unused methods.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
 Cleanup/refactor section on replica-handling.

M hbase-server/src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java
 Get hbase:meta schema from filesystem rather than from hard-coding.
2020-01-21 15:02:40 -08:00
Peter Somogyi 00e64d83e8
HBASE-23347 Allow custom authentication methods for RPCs; addendum (#1060)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-01-21 18:47:13 +01:00
Duo Zhang 167892ce64
HBASE-23680 RegionProcedureStore missing cleaning of hfile archive (#1022)
Signed-off-by: stack <stack@apache.org>
2020-01-18 20:30:47 +08:00
Peter Somogyi 65bcf55892
HBASE-23653 Expose content of meta table in web ui; addendum (#1061)
Fix error prone problem

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-01-18 09:33:21 +01:00
Viraj Jasani 3b64ea50ec
HBASE-23569 : Validate that all default chores of HRegionServer are scheduled (ADDENDUM)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-01-16 22:46:28 -08:00
Nick Dimiduk 00fc46756a
HBASE-23653 Expose content of meta table in web ui (#1020)
Adds a display of the content of 'hbase:meta' to the Master's
table.jsp, when that table is selected. Supports basic pagination,
filtering, &c.

Signed-off-by: stack <stack@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-01-16 08:46:39 -08:00
Josh Elser 04d789f1a7 HBASE-23347 Allow custom authentication methods for RPCs
Decouple the HBase internals such that someone can implement
their own SASL-based authentication mechanism and plug it into
HBase RegionServers/Masters.

Comes with a design doc in dev-support/design-docs and an example in
hbase-examples known as "Shade" which uses a flat-password file
for authenticating users.

Closes #884

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
2020-01-16 10:27:28 -05:00
Duo Zhang 0321f56765
HBASE-23652 Move the unsupported procedure type check before migrating to RegionProcedureStore (#1018)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-01-16 22:58:16 +08:00
Wellington Ramos Chevreuil 19d3bed1d4 HBASE-23694 After RegionProcedureStore completes migration of WALProcedureStore, still running WALProcedureStore.syncThread keeps trying to delete now inexistent log files. (#1048)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-01-16 14:12:37 +08:00
dingbaosheng 278d9fd153 HBASE-23674 Too many rit page Numbers show confusion 2020-01-15 13:54:29 -08:00
WenFeiYi a44f3b50e4 HBASE-23677 fix 500/NPE of region.jsp (#1033)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-01-15 18:06:57 +00:00
Wellington Chevreuil ceaeece2c5 Revert "fix 500/NPE of region.jsp (#1033)"
Missed the jira number on that commit message. Will re-apply it with the jira number.

This reverts commit d60ce17c17.
2020-01-15 18:05:23 +00:00
Bharath Vissapragada fd05aabf02 HBASE-23665: Split unit tests from TestTableName into a separate test-only class. (#1032)
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-01-15 09:16:28 -08:00
Wellington Ramos Chevreuil cb78b103a7
HBASE-23683 Make HBaseInterClusterReplicationEndpoint more extensible (#1027)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: binlijin <binlijin@gmail.com>
2020-01-15 11:48:29 +00:00
WenFeiYi d60ce17c17 fix 500/NPE of region.jsp (#1033)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-01-15 11:32:27 +00:00
Viraj Jasani a075d6a011
HBASE-23569 : Validate that all default chores of HMaster are scheduled
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2020-01-14 23:03:16 -08:00
Michael Stack 8b7b097905 HBASE-23687 DEBUG logging cleanup (#1040)
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-01-14 22:07:55 -08:00
Nick Dimiduk 499ff32f00
HBASE-23383 [hbck2] `fixHoles` should queue assignment procedures for any regions its fixing (#917)
The current process for an operator, after fixing holes in meta, is to
manually disable and enable the whole table. Let's try to avoid
bringing the whole table offline if we can. Have the master attempt to
queue up assignment procedures for any new regions it creates.

Signed-off-by: stack <stack@apache.org>
2020-01-13 16:34:02 -08:00
Josh Elser 1d2c3efc69 HBASE-23679 FileSystem objects leak when cleaned up in cleanupBulkLoad
The cleanupBulkLoad method is only called for the first Region in the
table which was being bulk loaded into. This means that potentially N-1
other RegionServers (where N is the number of RegionServers) will leak
one FileSystem object into the FileSystem cache which will never be
cleaned up. We need to do this clean-up as a part of secureBulkLoadHFiles
otherwise we cannot guarantee that heap usage won't grow unbounded.

Closes #1029

Signed-off-by: Sean Busbey <busbey@apache.org>
2020-01-13 18:30:30 -05:00
Duo Zhang 4ad12e03b8
HBASE-23681 Add UT for procedure store region flusher (#1024)
Signed-off-by: stack <stack@apache.org>
2020-01-12 13:22:48 +08:00
stack c8496bc694 Revert "HBASE-23055 Alter hbase:meta"
This reverts commit 9abdb7b5ae.
2020-01-11 09:17:17 -08:00
Viraj Jasani 2e0edacf72
HBASE-23662 : Replace HColumnDescriptor(String cf) with ColumnFamilyDescriptor
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-01-10 20:42:21 -08:00
stack 4ed466966c Revert "HBASE-23668 Master log start filling with "Flush journal status" messages"
Minor addendum fixing log message.
2020-01-10 15:59:02 -08:00
stack 8ca614857d HBASE-23668 Master log start filling with "Flush journal status" messages"
This reverts commit fb9fa04da7.
i.e. reapplication of patch that was preamaturely applied.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-01-10 10:28:04 -08:00
Guanghao Zhang 56842d04d7
HBASE-23659 BaseLoadBalancer#wouldLowerAvailability should consider region replicas (#1001)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-01-10 15:02:53 +08:00
stack 9abdb7b5ae HBASE-23055 Alter hbase:meta
Make it so hbase:meta can be altered. TableState for hbase:meta
    was hardcoded ENABLED. Make it dynamic. State is now kept in
    current active Master. It is transient so falls back to default
    if Master crashes. Add to registry a getMetaTableState
    which reads mirrored state from zookeeper (NOT from Master and
    defaults ENABLED if no implementation or error fetching state).
    hbase:meta schema will be bootstrapped from the filesystem.
    Changes to filesystem schema are atomic so we should be ok if
    Master fails mid-edit (TBD). Undoes a bunch of guards that
    prevented our being able to edit hbase:meta.

    TODO: Tests, more clarity around hbase:meta table state, and undoing
    references to hard-coded hbase:meta regioninfo.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
     Throw illegal access exception if you try to use MetaTableAccessor
     getting state of the hbase:meta table.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
     Add fetching of hbase:meta table state from registry. Adds cache of
     tablestates w/ a ttl of 1 second (adjustable).

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RawAsyncHBaseAdmin.java
      Add querying registry for hbase:meta table state.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ZKAsyncRegistry.java
      Add querying of mirrored table state for hbase:meta table.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/zookeeper/ZNodePaths.java
     Shutdown access.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/TableDescriptors.java
     Just cleanup.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/TableStateManager.java
     Add state holder for hbase:meta. Removed unused methods.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
     Shut down access.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DisableTableProcedure.java
     Allow hbase:meta to be disabled.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/EnableTableProcedure.java
     Allow hbase:meta to be enabled.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-01-09 16:25:16 -08:00
belugabehr f3bdb02280 HBASE-23378: Clean Up FSUtil setClusterId (#910)
Signed-off-by: stack <stack@apache.org>
2020-01-09 12:47:12 -08:00
Aman Poonia 12b79a3907 HBASE-22285 A normalizer which merges small size regions with adjacent regions (#978)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-01-09 11:49:48 -08:00
Guanghao Zhang fbf660c96b
HBASE-23658 Fix flaky TestSnapshotFromMaster (#998)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-01-09 14:38:52 +08:00
binlijin 9a2a799c96
HBASE-23660 hbase:meta's table.jsp ref to wrong rs address (#999)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-01-09 10:25:54 +08:00
binlijin 66d1258aa3
HBASE-23651 Region balance throttling can be disabled (#991)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2020-01-08 19:01:22 +08:00
binlijin 33f45d4413
HBASE-23636 Disable table may hang when regionserver stop or abort. (#982)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: virajjasani <34790606+virajjasani@users.noreply.github.com>
2020-01-06 15:19:10 +08:00
Viraj Jasani cd870790b1
HBASE-23588 : Cache index & bloom blocks on write (ADDENDUM)
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2020-01-06 11:49:03 +05:30
stack ae7f3ebf1e HBASE-23369 Auto-close 'unknown' Regions reported as OPEN on RegionServers; ADDENDUM on master 2020-01-03 15:42:55 -08:00
stack d75a7001b1 HBASE-23369 Auto-close 'unknown' Regions reported as OPEN on RegionServers
Master force-closes unknown/incorrect Regions OPEN on RS

M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
 Added a note and small refactor.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
 Fix an NPE when CJ ran.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
 Minor clean up of log message; make it clearer.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
 Make it so closeRegionSilentlyAndWait can be used w/o timeout.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
 If a RegionServer Report notes a Region is OPEN and the Master does not
 know of said Region, close it (We used to crash out the RegionServer)

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateNode.java
 Minor tweak of toString -- label should be state, not rit (confusing).

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
 Doc.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/TransitRegionStateProcedure.java
 Add region name to exception.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/HBCKServerCrashProcedure.java
 Be more careful about which Regions we queue up for reassign. This
 procedure is run by the operator so could happen at any time. We
 will likely be running this when Master has some accounting of
 cluster members so check its answers for what Regions were on
 server before running.

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 Doc and we were misrepresenting the case where a Region as not in RIT
 when we got CLOSE -- we were reporting it as though it was already
 trying to CLOSE.
2020-01-03 10:25:18 -08:00
Duo Zhang 85ecdd10a9
HBASE-23587 The FSYNC_WAL flag does not work on branch-2.x (#974)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-01-03 16:57:49 +08:00
Yiran Wu ccfbdadb0f HBASE-23098 [bulkload] If one of the peers in a cluster is configured with NAMESPACE level, its hfile-refs(zk) will be backlogged (#676)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 Signed-off-by: stack <stack@apache.org>
2020-01-02 21:10:41 -08:00
Michael Stack abcb1ee81b HBASE-23632 DeadServer cleanup (#979)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-01-02 20:25:18 -08:00
stack db7fb063d1 HBASE-23596 HBCKServerCrashProcedure can double assign
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>

Change its behavior so it will only look in hbase:meta
if the call to the super class turns up zero references.
Only then will it search hbase:meta for references to
'Unknown Servers'. Normal operation where we read Master
context is usual and sufficient. The scan of hbase:meta
is only for case where Master state has been corrupted
and we need to clear out 'Unknown Servers'.
2020-01-02 14:33:54 -08:00
belugabehr 280b94434c HBASE-23628: Remove Apache Commons Digest Base64 (#977)
Signed-off-by: stack <stack@apache.org>
2020-01-02 12:06:19 -08:00
Viraj Jasani 275cdc6052
HBASE-23588 : Cache index & bloom blocks on write if CacheCompactedBlocksOnWrite is enabled
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
Signed-off-by: chenxu14 <47170471+chenxu14@users.noreply.github.com>
2020-01-02 20:33:59 +05:30
Duo Zhang 4cb952ce34
HBASE-23624 Add a tool to dump the procedure info in HFile (#975)
Signed-off-by: stack <stack@apache.org>
2020-01-02 11:23:55 +08:00
Viraj Jasani e32dbe8ed2
HBASE-23590 : Update maxStoreFileRefCount to maxCompactedStoreFileRefCount for auto region recovery based on old reader references
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2020-01-01 22:16:54 +05:30
binlijin 06eff551c3
HBASE-23615 Use a dedicated thread for executing WorkerMonitor in Pro… (#961)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: virajjasani <34790606+virajjasani@users.noreply.github.com>
2019-12-31 10:03:01 +08:00
Duo Zhang c6a9a4d410 HBASE-23618 Addendum add main method 2019-12-29 22:12:31 +08:00
Duo Zhang bf775628dd
HBASE-23618 Add a tool to dump procedure info in the WAL file (#969)
Signed-off-by: stack <stack@apache.org>
2019-12-29 21:43:50 +08:00
Duo Zhang 0ba84d8e95
HBASE-23617 Add a stress test tool for region based procedure store (#962)
Signed-off-by: stack <stack@apache.org>
2019-12-27 22:28:12 +08:00
Guanghao Zhang def9ac7c45
HBASE-23286 Improve MTTR: Split WAL to HFile (#820)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-12-27 15:59:23 +08:00
Bharath Vissapragada 94346d8623 HBASE-23238: Remove 'static'ness of cell counter in LimitKVsReturnFilter (addendum) (#963)
Having it as static means the test cannot be parameterized (ran into
this issue in HBASE-23305). That happens because the field is not
reset between parameterized runs.
2019-12-26 11:10:39 -08:00
chenxu14 56f9db98d1 HBASE-23374 ExclusiveMemHFileBlock’s allocator should not be hardcoded as ByteBuffAllocator.HEAP
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2019-12-25 21:38:02 +01:00
Duo Zhang 1b049a2d34
HBASE-23326 Implement a ProcedureStore which stores procedures in a HRegion (#941)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: stack <stack@apache.org>
2019-12-25 12:02:12 +08:00
binlijin 9ff2507303
HBASE-23613 ProcedureExecutor check StuckWorkers blocked by DeadServe… (#960)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-12-25 09:56:48 +08:00
BukrosSzabolcs ee19008b12 HBASE-23589: FlushDescriptor contains non-matching family/output combinations (#949)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Lijin Bin <binlijin@apache.org>
2019-12-24 19:37:06 +08:00
Toshihiro Suzuki 00bb0f20d4 HBASE-23581 Creating table gets stuck when specifying an invalid split policy as METADATA (#942)
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apacher.org>
Signed-off-by: Xu Cang <xucang@apache.org>
2019-12-24 19:16:31 +08:00
stack 4b6ce0fcbf HBASE-23572 In 'HBCK Report', distringush between live, dead, and unknown servers
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-12-20 10:54:28 -08:00
binlijin ab40b9648b
HBASE-23564 RegionStates may has some expired serverinfo and make regions do not balance. (#930)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2019-12-20 15:00:27 +08:00
Duo Zhang f4860d8d2f
HBASE-23594 Procedure stuck due to region happen to recorded on two servers. (#953)
Signed-off-by: stack <stack@apache.org>
2019-12-20 13:40:39 +08:00
Duo Zhang 80ba354e2e
HBASE-20461 Implement fsync for AsyncFSWAL (#947)
Signed-off-by: stack <stack@apache.org>
2019-12-19 13:30:15 +08:00
Sean Busbey bc2f162749 HBASE-23239 Reporting on status of backing MOB files from client-facing cells (#785)
* Adds a new MapReduce job that builds a report of health of mob files
* Also builds background information on mob system use
* add a basic mob architecture in the ref guide to explain how mob takes the reference hfile value and finds the actual cell contents
* add a troubleshooting mob section to the ref guide to explain how to do a mob reference scan.

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-12-16 12:48:17 -06:00
Bharath Vissapragada b38446c017 HBASE-23575 Remove dead code in AsyncRegistry (addendum)
Additions to MiniHBaseCluster needed for branch-2.

Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Viraj Jasani <virajjasani007@gmail.com>
2019-12-16 10:32:44 -06:00
Ankit Singhal 29d1a97482 HBASE-23065 [hbtop] Top-N heavy hitter user and client drill downs
Signed-off-by: Toshihiro Suzuki <brfrn169@gmail.com>
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2019-12-16 23:44:05 +09:00
ramkrish86 eee9480cb4
HBASE-23066 Allow cache on write during compactions when prefetching … (#935)
* HBASE-23066 Allow cache on write during compactions when prefetching is enabled

* Fix checkstyle issues - recommit
2019-12-16 15:20:30 +05:30
David Mollitor 3be8ae2167 HBASE-23380 General cleanup of FSUtil (#912)
* Clean up JavaDocs
* Clean up logging and error messages
* Remove superfluous code
* Replace static code with library call
* Do not swallow Interrupted Exceptions
* Use try-with-resources
* User multi-Exception catches to reduce code size

Signed-off-by: Jan Hentschel <janh@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-12-13 16:32:18 -06:00
Bharath Vissapragada efa4fe901a HBASE-23575 Remove dead code in AsyncRegistry (#929)
Removes a bunch of dead code and fixes some checkstyle nits.

Signed-off-by: Viraj Jasani <virajjasani007@gmail.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-12-13 13:26:03 -06:00
meiyi 85a081925b HBASE-23555 TestQuotaThrottle is broken (#924) 2019-12-11 12:23:13 -08:00
stack b0233a294f HBASE-23554 Encoded regionname to regionname utility (#923); ADDENDUM 2019-12-11 11:02:59 -08:00
Michael Stack 8742265d85 HBASE-23554 Encoded regionname to regionname utility (#923)
Adds shell command regioninfo:

  hbase(main):001:0>  regioninfo '0e6aa5c19ae2b2627649dc7708ce27d0'
  {ENCODED => 0e6aa5c19ae2b2627649dc7708ce27d0, NAME => 'TestTable,,1575941375972.0e6aa5c19ae2b2627649dc7708ce27d0.', STARTKEY => '', ENDKEY => '00000000000000000000299441'}
  Took 0.4737 seconds

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-12-11 08:57:59 -08:00
Sean Busbey ec9bd20ec8 Revert "HBASE-23066 Allow cache on write during compactions when prefetching is (#919)"
TestCacheOnWrite failing all the time.

This reverts commit d561130e82.
2019-12-11 10:17:46 -06:00
meiyi fc816ac1b1
HBASE-23553 Snapshot referenced data files are deleted in some case (#922) 2019-12-11 09:31:11 +08:00
belugabehr c39339c004 HBASE-23379 Clean Up FSUtil getRegionLocalityMappingFromFS
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2019-12-10 21:56:53 +01:00
ramkrish86 d561130e82
HBASE-23066 Allow cache on write during compactions when prefetching is (#919)
Contributed by : Jacob LeBlanc
2019-12-10 22:52:59 +05:30
binlijin f87428e704
HBASE-23377 Balancer should skip disabled tables's regions (#908)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: GuangxuCheng  <guangxucheng@gmail.com>
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
2019-12-10 19:28:42 +08:00
binlijin dff7fffe85
HBASE-22280 Separate read/write handler for priority request(especial… (#202)
Signed-off-by: Yu Li <liyu@apache.org>
2019-12-09 16:11:35 +08:00
Sandeep Pal 60d9430195 HBASE-23309: Adding the flexibility to ChainWalEntryFilter to filter the whole entry if all cells get filtered (#837)
HBASE-23309: Adding the flexibility to ChainWalEntryFilter to filter the whole entry if all cells get filtered
2019-12-06 15:41:09 -08:00
Lars Hofhansl 9c82a65b29 HBASE-23364 HRegionServer sometimes does not shut down. 2019-12-05 18:25:41 -08:00
Wellington Ramos Chevreuil 6d251efa04
HBASE-23367 Remove unused constructor from WALPrettyPrinter (#901)
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: Xu Cang <xucang@apache.org>
2019-12-05 10:29:36 +00:00
Karthik Palanisamy 3e21bcf6d4 HBASE-23361 Limit two decimals in total average load (#897)
Signed-off-by: GuangxuCheng  <guangxucheng@gmail.com>
Signed-off-by: Xu Cang <xucang@apache.org>
2019-12-05 10:19:06 +08:00
binlijin bc8f1210ff
HBASE-23365 Minor change MemStoreFlusher's log (#900)
Signed-off-by: GuangxuCheng  <guangxucheng@gmail.com>
Signed-off-by: Xu Cang <xucang@apache.org>
2019-12-05 10:12:34 +08:00
Junegunn Choi 5e34a59d90 HBASE-22529 Add sanity check for in-memory compaction policy
Signed-off-by: Toshihiro Suzuki <brfrn169@gmail.com>
2019-12-04 21:01:22 +09:00
Bharath Vissapragada 570cc3c4a4 HBASE-23362: [WalPrettyPrinter] print/filter by table name. (#898)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2019-12-04 10:03:33 +00:00
binlijin 580d65ee4d
HBASE-23356 When construct StoreScanner throw exceptions it is possible to left some KeyValueScanner not closed. (#891)
Signed-off-by: GuangxuCheng  <guangxucheng@gmail.com>
2019-12-04 10:34:07 +08:00
Toshihiro Suzuki 0f166edc3d
HBASE-22096 /storeFile.jsp shows CorruptHFileException when the storeFile is a reference file (#888)
Signed-off-by: Lijin Bin <binlijin@apache.org>
2019-12-03 15:15:40 +09:00
XinSun 27cfe1bb27 HBASE-23345 Table need to replication unless all of cfs are excluded (#881)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2019-12-03 13:35:23 +08:00
Viraj Jasani ad607a1d76 HBASE-23335 : Improving cost functions array copy in StochasticLoadBalancer (#874) 2019-12-01 21:42:27 -08:00
Guanghao Zhang f7839f5eb6
HBASE-23298 Refactor LogRecoveredEditsOutputSink and BoundedLogWriterCreationOutputSink (#832)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-12-02 13:30:47 +08:00
Wellington Ramos Chevreuil 636fa2c6b0
HBASE-23313 [hbck2] setRegionState should update Master in-memory sta… (#864)
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: stack <stack@apache.org>
2019-11-27 08:41:23 +00:00
Sandeep Pal 0d7a6b9725 HBASE-23117: Bad enum in hbase:meta info:state column can fail loadMeta and stop startup (#867)
* Handling the BAD value in info:state columns in hbase:meta

* Adding a unit test and region encoded name in the LOG

* Adding a null check for region state to complete the test scenario and fixing the nit

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: GuangxuCheng  <guangxucheng@gmail.com>
Signed-off-by: stack <stack@apache.org>
2019-11-26 10:11:02 -08:00
chenxu14 17468efb88 HBASE-23293 [REPLICATION] make ship edits timeout configurable (#825)
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2019-11-26 19:05:53 +08:00
Wellington Ramos Chevreuil b08697ae4a
HBASE-23197 'IllegalArgumentException: Wrong FS' on edits replay when… (#740)
Signed-off-by: Josh Elser <elserj@apache.org>
2019-11-23 15:03:56 +00:00
Peter Somogyi ee730c8c79 HBASE-22969 A new binary component comparator(BinaryComponentComparator) to perform comparison of arbitrary length and position; ADDENDUM (#869)
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-11-22 13:42:27 -06:00
Michael Stack 8e52339cb8 HBASE-23307 Add running of ReplicationBarrierCleaner to hbck2 fixMeta invocation (#859)
Signed-off-by: Lijin Bin <binlijin@apache.org>
2019-11-22 08:35:55 -08:00
xuqinya1 ca6059f4db HBASE-23325 [UI]rsgoup average load keep two decimals (#860)
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
2019-11-22 14:29:45 +08:00
Karthik Palanisamy 0b49e3a174 HBASE-23237 Prevent Negative values in metrics requestsPerSecond
Closes #834

Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2019-11-21 22:13:58 -05:00
bsglz 4ea792246f HBASE-23223 Support the offsetLock of bucketCache to use strong ref (#764)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2019-11-21 08:39:58 +00:00
stack 77b4e8c972 HBASE-23322 [hbck2] Simplification on HBCKSCP scheduling
Signed-off-by: Lijin Bin <binlijin@apache.org>
2019-11-20 21:27:04 -08:00
Bharath Vissapragada 834ccb4bf6 HBASE-23259: Populate master address end points in cluster/rs configs (#807)
All the clients need to know the master RPC end points while using master
based registry for creating cluster connections. This patch amends the
test cluster utility to populate these configs in the base configuration
object used to spin up the cluster.

The config key added here ("hbase.master.addrs") is used in the subsequent
patches for HBASE-18095.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2019-11-20 10:15:43 -08:00
stack bcd2aa9a9e HBASE-23321 [hbck2] fixHoles of fixMeta doesn't update in-memory state 2019-11-20 09:06:58 -08:00
belugabehr 33bedf8d4d HBASE-23308: Review of NullPointerExceptions (#836)
Signed-off-by: stack <stack@apache.org>
2019-11-19 13:51:11 -08:00
stack ca6e67a6de HBASE-23315 Miscellaneous HBCK Report page cleanup
* Add a bit of javadoc around SerialReplicationChecker.
 * Miniscule edit to the profiler jsp page and then a bit of doc on how to make it work that might help.
 * Add some detail if NPE getting BitSetNode to help w/ debug.
 * Change HbckChore to log region names instead of encoded names; helps doing diagnostics; can take region name and query in shell to find out all about the region according to hbase:meta.
 * Add some fix-it help inline in the HBCK Report page – how to fix.
 * Add counts in procedures page so can see if making progress; move listing of WALs to end of the page.
2019-11-19 07:34:24 -08:00
Baiqiang Zhao e83bb205f4 HBASE-23278 Add a table-level compaction progress display on the UI (#816)
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
2019-11-19 16:21:36 +08:00
stack f8f9a26cb3 HBASE-23282 HBCKServerCrashProcedure for 'Unknown Servers'
Have the existing scheduleRecoveries launch a new HBCKSCP
instead of SCP. It gets regions to recover from Master
in-memory context AND from a scan of hbase:meta. This
new HBCKSCP is For processing 'Unknown Servers', servers that
are 'dead' and purged but still have references in
hbase:meta. Rare occurance but needs tooling to address.
Later have catalogjanitor take care of these deviations
between Master in-memory and hbase:meta content (usually
because of overdriven cluster with failed RPCs to hbase:meta,
etc)

Changed expireServers in ServerManager so could pass in
custom reaction to expired server.... This is how we
run our custom HBCKSCP while keeping all other aspects
of expiring services (rather than try replicate it
externally).
2019-11-18 14:28:55 -08:00
Udai Bhan Kashyap b5e8727617 HBASE-22969 A new binary component comparator(BinaryComponentComparator) to perform comparison of arbitrary length and position (#829)
Signed-off-by: Balazs Meszaros <meszibalu@apache.org>
2019-11-18 13:28:06 +01:00
belugabehr a3efa5911d HBASE-23102: Improper Usage of Map putIfAbsent (#828)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2019-11-17 10:13:52 +00:00
Duo Zhang e60764f5a1
HBASE-23294 ReplicationBarrierCleaner should delete all the barriers for a removed region which does not belong to any serial replication peer (#827)
Signed-off-by: stack <stack@apache.org>
2019-11-15 18:59:51 +08:00
Geoffrey Jacoby 77490f815a HBASE-23251 - Add Column Family and Table Names to HFileContext and use in HFileWriterImpl logging (#796)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Zheng Hu <openinx@gmail.com>
2019-11-11 16:18:43 +08:00
chenxu14 34661050c8 HBASE-22888 Share some stuffs with the initial reader when new stream reader created (#581)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-11-08 13:54:53 +08:00
Bharath Vissapragada c2e01f2398 HBASE-23257: Track clusterID in stand by masters (#798)
This patch implements a simple cache that all the masters
can lookup to serve cluster ID to clients. Active HMaster
is still responsible for creating it but all the masters
will read it from fs to serve clients.

RPCs exposing it will come in a separate patch as a part of
HBASE-18095.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Guangxu Cheng <guangxucheng@gmail.com>
2019-11-08 10:32:44 +08:00
Karthik Palanisamy 264d3e3b16 HBASE-23263 NPE in Quotas.jsp (#800)
Signed-off-by: Guangxu Cheng <guangxucheng@gmail.com>
2019-11-08 09:55:57 +08:00
Karthik Palanisamy c348e642d9 HBASE-23262 Cannot load Master UI (#799)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-11-08 09:49:35 +08:00
lijinbin 02f264321b HBASE-22480 Get block from BlockCache once and return this block to BlockCache twice make ref count error. 2019-11-08 09:34:58 +08:00
Nick Dimiduk 12c19a6e51
HBASE-23230 Enforce member visibility in HRegionServer (#775)
* Clean up a bunch of private variable leakage into other
   classes. Reduces visibility as much as possible, providing getters
   where access remains necessary or making use of getters that
   already exist. There remains an insidious relationship between
   `HRegionServer` and `RSRpcServices`.
 * Rename `fs` to `dataFs`, `rootDir` as `dataRootDir` so as to
   distinguish from the new `walFs`, `walRootDir` (and make it easier
   to spot bugs).
 * Cleanup or delete a bunch of lack-luster javadoc comments.
 * Delete a handful of methods that are unused according to static
   analysis.
 * Reduces the warning count as reported by IntelliJ from 100 to 7.

Signed-off-by: stack <stack@apache.org>
2019-11-07 15:27:02 -08:00
ravowlga123 56e63b106c HBASE-23250 Log message about CleanerChore delegate initialization should be at INFO
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2019-11-05 20:21:16 +01:00
Viraj Jasani 10cc64a7d6 HBASE-23212 : Dynamically reload configs for Region Recovery chore (#773)
* HBASE-23212 : Dynamically reload configs for Region Recovery chore

* remove redundant volatile
2019-11-04 11:36:38 -08:00
Michael Stack 2f8860b1da HBASE-23243 [pv2] Filter out SUCCESS procedures; on decent-sized cluster, plethora overwhelms problems (#790)
Signed-off-by: GuangxuCheng  <guangxucheng@gmail.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-11-04 09:18:09 -08:00
Michael Stack bbfc73789f HBASE-23247 [hbck2] Schedule SCPs for 'Unknown Servers' (#791)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-11-04 09:01:56 -08:00
binlijin 90007b7e80
HBASE-23241 TestExecutorService sometimes fail (#782)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2019-11-04 17:15:50 +08:00
stack 3ccfd50bd9 HBASE-23244 NPEs running Canary (#784)
Addendum to fix findbugs complaint.
2019-11-02 09:35:15 -07:00
Michael Stack c58e80fbe6 HBASE-23244 NPEs running Canary (#784)
Signed-off-by: Viraj Jasani <virajjasani007@gmail.com>
2019-11-01 21:41:22 -07:00
Wellington Chevreuil b8a4504a26 HBASE-23238 Additional test and checks for null references on ScannerCallableWithReplicas (#780)
Signed-off-by: Sean Busbey <busbey@apache.org>
(cherry picked from commit 577db5d7e5 - Test only, removed changes from no more exisiting ScannerCallableWithReplicas class)
2019-11-01 12:15:41 +00:00
Michael Stack e1b4a2a895 HBASE-23221 Polish the WAL interface after HBASE-23181 (#774)
Removes the closeRegion flag added by HBASE-23181 and instead
relies on reading meta WALEdit content. Modified how qualifier is
written when the meta WALEdit is for a RegionEventDescriptor
so the 'type' is added to the qualifer so can figure type
w/o having to deserialize protobuf value content: e.g.
HBASE::REGION_EVENT::REGION_CLOSE

Added doc on WALEdit and tried to formalize the 'meta' WALEdit
type and how it works. Needs complete redo in part as suggested
by HBASE-8457. Meantime, some doc and cleanup.

Also changed the LogRoller constructor to remove redundant param.
Because of constructor change, need to change also
TestFailedAppendAndSync, TestWALLockup, TestAsyncFSWAL &
WALPerformanceEvaluation.java

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Lijin Bin <binlijin@apache.org>
2019-10-31 18:52:04 +08:00
chenxu14 255be63e1d HBASE-23184 The HeapAllocation in WebUI is not accurate (#730)
Signed-off-by: stack <stack@apache.org>
2019-10-31 17:53:24 +08:00
Duo Zhang 2451c2c532 HBASE-23231 Addendum remove redundant metrics.clear in ReplicationSource
This is only a problem on master which was introduced when we merge sync replication feature branch back
2019-10-31 09:40:43 +08:00
binlijin 3152d9981b HBASE-23231 ReplicationSource do not update metrics after refresh (#778)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-10-31 09:23:59 +08:00
Ankit Singhal 0108e57309 HBASE-23175 Yarn unable to acquire delegation token for HBase Spark jobs 2019-10-30 14:49:04 -07:00
binlijin e808ea1d04 HBASE-22739 ArrayIndexOutOfBoundsException when balance (#729)
Signed-off-by: stack <stack@apache.org>
2019-10-30 13:28:01 -07:00
Michael Stack 54425bf87b
HBASE-23192 CatalogJanitor consistencyCheck does not log problematic row on exception (#734)
Adds logging of row and complaint if consistency check fails during CJ
checking. Adds a few more null checks. Does edit on the 'HBCK Report'
top line.

Signed-off-by: Reid Chan <reidchan@apache.org>
2019-10-30 13:10:19 -07:00
Karthik Palanisamy 0a4c0b2dd6 HBASE-23191 EOFE log spam (#733)
Convert log message added for 2.2.0 from INFO to DEBUG.

Signed-off-by: stack <stack@apache.org>
2019-10-30 12:49:04 -07:00
Ankit Singhal 41bec4b8b1 HBASE-20827 Use backoff on CallQueueTooBigException when reporting region state transition
Signed-off-by: Josh Elser <elserj@apache.org>
2019-10-30 11:32:43 -04:00
Viraj Jasani 7b4dd9587b Revert "HBASE-23194 Remove unused methods from TokenUtil (#737)"
This reverts commit d7b90b3199.
2019-10-29 10:51:32 +01:00
binlijin 9a80618765
HBASE-23187 Update parent region state to SPLIT in meta (#732)
* HBASE-23187 Update parent region state to SPLIT in meta
2019-10-29 11:35:46 +08:00
Karthik Palanisamy 02370911b0 HBASE-23199 Error populating Table-Attribute fields (#741)
Signed-off-by: GuangxuCheng  <guangxucheng@gmail.com>
2019-10-29 11:31:25 +08:00
Sean Busbey a881678ef4 HBASE-23222 MOB compaction supportability improvements
* better logging on MOB compaction process
* HFileCleanerDelegate to optionally halt removal of mob hfiles
* use archiving when removing committed mob file after bulkload ref failure

closes #763

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Balazs Meszaros <meszibalu@apache.org>
2019-10-28 08:59:01 -05:00
Duo Zhang 7b5cd0152f
HBASE-23181 Blocked WAL archive: "LogRoller: Failed to schedule flush of XXXX, because it is not online on us" (#753)
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: stack <stack@apache.org>
2019-10-26 20:37:37 +08:00
Viraj Jasani d7b90b3199 HBASE-23194 Remove unused methods from TokenUtil (#737)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-10-25 10:22:06 +02:00
Karthik Palanisamy 8f92a14cd1 HBASE-23203 NPE in RSGroup info (#747)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-10-25 08:43:34 +08:00
Pierre Zemb 42d535a57a HBASE-23073 Add an optional costFunction to balance regions according to a capacity rule (#677)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2019-10-24 08:46:34 +01:00
Andrew Purtell b14ba58c5a
HBASE-23207 Log a region open journal (#751)
Signed-off-by: Abhishek Singh Chouhan <achouhan@apache.org>
2019-10-23 17:17:06 -07:00
Ankit Singhal 59fc163e94 HBASE-15519 Add per-user metrics with lossy counting
Introducing property hbase.regionserver.user.metrics.enabled(Default:true)
to disable user metrics in case it accounts for any performance issues

Close #661

Signed-off-by: Josh Elser <elserj@apache.org>
2019-10-23 11:54:31 -04:00
Viraj Jasani 14dcf1d0c6 HBASE-22460 : Reopen regions with very high Store Ref Counts (#600)
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
2019-10-23 21:16:47 +05:30
Viraj Jasani 2ad62b0162 HBASE-22679 : Revamping CellUtil (#735)
* HBASE-22679 : Revamping CellUtil

* checkstyle fix

* incorporating review

* minor indentation
2019-10-21 23:04:21 -07:00
Caroline Zhou af1910ccf3 HBASE-23172 HBase Canary region success count metrics reflect column family successes, not region successes 2019-10-21 09:02:21 -07:00
meiyi da9a53ed4d
HBASE-23042 Parameters are incorrect in procedures jsp (#728) 2019-10-21 10:09:05 +08:00