Apache
/
hbase
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,535 Commits 95 Branches 866 Tags 550 MiB
Commit Graph

8286 Commits

Author SHA1 Message Date
stack 7a1fda667e Revert "HBASE-25068 Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space" 
This reverts commit b5a242f42a.
 2020-09-22 20:51:13 -07:00
stack b5a242f42a HBASE-25068 Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space 
Pass WALFactory to Replication instead of WALProvider. WALFactory has all
WALProviders in it, not just the user-space WALProvider. Do this so
ReplicationService has access to all WALProviders in the Server (To be
exploited by the follow-on patch in HBASE-25055)

Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-09-22 12:07:00 -07:00
stack 97979436f8 HBASE-25067 Edit of log messages around async WAL Replication; checkstyle fixes; and a bugfix 
Editing logging around region replicas: shortening and adding context.
Checkstyle fixes in edited files while I was in there.
Bug fix in AssignRegionHandler -- was using M_RS_CLOSE_META to open
a Region instead of a M_RS_OPEN_META.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-09-22 10:13:03 -07:00
stack cef1220590
HBASE-24896 'Stuck' in static initialization creating RegionInfo instance 
Closes #2423

Untangle RegionInfo, RegionInfoBuilder, and MutableRegionInfo static
initializations some. Move MutableRegionInfo from inner-class of
RegionInfoBuilder to be (package private) standalone. Undo static
initializing references from RI to RIB.

Co-authored-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-09-20 14:11:55 +05:30
Mohammad Arshad bc9921627a HBASE-25069: Display region name instead of encoded region name in HBCK report page. (#2428) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-09-20 09:02:13 +08:00
Joseph295 de2f538bc9 HBASE-24991 Replace MovedRegionsCleaner with guava cache (#2357) 
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-09-19 14:58:21 +08:00
Hyeran Lee 7adf80fa6c HBASE-25057: Fix typo memeber (#2414) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
 2020-09-19 14:39:50 +08:00
shahrs87 7fe07e90eb
[HBASE-24956] ConnectionManager#locateRegionInMeta waits for user region lock indefinitely. (#2322) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-Off-By: Andrew Purtell <apurtell@apache.org>
Signed-off by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-09-17 09:26:35 -07:00
Viraj Jasani 4316dc738c
HBASE-24528 : BalancerDecision queue implementation in HMaster with Admin API (#2411) 
* Admin API getLogEntries() for ring buffer use-cases: so far, provides balancerDecision and slowLogResponse
* Refactor RPC call for similar use-cases
* Single RPC API getLogEntries() for both Master.proto and Admin.proto

Closes #2261

Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2020-09-17 18:51:42 +05:30
Joseph295 dacedb9d07 HBASE-25047 WAL split edits number is negative in RegionServerUI (#2408) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-09-17 09:39:22 +08:00
Duo Zhang 7a3bb8aefe HBASE-25037 Lots of thread pool are changed to non daemon after HBASE-24750 which causes trouble when shutting down (#2407) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-09-16 22:03:42 +08:00
XinSun d15074981f
HBASE-25012 HBASE-24359 causes replication missed log of some RemoteException (#2383) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-09-16 18:00:35 +08:00
Mohammad Arshad 5a610cb055 HBASE-25009: Hbck chore logs wrong message when loading regions from RS (#2379) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-09-16 09:08:53 +08:00
Duo Zhang b0c6305686 HBASE-25022 Remove 'hbase.testing.nocluster' config (#2394) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-09-14 21:26:30 +08:00
Clara Xiong 4087743745
HBASE-25006 Make the cost functions optional for StochastoicBalancer 
Closes #2386

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-09-13 14:20:56 +05:30
Sandeep Pal a0e25c4dc3
HBASE-24974: Provide a flexibility to print only row key and filter for multiple tables in the WALPrettyPrinter (#2387) (#2388) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2020-09-12 12:49:10 -07:00
Bharath Vissapragada a2979db27d
Revert "HBASE-24974: Provide a flexibility to print only row key and filter for multiple tables in the WALPrettyPrinter (#2387)" 
This reverts commit b39199f43b.
 2020-09-11 13:06:17 -07:00
Sandeep Pal b39199f43b
HBASE-24974: Provide a flexibility to print only row key and filter for multiple tables in the WALPrettyPrinter (#2387) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2020-09-11 12:44:34 -07:00
Javier Akira Luca de Tena cd66d8cba5
HBASE-24994 Add hedgedReadOpsInCurThread metric (#2367) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-09-11 13:49:11 +08:00
ankitjain64 7df1b92528
HBASE-24764: Add support of adding default peer configs via hbase-site.xml for all replication peers. (#2284) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2020-09-10 10:10:58 -07:00
WenFeiYi aa3e9dedf0 HBASE-24958 CompactingMemStore.timeOfOldestEdit error update (#2321) 2020-09-10 09:48:09 -07:00
Viraj Jasani 5018c0b380
HBASE-25004 : Log RegionTooBusyException details (#2371) 
Signed-off-by: mnpoonia <apoonia@salesforce.com>
Signed-off-by: stack <stack@apache.org>
 2020-09-10 21:26:49 +05:30
GeorryHuang 4b815d77d5 HBASE-24857 Fix several problems when starting webUI (#2344) 2020-09-09 21:47:51 -07:00
Duo Zhang 4421012105 HBASE-25005 Refactor CatalogJanitor (#2373) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-09-10 12:15:02 +08:00
niuyulin 6a05eaf7d5 HBASE-24993 Remove OfflineMetaRebuildTestCore (#2359) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-09-09 12:29:48 +08:00
Mohammad Arshad 672bd1b3a8 HBASE-24995: MetaFixer fails to fix overlaps when multiple tables have overlaps (#2361) 
Signed-off-by: stack <stack@apache.org>
 2020-09-08 12:01:54 -07:00
Viraj Jasani 5a327eef92
HBASE-24979 : Client operation timeout test for batch requests 
Closes #2347

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-09-08 19:33:36 +05:30
Toshihiro Suzuki daccdb19a2
HBASE-24602 Add Increment and Append support to CheckAndMutate (#2363) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-09-08 21:32:35 +09:00
Mohammad Arshad 17eeaef6d9
HBASE-24940: runCatalogJanitor() API should return -1 to indicate already running status 
Closes #2331

Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-09-03 15:57:04 +05:30
Qi Yu 0c290bfa0b HBASE-24973 Remove read point parameter in method StoreFlush#performFlush and StoreFlush#createScanner (#2337) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-09-02 18:04:32 +08:00
Duo Zhang 70c6205bc2
HBASE-24964 addendum fix TestMetaTableMetrics (#2340) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-09-02 10:10:36 +08:00
Andrew Purtell ea4c7aad68 HBASE-24971 Upgrade JQuery to 3.5.1 (#2336) 
- Update embedded minimized jquery to 3.5.1
- Update embedded minimized jquery.tablesorter to 2.31.3

Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-09-01 10:40:18 -07:00
Zheng Wang 192be4a2f6
HBASE-24569 Get hostAndWeights in addition using localhost if it is null in local mode 
Closes #1909

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-09-01 18:18:00 +05:30
Viraj Jasani 859bf3ea9a
HBASE-24968 : Move META_COMPARATOR to subclass MetaCellComparator (#2332) 
* Break subclass referencing of MetaCellComparator from superclass CellComparatorImpl
  static initializer by moving META_COMPARATOR to subclass MetaCellComparator

Closes #2329

Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-08-31 19:51:00 +05:30
XinSun caeaf79f27 HBASE-24913 Refactor TestJMXConnectorServer (#2286) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-08-31 10:24:33 +08:00
Mohammad Arshad 48854d4c6d HBASE-24916: Region hole contains wrong regions pair when hole is cre… (#2304) 
Signed-off-by: stack <stack@apache.org>
 2020-08-30 10:06:14 -07:00
Zheng Wang a765d859f9
HBASE-24894 [Flakey Test] TestStochasticLoadBalancer.testMoveCostMultiplier 
Closes #2270

Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-30 20:22:11 +05:30
Duo Zhang e022094400 HBASE-24964 Remove MetaTableAccessor.tableExists (#2330) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-30 19:57:08 +08:00
Duo Zhang 54454f8de6 HBASE-24949 Optimize FSTableDescriptors.get to not always go to fs when cache miss (#2317) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-08-29 22:50:39 +08:00
bsglz 08b0b85901
HBASE-24898 Addendum to deal with Daylight Saving Time (#2318) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-08-27 11:18:01 +08:00
WenFeiYi 3a85bdd482 HBASE-24892 config 'hbase.hregion.memstore.mslab.indexchunksize' not be used (#2265) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-08-27 08:53:37 +08:00
Bharath Vissapragada bb9121da77
HBASE-24765: Dynamic master discovery (#2314) 
This patch adds the ability to discover newly added masters
dynamically on the master registry side. The trigger for the
re-fetch is either periodic (5 mins) or any registry RPC failure.
Master server information is cached in masters to avoid repeated
ZK lookups.

Updates the client side connection metrics to maintain a counter
per RPC type so that clients have visibility into counts grouped
by RPC method name.

I didn't add the method to ZK registry interface since there
is a design discussion going on in splittable meta doc. We can
add it later if needed.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 275a38e153)
 2020-08-26 09:13:34 -07:00
Duo Zhang 0be2b6ce55 HBASE-24945 Remove MetaTableAccessor.getRegionCount (#2303) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-25 21:33:15 +08:00
Duo Zhang e61a3460a7 HBASE-24942 MergeTableRegionsProcedure should not call clean merge region (#2301) 
Signed-off-by: stack <stack@apache.org>
 2020-08-25 15:15:46 +08:00
Duo Zhang 4455856e9c HBASE-23834 HBase fails to run on Hadoop 3.3.0/3.2.2/3.1.4 due to jetty version mismatch (#2222) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
 2020-08-25 15:02:55 +08:00
Michael Stack 4243536b19
HBASE-24885 STUCK RIT by hbck2 assigns (#2283) 
Adds region state check on hbck2 assigns/unassigns. Returns pid of -1
if in inappropriate state with logging explaination which suggests
passing override if operator wants to assign/unassign anyways. Here
is an example of what happens now if hbck2 tries an unassign and
Region already unassigned:

  2020-08-19 11:22:06,926 INFO  [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=50086] assignment.AssignmentManager(820): Failed {ENCODED => d1112e553991e938b6852f87774c91ee, NAME => 'TestHbck,zzzzz,1597861310769.d1112e553991e938b6852f87774c91ee.', STARTKEY => 'zzzzz', ENDKEY => ''} unassign, override=false; set override to by-pass state checks.
  org.apache.hadoop.hbase.client.DoNotRetryRegionException: Unexpected state for state=CLOSED, location=null, table=TestHbck, region=d1112e553991e938b6852f87774c91ee
          at org.apache.hadoop.hbase.master.assignment.AssignmentManager.preTransitCheck(AssignmentManager.java:583)
          at org.apache.hadoop.hbase.master.assignment.AssignmentManager.createOneUnassignProcedure(AssignmentManager.java:812)
          at org.apache.hadoop.hbase.master.MasterRpcServices.unassigns(MasterRpcServices.java:2616)
          at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$HbckService$2.callBlockingMethod(MasterProtos.java)
          at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:397)
          at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
          at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
          at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)

Previous it would just create the unassign anyways. Now must pass override
to queue the procedure regardless. Safer.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
 javadoc on assigns/unassigns. Minor refactor in assigns/unassigns to cater to
 case where procedure may come back null (if override not set and fails state checks).

hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
 checkstyle cleanups.
 Clarifying javadoc on how there is no state checking when bulk assigns creating/enabling
 tables.

 createOneAssignProcedure and createOneUnassignProcedure now handle exceptions which now
 can be thrown if no override and region state is not appropriate.

 Aggregation of createAssignProcedure and createUnassignProcedure instances adding in
 region state check invoked if override is NOT set.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateNode.java
 Change to setProcedure so it returns passed proc as result instead of void

Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-08-24 09:19:43 -07:00
XinSun 0e63b12648 HBASE-24871 Replication may loss data when refresh recovered replication sources (#2249) 
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Conflicts:
	hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
 2020-08-24 22:08:26 +08:00
Yechao Chen 836c04d356 HBASE-24811 Use class access static field or method (#2189) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
 2020-08-23 11:26:38 +08:00
Zheng Wang 34de350cb3
HBASE-24898 Use EnvironmentEdge.currentTime() instead of System.currentTimeMillis() in CurrentHourProvider 
Closes #2272

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-22 15:32:12 +05:30
Duo Zhang 3d6e64d248 HBASE-24926 Should call setFailure in MergeTableRegionsProcedure when isMergeable returns false (#2290) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-21 22:35:13 +08:00
Guanghao Zhang 4667a971b1 HBASE-24912 Enlarge MemstoreFlusherChore/CompactionChecker period for unit test (#2285) 
Signed-off-by: stack <stack@apache.org>
 2020-08-21 12:42:11 +08:00
Nick Dimiduk acfa08cd87 HBASE-24627 Normalize one table at a time 
Introduce an additional method to our Admin interface that allow an
operator to selectivly run the normalizer. The IPC protocol supports
general table name select via compound filter.

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-19 09:22:21 -07:00
Toshihiro Suzuki 00aa3bc9fc
HBASE-24884 BulkLoadHFilesTool/LoadIncrementalHFiles should accept -D options from command line parameters (#2275) 
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
 2020-08-19 19:55:34 +09:00
Nick Dimiduk 9c85f3641f HBASE-24583 Normalizer can't actually merge empty regions... 
when neighbor is larger than average size

* add `testMergeEmptyRegions` to explicitly cover different
  interleaving of 0-sized regions.
* fix bug where merging a 0-size region is skipped due to large
  neighbor.
* remove unused `splitPoint` from `SplitNormalizationPlan`.
* generate `toString`, `hashCode`, and `equals` methods from Apache
  Commons Lang3 template on `SplitNormalizationPlan` and
  `MergeNormalizationPlan`.
* simplify test to use equality matching over `*NormalizationPlan`
  instances as plain pojos.
* test make use of this handy `TableNameTestRule`.
* fix line-length issues in `TestSimpleRegionNormalizer`

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
Signed-off-by: Aman Poonia <aman.poonia.29@gmail.com>
 2020-08-12 10:11:40 -07:00
Viraj Jasani 8ccf643fdc
HBASE-24750 : All ExecutorService should use guava ThreadFactoryBuilder (#2214) 
Closes #2196

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Ted Yu <tyu@apache.org>
Signed-off-by: niuyulin <nyl353@163.com>
 2020-08-12 15:57:53 +05:30
Duo Zhang 96ea136a8e HBASE-24856 Fix error prone error in FlushTableSubprocedure (#2242) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
 2020-08-12 10:01:25 +08:00
Bharath Vissapragada 0533937b16
HBASE-24846: Address compaction races in TestFIFOCompactionPolicy (#2203) 
Looped through the test 100 times and it passes. Without the patch it fails
every ~10 runs or so.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
 2020-08-10 17:21:27 -07:00
Bharath Vissapragada cd97d1a452
Revert "HBASE-21905 (addendum): Address compaction races in TestFIFOCompactionPolicy (#2203)" 
This reverts commit c372ff19cf.

Reverting to apply as HBASE-24846.
 2020-08-10 17:21:05 -07:00
Nick Dimiduk 17a0c2aabf HBASE-24835 Normalizer should log a successful run at INFO level 
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-10 10:13:54 -07:00
Toshihiro Suzuki 22bf9a38c9
HBASE-24680 Refactor the checkAndMutate code on the server side (#2184) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
 2020-08-10 18:57:17 +09:00
bsglz ebf493f075
HBASE-24694 Support flush a single column family of table (#2218) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2020-08-10 10:36:13 +01:00
Duo Zhang ef7b9eb36e HBASE-23157 WAL unflushed seqId tracking may wrong when Durability.ASYNC_WAL is used (#762) 
Signed-off-by: stack <stack@apache.org>
 2020-08-10 10:54:30 +08:00
chenglei 457234c695 HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length(addendum) (#2055) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-08-10 10:04:05 +08:00
stack cee7431d0a HBASE-24830 Some tests involving RS crash fail with NullPointerException after HBASE-24632 in branch-2 2020-08-09 11:16:37 -07:00
Bo Cui 11cfb0f382 HBASE-21721 FSHLog : reduce write#syncs() times (#2217) 
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Anoop <anoopsamjohn@apache.org>
 2020-08-08 09:41:36 +05:30
Josh Elser 303db63b76 HBASE-24779 Report on the WAL edit buffer usage/limit for replication 
Closes #2193

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2020-08-07 14:33:30 -04:00
niuyulin e099aa5e20 HBASE-24659 Calculate FIXED_OVERHEAD automatically (#2018) 
Co-authored-by: niuyulin <niuyulin@xiaomi.com>

SIgned-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
 2020-08-06 14:52:20 -07:00
Bharath Vissapragada c372ff19cf
HBASE-21905 (addendum): Address compaction races in TestFIFOCompactionPolicy (#2203) 
Looped through the test 100 times and it passes. Without the patch it fails
every ~10 runs or so.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
 2020-08-06 09:40:32 -07:00
Michael Stack 9a564dc2bf
HBASE-24817 Allow configuring WALEntry filters on ReplicationSource (#2198) 
Allow specifying base WALEntry filter on construction of
ReplicationSource. Add means of being able to filter WALs by name.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 Add constructor that allows passing a predicate for filtering *in* WALs
 and a list of filters for filtering *out* WALEntries. The latter was
 hardcoded to filter out system-table WALEntries. The former did not
 exist but we'll need it if Replication takes in more than just the
 default Provider.

Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-06 09:29:08 -07:00
sguggilam 9f62e5645d
HBASE-24823 Port HBASE-22762 Print the delta between phases in the split/merge/compact/flush transaction journals 
Closes #2204

Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-06 18:51:38 +05:30
gkanade bfa558a0e3 HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751 
writer null check if not initialized yet during syncrunner run (#2201)

Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: AnoopSamJohn<anoopsamjohn@apache.org>
(cherry picked from commit b0863c5832)
 2020-08-05 23:41:13 -05:00
Sean Busbey ec39d59161 Revert "writer null check if not initialized yet during syncrunner run (#2201)" 
bad commit message

This reverts commit b0863c5832.
 2020-08-05 23:39:18 -05:00
Zheng Wang 64fe71dba3
HBASE-24821 simplify the logic of getRegionInfo in TestFlushFromClient to reduce redundancy code 
Closes #2200

Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-05 20:01:12 +05:30
gkanade b0863c5832
writer null check if not initialized yet during syncrunner run (#2201) 
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: AnoopSamJohn<anoopsamjohn@apache.org>
 2020-08-05 18:44:12 +05:30
Viraj Jasani 8979202c7a
HBASE-24795 : RegionMover to deal with unknown region while (un)loading 
* RegionMover to ignore move failures for split/merged regions with ack mode
* Refactor MoveWithAck and MoveWithoutAck as high level classes
* UT for RegionMover gracefully handling split/merged regions while loading regions and throwing failure while loading offline regions

Closes #2172

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Ted Yu <tyu@apache.org>
 2020-08-04 15:59:12 +05:30
Michael Stack 8c4f8e28e4
HBASE-24808 skip empty log cleaner delegate class names (WAS => cleaner.CleanerChore: Can NOT create CleanerDelegate= ClassNotFoundException) (#2181) 
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
 2020-08-03 13:15:19 -07:00
Michael Stack 86d2e37bc6
HBASE-24767 Change default to false for HBASE-15519 per-user metrics Set hbase.regionserver.user.metrics.enabled default to false; i.e. off. (#2132) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
 2020-08-03 11:36:19 -07:00
Zheng Wang 8b1c86a54d
HBASE-24704 Make Table Schema easier to view with multiple families 
Closes #2182

Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-02 23:39:09 +05:30
Sean Busbey 0806349ada
HBASE-24805 HBaseTestingUtility.getConnection should be threadsafe 
* refactor how we use connection to rely on the access method
* refactor initialization and cleanup of the shared connection
* incompatibly change HCTU's Configuration member variable to be final so it can be safely accessed from multiple threads.

Closes #2180

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 86ebbdd8a2)
 2020-08-02 00:51:03 -05:00
Anoop Sam John 86fccba0d0
HBASE-24695 FSHLog - close the current WAL file in a background thread. (#2183) 
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-08-01 22:46:32 +05:30
bsglz 1c45c8c7db
HBASE-24404 Support flush a single column family of region (#2098) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2020-07-31 09:31:14 +01:00
Sean Busbey cb179467fe HBASE-24794 hbase.rowlock.wait.duration should not be <= 0 (#2174) 
if hbase.rowlock.wait.duration is <=0 then log a message and treat it as a value of 1ms.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 840a55761b)
 2020-07-30 12:50:48 -05:00
Sandeep Pal 6c61c50df7
HBASE-24788: Fix the connection leaks on getting hbase admin from unclosed connection (#2173) 
Signed-off-by: Ted Yu <tyu@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2020-07-30 10:32:32 -07:00
ddupg 13f9f7a2c1
HBASE-24797 Move log code out of loop 
Closes #2175

Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-07-30 15:18:48 +05:30
Bharath Vissapragada 06236dbfcc
HBASE-20226: Parallelize region manifest deletes (#2159) 
We observed this delete call to be a bottleneck for table with lots of
regions. Patch attempts to parallelize them.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
(cherry picked from commit f07f30ae24)
 2020-07-29 11:08:19 -07:00
WenFeiYi bba70f08ea
HBASE-24752 NPE/500 accessing webui on master startup 
Closes #2148

Signed-off-by: Ted Yu <yuzhihong@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-07-29 20:24:22 +05:30
Michael Stack 20990b9c04
HBASE-24766 Document Remote Procedure Execution (#2131) 
Add outline of how the mechanism works generally.

Signed-off-by: Anoop Sam John anoopsamjohn@apache.org
Signed-off-by: Viraj Jasani vjasani@apache.org
Signed-off-by: Jan Hentschel jan.hentschel@ultratendency.com
 2020-07-29 07:44:01 -07:00
stack edb4cd534c HBASE-24632 Enable procedure-based log splitting as default in hbase3 Add deprecation of 'classic' zk-based WAL splitter. 
Also fix three bugs:

 * We were trying to delete non-empty directory; weren't doing
 accounting for meta WALs where meta had moved off the server
 (successfully)
 * We were deleting split WALs rather than archiving them.
 * We were not handling corrupt files.

Deprecations and removal of tests of old system.
 2020-07-27 08:23:29 -07:00
Wellington Ramos Chevreuil fce52fe6bb HBASE-24758 Avoid flooding replication source RSes logs when no sinks… (#2118) 
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>

(cherry picked from commit 8c0d7fa5b8)
 2020-07-27 12:43:57 +01:00
WenFeiYi 2bb76c0a51
HBASE-24665 (#2150) 
Co-authored-by: wen_yi <liu.wenwen@immomo.com>
 Signed-off-by: Anoop <anoopsamjohn@apache.org>
 Signed-off-by: Ramkrishna <ramkrishna@apache.org>
 Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-07-27 16:39:49 +05:30
Viraj Jasani 6cb51cc0f0
HBASE-24757 : ReplicationSink should limit row count in batch mutation based on hbase.rpc.rows.warning.threshold (#2136) 
Closes #2127

Signed-off-by: stack <stack@apache.org>
 2020-07-24 20:32:40 +05:30
Guanghao Zhang 70ab0dc324
HBASE-24743 Reject to add a peer which replicate to itself earlier (#2124) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2020-07-24 08:09:02 +08:00
Viraj Jasani ce4e692699
HBASE-24718 : Generic NamedQueue framework for multiple use-cases (Refactor SlowLog responses) (#2110) 
Closes #2052

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2020-07-22 11:36:19 +05:30
surbhi 33102a1265 HBASE-22146 Removing a namespace-level space quota does not remove policies against contained tables 
Closes #1935

Signed-off-by: Josh Elser <elserj@apache.org>
 2020-07-21 21:24:06 -04:00
Viraj Jasani 5bb76bf22d
Revert "HBASE-24743 Reject to add a peer which replicate to itself earlier (#2071)" 
This reverts commit 5db3ec2cfb.

TestReplicationAdmin and TestReplicationShell are broken on branch-2 and master respectively
 2020-07-21 20:20:51 +05:30
Mingliang Liu 3c91c33452
HBASE-24696 Include JVM information on Web UI under "Software Attributes" 
Closes #2087

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2020-07-21 16:55:44 +05:30
Sanjeet Nishad 70d1b35a4b
HBASE-24714 Error message is displayed in UI of table's compaction state if any region of that table is not open 
Closes #2047

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-07-21 12:35:08 +05:30
sguggilam 19d0f364ed
HBASE-24740 Enable journal logging for HBase snapshot operation (#2104) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
(cherry picked from commit 430602aed8)
 2020-07-20 18:56:37 -07:00
Guanghao Zhang 5db3ec2cfb HBASE-24743 Reject to add a peer which replicate to itself earlier (#2071) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2020-07-21 09:06:54 +08:00
stack 620470607e Revert "HBASE-24718 : Generic NamedQueue framework for multiple use-cases (Refactor SlowLog responses)" 
Causes TestAdminShell and TestThriftHBaseServiceHandler to fail 100% of the time in branch-2.

This reverts commit 8ae3480e70.
 2020-07-20 14:50:03 -07:00
Wellington Ramos Chevreuil 8c89582633 HBASE-24747 Log an ERROR if HBaseSaslRpcServer initialisation fails with an uncaught exception (#2079) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
(cherry picked from commit bd42c75cac)
 2020-07-20 10:29:04 +01:00
Zheng Wang 0a8a7fa2ed
HBASE-24709 Support MoveCostFunction use a lower multiplier in offpeak hours 
Closes #2099

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-07-19 20:11:20 +05:30
Michael Stack 552008c577 HBASE-24698 Turn OFF Canary WebUI as default (#2037) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2020-07-17 15:52:21 -07:00
Zheng Wang 4001d3f2d2
HBASE-24746 The sort icons overlap the col name in master UI 
Closes #2078

Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-07-17 21:51:49 +05:30
Viraj Jasani 8ae3480e70
HBASE-24718 : Generic NamedQueue framework for multiple use-cases (Refactor SlowLog responses) 
Closes #2052

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2020-07-17 19:57:05 +05:30
bsglz ec3beaf5a2
HBASE-24664 Some changing of split region by overall region size rather than only one store size (#2054) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2020-07-17 10:47:01 +01:00
bsglz bf368a01bc
HBASE-24382 Flush partial stores of region filtered by seqId when archive wal due to too many wals (#2049) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2020-07-17 10:40:50 +01:00
Lars 86f00e4749 HBASE-24742 Improve performance of SKIP vs SEEK logic. 2020-07-16 13:10:06 -07:00
huaxiangsun 0e45e377b8
HBASE-24705 MetaFixer#fixHoles() does not include the case for read replicas (i.e, replica regions are not created) (#2062) (#2068) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-07-14 22:38:01 -07:00
BukrosSzabolcs 7900afdb4f HBASE-24720: Meta replicas not cleaned when disabled (#2057) 
- make sure to always clean up excess meta replicas not just when their
number get decreased
- make sure NotServingRegionException is handled properly even when
wrapped
- add test

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
 2020-07-14 19:06:01 +02:00
WenFeiYi e1d1304cb1
HBASE-24578 [WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count 
Signed-off-by: Reid Chan <reidchan@apache.org>
 2020-07-14 10:39:19 +08:00
Geoffrey Jacoby a3567037c6
HBASE-23744 - FastPathBalancedQueueRpcExecutor should enforce queue length of 0 
Closes #1094

Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-07-13 12:19:02 +05:30
Zheng Wang fce1df9a3c
HBASE-24586 Add table level locality in table.jsp 
Closes #2048

Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-07-11 17:20:22 +05:30
chenglei deff4c88d3
HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length.(#2034) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2020-07-11 16:35:10 +08:00
Joseph295 ff2951e672 HBASE-24663 Add procedure process time statistics UI (#2035) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-07-10 15:05:57 +08:00
niuyulin eee467d119 HBASE-24653 Show snapshot owner on Master WebUI (#2016) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-07-10 15:05:48 +08:00
XinSun 5fcffae5db
HBASE-24431 RSGroupInfo add configuration map to store something extra (#2031) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-07-08 14:39:41 +08:00
Toshihiro Suzuki f771fd26b4
HBASE-24650 Change the return types of the new checkAndMutate methods introduced in HBASE-8458 (#2033) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-07-08 08:00:48 +09:00
Duo Zhang 28602d6fdb Revert "HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length. (#1970)" 
This reverts commit f834919929.
 2020-07-07 18:16:58 +08:00
WenFeiYi 5416cef27f HBASE-24546 CloneSnapshotProcedure unlimited retry (#1900) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-07-06 13:59:07 +08:00
chenglei f834919929 HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length. (#1970) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-07-04 21:12:07 +08:00
Baiqiang Zhao cb154d50aa
HBASE-24560 Add a new option of designatedfile in RegionMover 
Closes #1901

Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-07-01 20:28:35 +05:30
pengmq1 2a12fd283e HBASE-24289 Heterogeneous Storage for Date Tiered Compaction (#1730) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-06-30 15:19:30 +08:00
stack 0dd42f7916 HBASE-24648 Remove the legacy 'forceSplit' related code at region server side (#1990) 
Addendum to fix TestHeapSize
 2020-06-29 14:19:44 -07:00
Michael Stack 91da188d78
HBASE-24656 [Flakey Tests] branch-2 TestMasterNoCluster.testStopDuringStart (#1998) 2020-06-29 11:10:14 -07:00
Duo Zhang 3effd28a75 HBASE-24648 Remove the legacy 'forceSplit' related code at region server side (#1990) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-06-29 23:16:21 +08:00
Duo Zhang ed6104418c HBASE-24646 Set the log level for ScheduledChore to INFO in HBTU (#1987) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
 2020-06-28 23:27:42 +08:00
Viraj Jasani 0fe85947ec
HBASE-24588 : Submit task for NormalizationPlan (#1933) (#1983) 
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2020-06-28 01:10:42 +05:30
Duo Zhang 7c78356218 HBASE-24635 Split TestMetaWithReplicas (#1980) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-06-27 11:11:36 +08:00
huaxiangsun 2c201cc034
HBASE-24552 Replica region needs to check if primary region directory exists at file system in TransitRegionStateProcedure (#1924) (#1971) 
Signed-off-by:  stack <stack@apache.org>
 2020-06-26 11:27:30 -07:00
Nick Dimiduk a499eae1aa HBASE-24221 addendum to restore public interface on LoadIncrementalHFiles 
Signed-off-by: niuyulin <nyl353@163.com>
Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-06-26 09:06:32 -07:00
BukrosSzabolcs 50d1a79255 HBASE-24562: Stabilize master startup with meta replicas enabled (#1903) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
(cherry picked from commit 8cdb2cca44)
 2020-06-26 16:50:55 +01:00
Bharath Vissapragada a6c887012e
HBASE-24603: Make Zookeeper sync() call synchronous (#1945) (#1975) 
Writing a test for this is tricky. There is enough coverage for
functional tests. Only concern is performance, but there is enough
logging for it to detect timed out/badly performing sync calls.

Additionally, this patch decouples the ZK event processing into it's
own thread rather than doing it in the EventThread's context. That
avoids deadlocks and stalls of the event thread.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 84e246f9b1)
 2020-06-25 16:41:11 -07:00
Michael Stack e6639f9d4e
HBASE-24616 Remove BoundedRecoveredHFilesOutputSink dependency on a TableDescriptor (#1955) 
Purge query Master for table descriptors; make do w/ generic options.

Logging cleanup.

hbase-server/src/main/java/org/apache/hadoop/hbase/wal/BoundedRecoveredHFilesOutputSink.java
 Undo fetching Table Descriptor. Not reliably available at recovery time.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-06-25 11:45:06 -07:00
xincunSong a787d2e292 HBASE-24605 Break long region names in the web UI (#1942) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
 2020-06-23 10:06:31 +08:00
Viraj Jasani 4506f8d8ab
HBASE-24102 : Undo visibility change for RegionMover fields (ADDENDUM) 2020-06-22 22:48:54 +05:30
Viraj Jasani 92b6621fa5
HBASE-24594 : Separate resource allocation for TestReplicationStatusSink (#1934) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-06-21 16:15:31 +05:30
Viraj Jasani 48fda91c5b
HBASE-24380 : Provide WAL splitting journal logging (#1860) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>

Conflicts:
	hbase-server/src/main/java/org/apache/hadoop/hbase/wal/BoundedRecoveredEditsOutputSink.java
	hbase-server/src/main/java/org/apache/hadoop/hbase/wal/BoundedRecoveredHFilesOutputSink.java
	hbase-server/src/main/java/org/apache/hadoop/hbase/wal/RecoveredEditsOutputSink.java
 2020-06-19 16:27:48 -07:00
stack 7b171a3c1a HBASE-24574 Procedure V2 - Distributed WAL Splitting => LOGGING (#1912) 
Addendum; minor log edits
 2020-06-18 08:37:19 -07:00
Michael Stack 6eb93287cc
HBASE-24574 Procedure V2 - Distributed WAL Splitting => LOGGING (#1912) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
 Don't register as a chore on construction if no coordination state
 manager instance (there is no instance when procv2 WAL splitter).

hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitWALManager.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
 Edit logs.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionRemoteProcedureBase.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/SplitWALProcedure.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/SplitWALRemoteProcedure.java
 Add proc name rather than rely on default behavior. Add detail to the
 toString.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 Factoring

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
 Print the maxLogs... we don't do it any where.

hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WAL.java
 Utility method to strip prefix from wal path.

Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: tianjingyun <tianjy@apache.org>
 2020-06-18 08:32:48 -07:00
ramkrish86 ef809c198b
HBASE-24205 - Create metric to know the number of reads that happens (#1920) 
* HBASE-24205 - Create metric to know the number of reads that happens
from memstore (branch-2)

* Add the optimization as in master and fix whitestyle and checkstyle

* Fix compilation error that accidently crept in

Authored-by: Ramkrishna <ramkrishna@apache.org>
Signed-off by:Anoop Sam John<anoopsamjohn@gmail.com>
Signed-off by:Viraj Jasani<virajjasani@apache.org>
 2020-06-18 18:59:46 +05:30
Michael Stack 812d1e2bb5
HBASE-24577 Doc WALSplitter classes (#1913) 
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-06-17 14:59:30 -07:00
Viraj Jasani e202c98fb0
HBASE-24518 : waitForNamespaceOnline() should return false if any region is offline (#1869) 
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
 2020-06-17 20:05:15 +05:30
Bharath Vissapragada c49df36e85 HBASE-24564: Make RS abort call idempotent. 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 2567d15218)
 2020-06-16 08:43:06 -07:00
xincunSong 36ee96ca36 HBASE-24478 The regionInfo parameter for MasterProcedureScheduler#wai… (#1819) 
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
Signed-off-by: clarax <clarax98007@gmail.com>
 2020-06-16 17:35:17 +08:00
Sandeep Pal cb3d0d7d21
HBASE-24015: Test for Assign and Unassign of Regions on RegionServer on failure (#1898) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-06-16 12:54:02 +05:30
bsglz 035ea516e9
HBASE-24038 Add a metric to show the locality of ssd in table.jsp (#1902) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-06-15 19:28:07 +05:30
Duo Zhang 700c5a7e3b HBASE-24517 Addendum add more comments (#1882) 
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-06-15 10:44:30 +08:00
Toshihiro Suzuki 6ff6fe49a3
HBASE-8458 Support for batch version of checkAndMutate() (#1897) 
Signed-off-by: Josh Elser <elserj@apache.org>
 2020-06-14 13:55:06 +09:00
zhaoym6 2d57595f54 HBASE-23195 FSDataInputStreamWrapper unbuffer can NOT invoke the classes that NOT implements CanUnbuffer but its parents class implements CanUnbuffer 
Closes #746

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
 2020-06-12 13:27:53 -04:00
Michael Stack a4e1d073f4 HBASE-24545 Add backoff to SCP check on WAL split completion (#1891) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-06-12 07:58:02 -07:00
sguggilam d187ce2029
HBASE-24446 Use EnvironmentEdgeManager to compute clock skew in Master (#1885) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-06-12 16:52:15 +05:30
Toshihiro Suzuki 043a9e862f HBASE-24529 hbase.rs.evictblocksonclose is not honored when removing compacted files and closing the storefiles (#1881) 
Signed-off-by: Anoop Sam John <anoop.hbase@gmail.com>
 2020-06-12 19:02:20 +09:00
Anoop Sam John ed7dc9ed4c HBASE-24189 WALSplit recreates region dirs for deleted table with recovered edits data. 2020-06-11 21:28:32 +05:30
Viraj Jasani 1b1c269911
HBASE-24532 : Execute region plans with throttle should return succeeded plans (#1884) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2020-06-11 18:23:32 +05:30
sguggilam c02554024f
HBASE-24511 Ability to configure timeout between RPC retry to RS from master (#1861) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-06-10 13:04:47 +05:30
xincunSong dc2cf6ab5c HBASE-24441 CacheConfig details logged at Store open is not really u… (#1801) 
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: binlijin <binlijin@gmail.com>
 2020-06-09 14:19:15 +08:00
xincunSong 101023bcc1 HBASE-24468 Add region info when log meessages in HStore. (#1803) 
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: binlijin <binlijin@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-06-09 14:14:05 +08:00
huaxiangsun 02e7beaca1
HBASE-23202 ExportSnapshot (import) will fail if copying files to root directory takes longer than cleaner TTL (#1874) (#1875) 
Co-authored-by: Guangxu Cheng <guangxucheng@gmail.com>

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Zach York <zyork@apache.org>
 2020-06-08 22:51:48 -07:00
Duo Zhang c5dacfbbea HBASE-24117 Shutdown AssignmentManager before ProcedureExecutor may cause SCP to accidentally skip assigning a region (#1865) 
Signed-off-by: Michael Stack <stack@apache.org>
 2020-06-09 11:16:49 +08:00
Duo Zhang 6d96694a25 HBASE-24517 AssignmentManager.start should add meta region to ServerStateNode (#1866) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
 2020-06-09 08:27:17 +08:00
WenFeiYi 9ba3be61b3 HBASE-24444 Should shutdown mini cluster after class in TestMetaAssignmentWithStopMaster (#1802) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-06-09 08:21:52 +08:00
gkanade 82c50d72e2
HBASE-24208 Remove RS entry from zk draining servers node after RS been stopped (#1841) 
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-06-08 22:50:26 +05:30
XinSun 67fa367bb6 HBASE-24359 Optionally ignore edits for deleted CFs for replication (#1855) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-06-08 13:14:50 +08:00
Duo Zhang 108e23630a HBASE-24506 async client deadlock (#1858) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2020-06-08 09:57:33 +08:00
Toshihiro Suzuki 467efa573c HBASE-24515 batch Increment/Append fails when retrying the RPC 
Signed-off-by: Viraj Jasani <virajjasani007@gmail.com>
 2020-06-08 10:41:41 +09:00
bsglz 0c9806faec
HBASE-24496 The tag of Base Stats not actived by default in table.jsp (#1837) 
Signed-off-by: clarax
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-06-06 19:17:41 +05:30
Duo Zhang 72dadcd615 HBASE-24500 The behavior of RegionInfoBuilder.newBuilder(RegionInfo) is strange (#1850) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-06-05 15:24:39 +08:00
Duo Zhang 82b8feecef HBASE-24505 Reimplement Hbck.setRegionStateInMeta (#1852) 
Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
 2020-06-05 08:55:27 +08:00
Viraj Jasani cc4c28c575
HBASE-24456 : Create ImmutableScan and use it for CustomizedScanInfoBuilder (#1818) 
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-06-04 16:30:32 +05:30
Duo Zhang 4a84e28937 Revert "HBASE-24500 The behavior of RegionInfoBuilder.newBuilder(RegionInfo) is strange (#1840)" 
This reverts commit 8815215b3e.
 2020-06-04 12:17:15 +08:00
Duo Zhang 8815215b3e HBASE-24500 The behavior of RegionInfoBuilder.newBuilder(RegionInfo) is strange (#1840) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
 2020-06-04 11:18:25 +08:00
Bharath Vissapragada e59f39e09e
HBASE-21905: [FIFOCompaction] Don't compact already inflight store files (#1829) 
This one surfaced as a flake test but turns out to be a legit bug
in FIFOCompaction code. FifoCompaction does not check if an empty
store file is already being compacted by an in-flight compaction
request and still enqueues. It can potentially race with a running
compaction (as in this test case, see jira for the exact exception).

Fixes the edge case and cleans up the test code a bit.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2020-06-03 17:03:24 -07:00
Nick Dimiduk 4884773676 HBASE-24418 Consolidate Normalizer implementations 
Simplify our Normalizer story to have just a single, configurable
implementation.

* fold the features of `MergeNormalizer` into
  `SimpleRegionNormalizer`, removing the intermediate abstract class.
* configuration keys for merge-only features now share a common
  structure.
* add configuration to selectively disable normalizer split/merge
  operations.
* `RegionNormalizer` now extends `Configurable` instead of creating a
  new instance of `HBaseConfiguration` or snooping one off of other
  fields.
* avoid the extra RPCs by using `MasterServices` instead of
  `MasterRpcServices`.
* boost test coverage of all the various flags and feature
  combinations.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
 2020-06-03 16:43:40 -07:00
Wellington Ramos Chevreuil 11d093bc39 HBASE-21406 "status 'replication'" should not show SINK if the cluste… (#1761) 
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>

(Cherry picked from commit e5345b3a7c)
 2020-06-03 09:33:36 +01:00
Viraj Jasani b3c6af9500
HBASE-24492 : Remove infinite loop from ProtobufLogReader#readNext (#1831) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-06-03 13:44:19 +05:30
niuyulin 571ef7e811 HBASE-24412 Canary support check only one column family per RegionTask (#1817) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-06-03 09:16:11 +08:00
stack 9c8895cfbf HBASE-24474 Rename LocalRegion to MasterRegion (#1811) 
Addendum: HRegion changed size.
 2020-06-02 10:44:48 -07:00
Duo Zhang 6b43015fc9 HBASE-24474 Rename LocalRegion to MasterRegion (#1811) 
Signed-off-by: Michael Stack <stack@apache.org>
 2020-06-02 10:43:27 +08:00
Bharath Vissapragada a7a8afdf6a
HBASE-24479: Deflake TestCompaction#testStopStartCompaction (#1820) 
Polling of active compaction count is racy. Tightened the asserts
to be more reliable.

Signed-off-by: Reid Chan <reidchan@apache.org>
 2020-05-31 23:25:39 -07:00
Bharath Vissapragada d7f388518c
HBASE-24477: Move ConfigurationObserver and related classes to hbase-common 
This utility is useful for any module that wants to detect
dynamic config changes. Having it to hbase-common makes it
accessible to all the other modules.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-05-31 14:14:30 -07:00
bsglz 327d090e1b HBASE-24475 Clean up the master thread name getting in SplitLogManager and AssignmentManager (#1810) 2020-05-30 22:24:05 -07:00
jacob-leblanc 7b17e1cdaa HBASE-24454 - Read ioErrorStartTime to local temporary variable to avoid issue when it is set to -1 between greater than zero check and calculation of error duration (#1816) 
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
 2020-05-30 21:07:46 -07:00
Duo Zhang 4d5efec767 HBASE-24471 The way we bootstrap meta table is confusing (#1806) 
Signed-off-by: Michael Stack <stack@apache.org>
 2020-05-30 16:37:41 +08:00
Wellington Ramos Chevreuil 2dd160e4c7 HBASE-23716 MasterFileSystem should not create MasterProcWALs dir on … (#1078) 
Signed-off-by: Josh Elser <elserj@apache.org>
 2020-05-30 16:37:30 +08:00
Duo Zhang 1ac00e11cb Revert "HBASE-23716 MasterFileSystem should not create MasterProcWALs dir on … (#1078)" 
Wrong jira id

This reverts commit aca7f6fcf3.
 2020-05-30 16:37:19 +08:00
Viraj Jasani b2d24a1812
HBASE-23941 : FilterBy operator support in get_slowlog_responses API (#1793) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2020-05-30 13:57:32 +05:30
Wellington Ramos Chevreuil aca7f6fcf3 HBASE-23716 MasterFileSystem should not create MasterProcWALs dir on … (#1078) 
Signed-off-by: Josh Elser <elserj@apache.org>
 2020-05-30 16:11:32 +08:00
Michael Stack 6f1992f32f
HBASE-22287 inifinite retries on failed server in RSProcedureDispatcher (#1800) 
Adds backoff in place of retry every 100ms.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-05-29 10:00:53 -07:00
xincunSong 8edcbb4441
HBASE-24470 : Add store file info when log meessages in HStore. (#1804) 
Signed-off-by: binlijin <binlijin@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-05-29 19:01:12 +05:30
Viraj Jasani bce06627bb
HBASE-24428 : Fix for testSplitCompactWithPriority (ADDENDUM) 2020-05-29 18:24:12 +05:30
bsglz 22efe67758
HBASE-24451 Remove the HasThread because the related bug had been fixed since jdk7 (#1797) 
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-05-29 17:46:03 +05:30
Viraj Jasani 1e86ff09d7
HBASE-24428 : Update compaction priority for recently split daughter regions (#1784) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2020-05-27 20:58:57 +05:30
binlijin c94b946b31 HBASE-24371 Add more details when print CompactionConfiguration info (#1711) 
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
 2020-05-27 19:48:53 +08:00
Duo Zhang 441935a9d9 HBASE-24437 Addendum just start mini dfs cluster, and make the log roll more robust (#1787) 
Signed-off-by: stack <stack@apache.org>
 2020-05-27 11:26:35 +08:00
Michael Stack f8a5cc92af
HBASE-24425 Run hbck_chore_run and catalogjanitor_run on draw of 'HBCK Report' page (#1771) 
Signed-off-by: clarax
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
 2020-05-26 19:50:23 -07:00
Zheng Wang 1e16d385f8 HBASE-24423 No need to get lock in canSplit because hasReferences will get lock too 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2020-05-26 13:37:29 -07:00
stack 3ef44a4dcf HBASE-24437 Flaky test, TestLocalRegionOnTwoFileSystems#testFlushAndCompact 
Force another WAL roll just in case and make the check more loose (its
about whether the files are present, not their count)
 2020-05-26 11:55:09 -07:00
huaxiangsun 90bd19e55b
HBASE-24379 CatalogJanitor misreports region holes when there are actually over laps. (#1741) (#1777) 
Signed-off-by: stack <stack@apache.org>
 2020-05-26 10:06:13 -07:00
XinSun d285c6b681 HBASE-24416 RegionNormalizer spliting region should not be limited by hbase.normalizer.min.region.count (#1770) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-05-26 17:05:16 +08:00