Apache
/
hbase
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
17,691 Commits 95 Branches 866 Tags 550 MiB
Commit Graph

8393 Commits

Author SHA1 Message Date
Duo Zhang f973a8087f HBASE-25103 Remove ZNodePaths.metaReplicaZNodes (#2464) 
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
 2020-09-29 15:51:06 +08:00
Toshihiro Suzuki 4941fc02ea HBASE-25096 WAL size in RegionServer UI is wrong (#2456) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-09-28 14:49:04 +09:00
Mohammad Arshad bd2dd21e48 HBASE-25077: hbck.jsp page loading fails, logs NPE in master log. (#2433) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
 2020-09-27 14:06:29 +08:00
Joseph295 6b93373a0a
HBASE-25088 CatalogFamilyFormat/MetaTableAccessor.parseRegionInfoFromRegionName incorrectly setEndKey to regionId (#2462) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-09-27 11:05:16 +08:00
meiyi 0e6775c919 HBASE-25097 Wrong RIT page number in Master UI (#2458) 
Signed-off-by: Viraj Jasani<virajjasani@apache.org>
 2020-09-25 18:41:36 +08:00
Bharath Vissapragada 505ceacb4b
HBASE-25082: Per table WAL metrics: appendCount and appendSize (#2440) 
Signed-off-by: Geoffrey Jacoby <gjacoby@apache.org>
Signed-off-by: Ankit Jain <jain.ankit@salesforce.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 56c7505f8f)
 2020-09-23 21:07:44 -07:00
Andrew Purtell b2f2c79d8f HBASE-25079 Upgrade Bootstrap to 3.3.7 (#2442) 
Signed-off-by: Viraj Jasani<virajjasani@apache.org>
 2020-09-23 14:57:42 -07:00
stack 7a1fda667e Revert "HBASE-25068 Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space" 
This reverts commit b5a242f42a.
 2020-09-22 20:51:13 -07:00
stack b5a242f42a HBASE-25068 Pass WALFactory to Replication so it knows of all WALProviders, not just default/user-space 
Pass WALFactory to Replication instead of WALProvider. WALFactory has all
WALProviders in it, not just the user-space WALProvider. Do this so
ReplicationService has access to all WALProviders in the Server (To be
exploited by the follow-on patch in HBASE-25055)

Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-09-22 12:07:00 -07:00
stack 97979436f8 HBASE-25067 Edit of log messages around async WAL Replication; checkstyle fixes; and a bugfix 
Editing logging around region replicas: shortening and adding context.
Checkstyle fixes in edited files while I was in there.
Bug fix in AssignRegionHandler -- was using M_RS_CLOSE_META to open
a Region instead of a M_RS_OPEN_META.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-09-22 10:13:03 -07:00
stack cef1220590
HBASE-24896 'Stuck' in static initialization creating RegionInfo instance 
Closes #2423

Untangle RegionInfo, RegionInfoBuilder, and MutableRegionInfo static
initializations some. Move MutableRegionInfo from inner-class of
RegionInfoBuilder to be (package private) standalone. Undo static
initializing references from RI to RIB.

Co-authored-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-09-20 14:11:55 +05:30
Mohammad Arshad bc9921627a HBASE-25069: Display region name instead of encoded region name in HBCK report page. (#2428) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-09-20 09:02:13 +08:00
Joseph295 de2f538bc9 HBASE-24991 Replace MovedRegionsCleaner with guava cache (#2357) 
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-09-19 14:58:21 +08:00
Hyeran Lee 7adf80fa6c HBASE-25057: Fix typo memeber (#2414) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
 2020-09-19 14:39:50 +08:00
shahrs87 7fe07e90eb
[HBASE-24956] ConnectionManager#locateRegionInMeta waits for user region lock indefinitely. (#2322) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-Off-By: Andrew Purtell <apurtell@apache.org>
Signed-off by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-09-17 09:26:35 -07:00
Viraj Jasani 4316dc738c
HBASE-24528 : BalancerDecision queue implementation in HMaster with Admin API (#2411) 
* Admin API getLogEntries() for ring buffer use-cases: so far, provides balancerDecision and slowLogResponse
* Refactor RPC call for similar use-cases
* Single RPC API getLogEntries() for both Master.proto and Admin.proto

Closes #2261

Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2020-09-17 18:51:42 +05:30
Joseph295 dacedb9d07 HBASE-25047 WAL split edits number is negative in RegionServerUI (#2408) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-09-17 09:39:22 +08:00
Duo Zhang 7a3bb8aefe HBASE-25037 Lots of thread pool are changed to non daemon after HBASE-24750 which causes trouble when shutting down (#2407) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-09-16 22:03:42 +08:00
XinSun d15074981f
HBASE-25012 HBASE-24359 causes replication missed log of some RemoteException (#2383) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-09-16 18:00:35 +08:00
Mohammad Arshad 5a610cb055 HBASE-25009: Hbck chore logs wrong message when loading regions from RS (#2379) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-09-16 09:08:53 +08:00
Duo Zhang b0c6305686 HBASE-25022 Remove 'hbase.testing.nocluster' config (#2394) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-09-14 21:26:30 +08:00
Clara Xiong 4087743745
HBASE-25006 Make the cost functions optional for StochastoicBalancer 
Closes #2386

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-09-13 14:20:56 +05:30
Sandeep Pal a0e25c4dc3
HBASE-24974: Provide a flexibility to print only row key and filter for multiple tables in the WALPrettyPrinter (#2387) (#2388) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2020-09-12 12:49:10 -07:00
Bharath Vissapragada a2979db27d
Revert "HBASE-24974: Provide a flexibility to print only row key and filter for multiple tables in the WALPrettyPrinter (#2387)" 
This reverts commit b39199f43b.
 2020-09-11 13:06:17 -07:00
Sandeep Pal b39199f43b
HBASE-24974: Provide a flexibility to print only row key and filter for multiple tables in the WALPrettyPrinter (#2387) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2020-09-11 12:44:34 -07:00
Javier Akira Luca de Tena cd66d8cba5
HBASE-24994 Add hedgedReadOpsInCurThread metric (#2367) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-09-11 13:49:11 +08:00
ankitjain64 7df1b92528
HBASE-24764: Add support of adding default peer configs via hbase-site.xml for all replication peers. (#2284) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2020-09-10 10:10:58 -07:00
WenFeiYi aa3e9dedf0 HBASE-24958 CompactingMemStore.timeOfOldestEdit error update (#2321) 2020-09-10 09:48:09 -07:00
Viraj Jasani 5018c0b380
HBASE-25004 : Log RegionTooBusyException details (#2371) 
Signed-off-by: mnpoonia <apoonia@salesforce.com>
Signed-off-by: stack <stack@apache.org>
 2020-09-10 21:26:49 +05:30
GeorryHuang 4b815d77d5 HBASE-24857 Fix several problems when starting webUI (#2344) 2020-09-09 21:47:51 -07:00
Duo Zhang 4421012105 HBASE-25005 Refactor CatalogJanitor (#2373) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-09-10 12:15:02 +08:00
niuyulin 6a05eaf7d5 HBASE-24993 Remove OfflineMetaRebuildTestCore (#2359) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-09-09 12:29:48 +08:00
Mohammad Arshad 672bd1b3a8 HBASE-24995: MetaFixer fails to fix overlaps when multiple tables have overlaps (#2361) 
Signed-off-by: stack <stack@apache.org>
 2020-09-08 12:01:54 -07:00
Viraj Jasani 5a327eef92
HBASE-24979 : Client operation timeout test for batch requests 
Closes #2347

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-09-08 19:33:36 +05:30
Toshihiro Suzuki daccdb19a2
HBASE-24602 Add Increment and Append support to CheckAndMutate (#2363) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-09-08 21:32:35 +09:00
Mohammad Arshad 17eeaef6d9
HBASE-24940: runCatalogJanitor() API should return -1 to indicate already running status 
Closes #2331

Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-09-03 15:57:04 +05:30
Qi Yu 0c290bfa0b HBASE-24973 Remove read point parameter in method StoreFlush#performFlush and StoreFlush#createScanner (#2337) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-09-02 18:04:32 +08:00
Duo Zhang 70c6205bc2
HBASE-24964 addendum fix TestMetaTableMetrics (#2340) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-09-02 10:10:36 +08:00
Andrew Purtell ea4c7aad68 HBASE-24971 Upgrade JQuery to 3.5.1 (#2336) 
- Update embedded minimized jquery to 3.5.1
- Update embedded minimized jquery.tablesorter to 2.31.3

Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-09-01 10:40:18 -07:00
Zheng Wang 192be4a2f6
HBASE-24569 Get hostAndWeights in addition using localhost if it is null in local mode 
Closes #1909

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-09-01 18:18:00 +05:30
Viraj Jasani 859bf3ea9a
HBASE-24968 : Move META_COMPARATOR to subclass MetaCellComparator (#2332) 
* Break subclass referencing of MetaCellComparator from superclass CellComparatorImpl
  static initializer by moving META_COMPARATOR to subclass MetaCellComparator

Closes #2329

Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-08-31 19:51:00 +05:30
XinSun caeaf79f27 HBASE-24913 Refactor TestJMXConnectorServer (#2286) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-08-31 10:24:33 +08:00
Mohammad Arshad 48854d4c6d HBASE-24916: Region hole contains wrong regions pair when hole is cre… (#2304) 
Signed-off-by: stack <stack@apache.org>
 2020-08-30 10:06:14 -07:00
Zheng Wang a765d859f9
HBASE-24894 [Flakey Test] TestStochasticLoadBalancer.testMoveCostMultiplier 
Closes #2270

Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-30 20:22:11 +05:30
Duo Zhang e022094400 HBASE-24964 Remove MetaTableAccessor.tableExists (#2330) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-30 19:57:08 +08:00
Duo Zhang 54454f8de6 HBASE-24949 Optimize FSTableDescriptors.get to not always go to fs when cache miss (#2317) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-08-29 22:50:39 +08:00
bsglz 08b0b85901
HBASE-24898 Addendum to deal with Daylight Saving Time (#2318) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-08-27 11:18:01 +08:00
WenFeiYi 3a85bdd482 HBASE-24892 config 'hbase.hregion.memstore.mslab.indexchunksize' not be used (#2265) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-08-27 08:53:37 +08:00
Bharath Vissapragada bb9121da77
HBASE-24765: Dynamic master discovery (#2314) 
This patch adds the ability to discover newly added masters
dynamically on the master registry side. The trigger for the
re-fetch is either periodic (5 mins) or any registry RPC failure.
Master server information is cached in masters to avoid repeated
ZK lookups.

Updates the client side connection metrics to maintain a counter
per RPC type so that clients have visibility into counts grouped
by RPC method name.

I didn't add the method to ZK registry interface since there
is a design discussion going on in splittable meta doc. We can
add it later if needed.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 275a38e153)
 2020-08-26 09:13:34 -07:00
Duo Zhang 0be2b6ce55 HBASE-24945 Remove MetaTableAccessor.getRegionCount (#2303) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-25 21:33:15 +08:00
Duo Zhang e61a3460a7 HBASE-24942 MergeTableRegionsProcedure should not call clean merge region (#2301) 
Signed-off-by: stack <stack@apache.org>
 2020-08-25 15:15:46 +08:00
Duo Zhang 4455856e9c HBASE-23834 HBase fails to run on Hadoop 3.3.0/3.2.2/3.1.4 due to jetty version mismatch (#2222) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
 2020-08-25 15:02:55 +08:00
Michael Stack 4243536b19
HBASE-24885 STUCK RIT by hbck2 assigns (#2283) 
Adds region state check on hbck2 assigns/unassigns. Returns pid of -1
if in inappropriate state with logging explaination which suggests
passing override if operator wants to assign/unassign anyways. Here
is an example of what happens now if hbck2 tries an unassign and
Region already unassigned:

  2020-08-19 11:22:06,926 INFO  [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=50086] assignment.AssignmentManager(820): Failed {ENCODED => d1112e553991e938b6852f87774c91ee, NAME => 'TestHbck,zzzzz,1597861310769.d1112e553991e938b6852f87774c91ee.', STARTKEY => 'zzzzz', ENDKEY => ''} unassign, override=false; set override to by-pass state checks.
  org.apache.hadoop.hbase.client.DoNotRetryRegionException: Unexpected state for state=CLOSED, location=null, table=TestHbck, region=d1112e553991e938b6852f87774c91ee
          at org.apache.hadoop.hbase.master.assignment.AssignmentManager.preTransitCheck(AssignmentManager.java:583)
          at org.apache.hadoop.hbase.master.assignment.AssignmentManager.createOneUnassignProcedure(AssignmentManager.java:812)
          at org.apache.hadoop.hbase.master.MasterRpcServices.unassigns(MasterRpcServices.java:2616)
          at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$HbckService$2.callBlockingMethod(MasterProtos.java)
          at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:397)
          at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
          at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
          at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)

Previous it would just create the unassign anyways. Now must pass override
to queue the procedure regardless. Safer.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
 javadoc on assigns/unassigns. Minor refactor in assigns/unassigns to cater to
 case where procedure may come back null (if override not set and fails state checks).

hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
 checkstyle cleanups.
 Clarifying javadoc on how there is no state checking when bulk assigns creating/enabling
 tables.

 createOneAssignProcedure and createOneUnassignProcedure now handle exceptions which now
 can be thrown if no override and region state is not appropriate.

 Aggregation of createAssignProcedure and createUnassignProcedure instances adding in
 region state check invoked if override is NOT set.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateNode.java
 Change to setProcedure so it returns passed proc as result instead of void

Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-08-24 09:19:43 -07:00
XinSun 0e63b12648 HBASE-24871 Replication may loss data when refresh recovered replication sources (#2249) 
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Conflicts:
	hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
 2020-08-24 22:08:26 +08:00
Yechao Chen 836c04d356 HBASE-24811 Use class access static field or method (#2189) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
 2020-08-23 11:26:38 +08:00
Zheng Wang 34de350cb3
HBASE-24898 Use EnvironmentEdge.currentTime() instead of System.currentTimeMillis() in CurrentHourProvider 
Closes #2272

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-22 15:32:12 +05:30
Duo Zhang 3d6e64d248 HBASE-24926 Should call setFailure in MergeTableRegionsProcedure when isMergeable returns false (#2290) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-21 22:35:13 +08:00
Guanghao Zhang 4667a971b1 HBASE-24912 Enlarge MemstoreFlusherChore/CompactionChecker period for unit test (#2285) 
Signed-off-by: stack <stack@apache.org>
 2020-08-21 12:42:11 +08:00
Nick Dimiduk acfa08cd87 HBASE-24627 Normalize one table at a time 
Introduce an additional method to our Admin interface that allow an
operator to selectivly run the normalizer. The IPC protocol supports
general table name select via compound filter.

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-19 09:22:21 -07:00
Toshihiro Suzuki 00aa3bc9fc
HBASE-24884 BulkLoadHFilesTool/LoadIncrementalHFiles should accept -D options from command line parameters (#2275) 
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
 2020-08-19 19:55:34 +09:00
Nick Dimiduk 9c85f3641f HBASE-24583 Normalizer can't actually merge empty regions... 
when neighbor is larger than average size

* add `testMergeEmptyRegions` to explicitly cover different
  interleaving of 0-sized regions.
* fix bug where merging a 0-size region is skipped due to large
  neighbor.
* remove unused `splitPoint` from `SplitNormalizationPlan`.
* generate `toString`, `hashCode`, and `equals` methods from Apache
  Commons Lang3 template on `SplitNormalizationPlan` and
  `MergeNormalizationPlan`.
* simplify test to use equality matching over `*NormalizationPlan`
  instances as plain pojos.
* test make use of this handy `TableNameTestRule`.
* fix line-length issues in `TestSimpleRegionNormalizer`

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
Signed-off-by: Aman Poonia <aman.poonia.29@gmail.com>
 2020-08-12 10:11:40 -07:00
Viraj Jasani 8ccf643fdc
HBASE-24750 : All ExecutorService should use guava ThreadFactoryBuilder (#2214) 
Closes #2196

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Ted Yu <tyu@apache.org>
Signed-off-by: niuyulin <nyl353@163.com>
 2020-08-12 15:57:53 +05:30
Duo Zhang 96ea136a8e HBASE-24856 Fix error prone error in FlushTableSubprocedure (#2242) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
 2020-08-12 10:01:25 +08:00
Bharath Vissapragada 0533937b16
HBASE-24846: Address compaction races in TestFIFOCompactionPolicy (#2203) 
Looped through the test 100 times and it passes. Without the patch it fails
every ~10 runs or so.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
 2020-08-10 17:21:27 -07:00
Bharath Vissapragada cd97d1a452
Revert "HBASE-21905 (addendum): Address compaction races in TestFIFOCompactionPolicy (#2203)" 
This reverts commit c372ff19cf.

Reverting to apply as HBASE-24846.
 2020-08-10 17:21:05 -07:00
Nick Dimiduk 17a0c2aabf HBASE-24835 Normalizer should log a successful run at INFO level 
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-10 10:13:54 -07:00
Toshihiro Suzuki 22bf9a38c9
HBASE-24680 Refactor the checkAndMutate code on the server side (#2184) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
 2020-08-10 18:57:17 +09:00
bsglz ebf493f075
HBASE-24694 Support flush a single column family of table (#2218) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2020-08-10 10:36:13 +01:00
Duo Zhang ef7b9eb36e HBASE-23157 WAL unflushed seqId tracking may wrong when Durability.ASYNC_WAL is used (#762) 
Signed-off-by: stack <stack@apache.org>
 2020-08-10 10:54:30 +08:00
chenglei 457234c695 HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length(addendum) (#2055) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-08-10 10:04:05 +08:00
stack cee7431d0a HBASE-24830 Some tests involving RS crash fail with NullPointerException after HBASE-24632 in branch-2 2020-08-09 11:16:37 -07:00
Bo Cui 11cfb0f382 HBASE-21721 FSHLog : reduce write#syncs() times (#2217) 
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Anoop <anoopsamjohn@apache.org>
 2020-08-08 09:41:36 +05:30
Josh Elser 303db63b76 HBASE-24779 Report on the WAL edit buffer usage/limit for replication 
Closes #2193

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2020-08-07 14:33:30 -04:00
niuyulin e099aa5e20 HBASE-24659 Calculate FIXED_OVERHEAD automatically (#2018) 
Co-authored-by: niuyulin <niuyulin@xiaomi.com>

SIgned-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
 2020-08-06 14:52:20 -07:00
Bharath Vissapragada c372ff19cf
HBASE-21905 (addendum): Address compaction races in TestFIFOCompactionPolicy (#2203) 
Looped through the test 100 times and it passes. Without the patch it fails
every ~10 runs or so.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
 2020-08-06 09:40:32 -07:00
Michael Stack 9a564dc2bf
HBASE-24817 Allow configuring WALEntry filters on ReplicationSource (#2198) 
Allow specifying base WALEntry filter on construction of
ReplicationSource. Add means of being able to filter WALs by name.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 Add constructor that allows passing a predicate for filtering *in* WALs
 and a list of filters for filtering *out* WALEntries. The latter was
 hardcoded to filter out system-table WALEntries. The former did not
 exist but we'll need it if Replication takes in more than just the
 default Provider.

Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-06 09:29:08 -07:00
sguggilam 9f62e5645d
HBASE-24823 Port HBASE-22762 Print the delta between phases in the split/merge/compact/flush transaction journals 
Closes #2204

Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-06 18:51:38 +05:30
gkanade bfa558a0e3 HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751 
writer null check if not initialized yet during syncrunner run (#2201)

Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: AnoopSamJohn<anoopsamjohn@apache.org>
(cherry picked from commit b0863c5832)
 2020-08-05 23:41:13 -05:00
Sean Busbey ec39d59161 Revert "writer null check if not initialized yet during syncrunner run (#2201)" 
bad commit message

This reverts commit b0863c5832.
 2020-08-05 23:39:18 -05:00
Zheng Wang 64fe71dba3
HBASE-24821 simplify the logic of getRegionInfo in TestFlushFromClient to reduce redundancy code 
Closes #2200

Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-05 20:01:12 +05:30
gkanade b0863c5832
writer null check if not initialized yet during syncrunner run (#2201) 
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: AnoopSamJohn<anoopsamjohn@apache.org>
 2020-08-05 18:44:12 +05:30
Viraj Jasani 8979202c7a
HBASE-24795 : RegionMover to deal with unknown region while (un)loading 
* RegionMover to ignore move failures for split/merged regions with ack mode
* Refactor MoveWithAck and MoveWithoutAck as high level classes
* UT for RegionMover gracefully handling split/merged regions while loading regions and throwing failure while loading offline regions

Closes #2172

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Ted Yu <tyu@apache.org>
 2020-08-04 15:59:12 +05:30
Michael Stack 8c4f8e28e4
HBASE-24808 skip empty log cleaner delegate class names (WAS => cleaner.CleanerChore: Can NOT create CleanerDelegate= ClassNotFoundException) (#2181) 
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
 2020-08-03 13:15:19 -07:00
Michael Stack 86d2e37bc6
HBASE-24767 Change default to false for HBASE-15519 per-user metrics Set hbase.regionserver.user.metrics.enabled default to false; i.e. off. (#2132) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
 2020-08-03 11:36:19 -07:00
Zheng Wang 8b1c86a54d
HBASE-24704 Make Table Schema easier to view with multiple families 
Closes #2182

Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-08-02 23:39:09 +05:30
Sean Busbey 0806349ada
HBASE-24805 HBaseTestingUtility.getConnection should be threadsafe 
* refactor how we use connection to rely on the access method
* refactor initialization and cleanup of the shared connection
* incompatibly change HCTU's Configuration member variable to be final so it can be safely accessed from multiple threads.

Closes #2180

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 86ebbdd8a2)
 2020-08-02 00:51:03 -05:00
Anoop Sam John 86fccba0d0
HBASE-24695 FSHLog - close the current WAL file in a background thread. (#2183) 
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-08-01 22:46:32 +05:30
bsglz 1c45c8c7db
HBASE-24404 Support flush a single column family of region (#2098) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2020-07-31 09:31:14 +01:00
Sean Busbey cb179467fe HBASE-24794 hbase.rowlock.wait.duration should not be <= 0 (#2174) 
if hbase.rowlock.wait.duration is <=0 then log a message and treat it as a value of 1ms.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 840a55761b)
 2020-07-30 12:50:48 -05:00
Sandeep Pal 6c61c50df7
HBASE-24788: Fix the connection leaks on getting hbase admin from unclosed connection (#2173) 
Signed-off-by: Ted Yu <tyu@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2020-07-30 10:32:32 -07:00
ddupg 13f9f7a2c1
HBASE-24797 Move log code out of loop 
Closes #2175

Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-07-30 15:18:48 +05:30
Bharath Vissapragada 06236dbfcc
HBASE-20226: Parallelize region manifest deletes (#2159) 
We observed this delete call to be a bottleneck for table with lots of
regions. Patch attempts to parallelize them.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
(cherry picked from commit f07f30ae24)
 2020-07-29 11:08:19 -07:00
WenFeiYi bba70f08ea
HBASE-24752 NPE/500 accessing webui on master startup 
Closes #2148

Signed-off-by: Ted Yu <yuzhihong@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-07-29 20:24:22 +05:30
Michael Stack 20990b9c04
HBASE-24766 Document Remote Procedure Execution (#2131) 
Add outline of how the mechanism works generally.

Signed-off-by: Anoop Sam John anoopsamjohn@apache.org
Signed-off-by: Viraj Jasani vjasani@apache.org
Signed-off-by: Jan Hentschel jan.hentschel@ultratendency.com
 2020-07-29 07:44:01 -07:00
stack edb4cd534c HBASE-24632 Enable procedure-based log splitting as default in hbase3 Add deprecation of 'classic' zk-based WAL splitter. 
Also fix three bugs:

 * We were trying to delete non-empty directory; weren't doing
 accounting for meta WALs where meta had moved off the server
 (successfully)
 * We were deleting split WALs rather than archiving them.
 * We were not handling corrupt files.

Deprecations and removal of tests of old system.
 2020-07-27 08:23:29 -07:00
Wellington Ramos Chevreuil fce52fe6bb HBASE-24758 Avoid flooding replication source RSes logs when no sinks… (#2118) 
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>

(cherry picked from commit 8c0d7fa5b8)
 2020-07-27 12:43:57 +01:00
WenFeiYi 2bb76c0a51
HBASE-24665 (#2150) 
Co-authored-by: wen_yi <liu.wenwen@immomo.com>
 Signed-off-by: Anoop <anoopsamjohn@apache.org>
 Signed-off-by: Ramkrishna <ramkrishna@apache.org>
 Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-07-27 16:39:49 +05:30
Viraj Jasani 6cb51cc0f0
HBASE-24757 : ReplicationSink should limit row count in batch mutation based on hbase.rpc.rows.warning.threshold (#2136) 
Closes #2127

Signed-off-by: stack <stack@apache.org>
 2020-07-24 20:32:40 +05:30
Guanghao Zhang 70ab0dc324
HBASE-24743 Reject to add a peer which replicate to itself earlier (#2124) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2020-07-24 08:09:02 +08:00
Viraj Jasani ce4e692699
HBASE-24718 : Generic NamedQueue framework for multiple use-cases (Refactor SlowLog responses) (#2110) 
Closes #2052

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2020-07-22 11:36:19 +05:30
surbhi 33102a1265 HBASE-22146 Removing a namespace-level space quota does not remove policies against contained tables 
Closes #1935

Signed-off-by: Josh Elser <elserj@apache.org>
 2020-07-21 21:24:06 -04:00
Viraj Jasani 5bb76bf22d
Revert "HBASE-24743 Reject to add a peer which replicate to itself earlier (#2071)" 
This reverts commit 5db3ec2cfb.

TestReplicationAdmin and TestReplicationShell are broken on branch-2 and master respectively
 2020-07-21 20:20:51 +05:30
Mingliang Liu 3c91c33452
HBASE-24696 Include JVM information on Web UI under "Software Attributes" 
Closes #2087

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2020-07-21 16:55:44 +05:30
Sanjeet Nishad 70d1b35a4b
HBASE-24714 Error message is displayed in UI of table's compaction state if any region of that table is not open 
Closes #2047

Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-07-21 12:35:08 +05:30
sguggilam 19d0f364ed
HBASE-24740 Enable journal logging for HBase snapshot operation (#2104) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
(cherry picked from commit 430602aed8)
 2020-07-20 18:56:37 -07:00
Guanghao Zhang 5db3ec2cfb HBASE-24743 Reject to add a peer which replicate to itself earlier (#2071) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2020-07-21 09:06:54 +08:00
stack 620470607e Revert "HBASE-24718 : Generic NamedQueue framework for multiple use-cases (Refactor SlowLog responses)" 
Causes TestAdminShell and TestThriftHBaseServiceHandler to fail 100% of the time in branch-2.

This reverts commit 8ae3480e70.
 2020-07-20 14:50:03 -07:00
Wellington Ramos Chevreuil 8c89582633 HBASE-24747 Log an ERROR if HBaseSaslRpcServer initialisation fails with an uncaught exception (#2079) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
(cherry picked from commit bd42c75cac)
 2020-07-20 10:29:04 +01:00
Zheng Wang 0a8a7fa2ed
HBASE-24709 Support MoveCostFunction use a lower multiplier in offpeak hours 
Closes #2099

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-07-19 20:11:20 +05:30
Michael Stack 552008c577 HBASE-24698 Turn OFF Canary WebUI as default (#2037) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2020-07-17 15:52:21 -07:00
Zheng Wang 4001d3f2d2
HBASE-24746 The sort icons overlap the col name in master UI 
Closes #2078

Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-07-17 21:51:49 +05:30
Viraj Jasani 8ae3480e70
HBASE-24718 : Generic NamedQueue framework for multiple use-cases (Refactor SlowLog responses) 
Closes #2052

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2020-07-17 19:57:05 +05:30
bsglz ec3beaf5a2
HBASE-24664 Some changing of split region by overall region size rather than only one store size (#2054) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2020-07-17 10:47:01 +01:00
bsglz bf368a01bc
HBASE-24382 Flush partial stores of region filtered by seqId when archive wal due to too many wals (#2049) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2020-07-17 10:40:50 +01:00
Lars 86f00e4749 HBASE-24742 Improve performance of SKIP vs SEEK logic. 2020-07-16 13:10:06 -07:00
huaxiangsun 0e45e377b8
HBASE-24705 MetaFixer#fixHoles() does not include the case for read replicas (i.e, replica regions are not created) (#2062) (#2068) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-07-14 22:38:01 -07:00
BukrosSzabolcs 7900afdb4f HBASE-24720: Meta replicas not cleaned when disabled (#2057) 
- make sure to always clean up excess meta replicas not just when their
number get decreased
- make sure NotServingRegionException is handled properly even when
wrapped
- add test

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
 2020-07-14 19:06:01 +02:00
WenFeiYi e1d1304cb1
HBASE-24578 [WAL] Add a parameter to config RingBufferEventHandler's SyncFuture count 
Signed-off-by: Reid Chan <reidchan@apache.org>
 2020-07-14 10:39:19 +08:00
Geoffrey Jacoby a3567037c6
HBASE-23744 - FastPathBalancedQueueRpcExecutor should enforce queue length of 0 
Closes #1094

Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-07-13 12:19:02 +05:30
Zheng Wang fce1df9a3c
HBASE-24586 Add table level locality in table.jsp 
Closes #2048

Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-07-11 17:20:22 +05:30
chenglei deff4c88d3
HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length.(#2034) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2020-07-11 16:35:10 +08:00
Joseph295 ff2951e672 HBASE-24663 Add procedure process time statistics UI (#2035) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-07-10 15:05:57 +08:00
niuyulin eee467d119 HBASE-24653 Show snapshot owner on Master WebUI (#2016) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-07-10 15:05:48 +08:00
XinSun 5fcffae5db
HBASE-24431 RSGroupInfo add configuration map to store something extra (#2031) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-07-08 14:39:41 +08:00
Toshihiro Suzuki f771fd26b4
HBASE-24650 Change the return types of the new checkAndMutate methods introduced in HBASE-8458 (#2033) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-07-08 08:00:48 +09:00
Duo Zhang 28602d6fdb Revert "HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length. (#1970)" 
This reverts commit f834919929.
 2020-07-07 18:16:58 +08:00
WenFeiYi 5416cef27f HBASE-24546 CloneSnapshotProcedure unlimited retry (#1900) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-07-06 13:59:07 +08:00
chenglei f834919929 HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length. (#1970) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-07-04 21:12:07 +08:00
Baiqiang Zhao cb154d50aa
HBASE-24560 Add a new option of designatedfile in RegionMover 
Closes #1901

Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-07-01 20:28:35 +05:30
pengmq1 2a12fd283e HBASE-24289 Heterogeneous Storage for Date Tiered Compaction (#1730) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-06-30 15:19:30 +08:00
stack 0dd42f7916 HBASE-24648 Remove the legacy 'forceSplit' related code at region server side (#1990) 
Addendum to fix TestHeapSize
 2020-06-29 14:19:44 -07:00
Michael Stack 91da188d78
HBASE-24656 [Flakey Tests] branch-2 TestMasterNoCluster.testStopDuringStart (#1998) 2020-06-29 11:10:14 -07:00
Duo Zhang 3effd28a75 HBASE-24648 Remove the legacy 'forceSplit' related code at region server side (#1990) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-06-29 23:16:21 +08:00
Duo Zhang ed6104418c HBASE-24646 Set the log level for ScheduledChore to INFO in HBTU (#1987) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
 2020-06-28 23:27:42 +08:00
Viraj Jasani 0fe85947ec
HBASE-24588 : Submit task for NormalizationPlan (#1933) (#1983) 
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2020-06-28 01:10:42 +05:30
Duo Zhang 7c78356218 HBASE-24635 Split TestMetaWithReplicas (#1980) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-06-27 11:11:36 +08:00
huaxiangsun 2c201cc034
HBASE-24552 Replica region needs to check if primary region directory exists at file system in TransitRegionStateProcedure (#1924) (#1971) 
Signed-off-by:  stack <stack@apache.org>
 2020-06-26 11:27:30 -07:00
Nick Dimiduk a499eae1aa HBASE-24221 addendum to restore public interface on LoadIncrementalHFiles 
Signed-off-by: niuyulin <nyl353@163.com>
Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-06-26 09:06:32 -07:00
BukrosSzabolcs 50d1a79255 HBASE-24562: Stabilize master startup with meta replicas enabled (#1903) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
(cherry picked from commit 8cdb2cca44)
 2020-06-26 16:50:55 +01:00
Bharath Vissapragada a6c887012e
HBASE-24603: Make Zookeeper sync() call synchronous (#1945) (#1975) 
Writing a test for this is tricky. There is enough coverage for
functional tests. Only concern is performance, but there is enough
logging for it to detect timed out/badly performing sync calls.

Additionally, this patch decouples the ZK event processing into it's
own thread rather than doing it in the EventThread's context. That
avoids deadlocks and stalls of the event thread.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 84e246f9b1)
 2020-06-25 16:41:11 -07:00
Michael Stack e6639f9d4e
HBASE-24616 Remove BoundedRecoveredHFilesOutputSink dependency on a TableDescriptor (#1955) 
Purge query Master for table descriptors; make do w/ generic options.

Logging cleanup.

hbase-server/src/main/java/org/apache/hadoop/hbase/wal/BoundedRecoveredHFilesOutputSink.java
 Undo fetching Table Descriptor. Not reliably available at recovery time.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-06-25 11:45:06 -07:00
xincunSong a787d2e292 HBASE-24605 Break long region names in the web UI (#1942) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
 2020-06-23 10:06:31 +08:00
Viraj Jasani 4506f8d8ab
HBASE-24102 : Undo visibility change for RegionMover fields (ADDENDUM) 2020-06-22 22:48:54 +05:30
Viraj Jasani 92b6621fa5
HBASE-24594 : Separate resource allocation for TestReplicationStatusSink (#1934) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-06-21 16:15:31 +05:30
Viraj Jasani 48fda91c5b
HBASE-24380 : Provide WAL splitting journal logging (#1860) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>

Conflicts:
	hbase-server/src/main/java/org/apache/hadoop/hbase/wal/BoundedRecoveredEditsOutputSink.java
	hbase-server/src/main/java/org/apache/hadoop/hbase/wal/BoundedRecoveredHFilesOutputSink.java
	hbase-server/src/main/java/org/apache/hadoop/hbase/wal/RecoveredEditsOutputSink.java
 2020-06-19 16:27:48 -07:00
stack 7b171a3c1a HBASE-24574 Procedure V2 - Distributed WAL Splitting => LOGGING (#1912) 
Addendum; minor log edits
 2020-06-18 08:37:19 -07:00
Michael Stack 6eb93287cc
HBASE-24574 Procedure V2 - Distributed WAL Splitting => LOGGING (#1912) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java
 Don't register as a chore on construction if no coordination state
 manager instance (there is no instance when procv2 WAL splitter).

hbase-server/src/main/java/org/apache/hadoop/hbase/master/SplitWALManager.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
 Edit logs.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionRemoteProcedureBase.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/SplitWALProcedure.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/SplitWALRemoteProcedure.java
 Add proc name rather than rely on default behavior. Add detail to the
 toString.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 Factoring

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AbstractFSWAL.java
 Print the maxLogs... we don't do it any where.

hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WAL.java
 Utility method to strip prefix from wal path.

Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: tianjingyun <tianjy@apache.org>
 2020-06-18 08:32:48 -07:00
ramkrish86 ef809c198b
HBASE-24205 - Create metric to know the number of reads that happens (#1920) 
* HBASE-24205 - Create metric to know the number of reads that happens
from memstore (branch-2)

* Add the optimization as in master and fix whitestyle and checkstyle

* Fix compilation error that accidently crept in

Authored-by: Ramkrishna <ramkrishna@apache.org>
Signed-off by:Anoop Sam John<anoopsamjohn@gmail.com>
Signed-off by:Viraj Jasani<virajjasani@apache.org>
 2020-06-18 18:59:46 +05:30
Michael Stack 812d1e2bb5
HBASE-24577 Doc WALSplitter classes (#1913) 
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-06-17 14:59:30 -07:00
Viraj Jasani e202c98fb0
HBASE-24518 : waitForNamespaceOnline() should return false if any region is offline (#1869) 
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
 2020-06-17 20:05:15 +05:30
Bharath Vissapragada c49df36e85 HBASE-24564: Make RS abort call idempotent. 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 2567d15218)
 2020-06-16 08:43:06 -07:00
xincunSong 36ee96ca36 HBASE-24478 The regionInfo parameter for MasterProcedureScheduler#wai… (#1819) 
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
Signed-off-by: clarax <clarax98007@gmail.com>
 2020-06-16 17:35:17 +08:00
Sandeep Pal cb3d0d7d21
HBASE-24015: Test for Assign and Unassign of Regions on RegionServer on failure (#1898) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-06-16 12:54:02 +05:30
bsglz 035ea516e9
HBASE-24038 Add a metric to show the locality of ssd in table.jsp (#1902) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-06-15 19:28:07 +05:30
Duo Zhang 700c5a7e3b HBASE-24517 Addendum add more comments (#1882) 
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-06-15 10:44:30 +08:00
Toshihiro Suzuki 6ff6fe49a3
HBASE-8458 Support for batch version of checkAndMutate() (#1897) 
Signed-off-by: Josh Elser <elserj@apache.org>
 2020-06-14 13:55:06 +09:00
zhaoym6 2d57595f54 HBASE-23195 FSDataInputStreamWrapper unbuffer can NOT invoke the classes that NOT implements CanUnbuffer but its parents class implements CanUnbuffer 
Closes #746

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
 2020-06-12 13:27:53 -04:00
Michael Stack a4e1d073f4 HBASE-24545 Add backoff to SCP check on WAL split completion (#1891) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-06-12 07:58:02 -07:00
sguggilam d187ce2029
HBASE-24446 Use EnvironmentEdgeManager to compute clock skew in Master (#1885) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-06-12 16:52:15 +05:30
Toshihiro Suzuki 043a9e862f HBASE-24529 hbase.rs.evictblocksonclose is not honored when removing compacted files and closing the storefiles (#1881) 
Signed-off-by: Anoop Sam John <anoop.hbase@gmail.com>
 2020-06-12 19:02:20 +09:00
Anoop Sam John ed7dc9ed4c HBASE-24189 WALSplit recreates region dirs for deleted table with recovered edits data. 2020-06-11 21:28:32 +05:30
Viraj Jasani 1b1c269911
HBASE-24532 : Execute region plans with throttle should return succeeded plans (#1884) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2020-06-11 18:23:32 +05:30
sguggilam c02554024f
HBASE-24511 Ability to configure timeout between RPC retry to RS from master (#1861) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-06-10 13:04:47 +05:30
xincunSong dc2cf6ab5c HBASE-24441 CacheConfig details logged at Store open is not really u… (#1801) 
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: binlijin <binlijin@gmail.com>
 2020-06-09 14:19:15 +08:00
xincunSong 101023bcc1 HBASE-24468 Add region info when log meessages in HStore. (#1803) 
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: binlijin <binlijin@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-06-09 14:14:05 +08:00
huaxiangsun 02e7beaca1
HBASE-23202 ExportSnapshot (import) will fail if copying files to root directory takes longer than cleaner TTL (#1874) (#1875) 
Co-authored-by: Guangxu Cheng <guangxucheng@gmail.com>

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Zach York <zyork@apache.org>
 2020-06-08 22:51:48 -07:00
Duo Zhang c5dacfbbea HBASE-24117 Shutdown AssignmentManager before ProcedureExecutor may cause SCP to accidentally skip assigning a region (#1865) 
Signed-off-by: Michael Stack <stack@apache.org>
 2020-06-09 11:16:49 +08:00
Duo Zhang 6d96694a25 HBASE-24517 AssignmentManager.start should add meta region to ServerStateNode (#1866) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
 2020-06-09 08:27:17 +08:00
WenFeiYi 9ba3be61b3 HBASE-24444 Should shutdown mini cluster after class in TestMetaAssignmentWithStopMaster (#1802) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-06-09 08:21:52 +08:00
gkanade 82c50d72e2
HBASE-24208 Remove RS entry from zk draining servers node after RS been stopped (#1841) 
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-06-08 22:50:26 +05:30
XinSun 67fa367bb6 HBASE-24359 Optionally ignore edits for deleted CFs for replication (#1855) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-06-08 13:14:50 +08:00
Duo Zhang 108e23630a HBASE-24506 async client deadlock (#1858) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2020-06-08 09:57:33 +08:00
Toshihiro Suzuki 467efa573c HBASE-24515 batch Increment/Append fails when retrying the RPC 
Signed-off-by: Viraj Jasani <virajjasani007@gmail.com>
 2020-06-08 10:41:41 +09:00
bsglz 0c9806faec
HBASE-24496 The tag of Base Stats not actived by default in table.jsp (#1837) 
Signed-off-by: clarax
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-06-06 19:17:41 +05:30
Duo Zhang 72dadcd615 HBASE-24500 The behavior of RegionInfoBuilder.newBuilder(RegionInfo) is strange (#1850) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-06-05 15:24:39 +08:00
Duo Zhang 82b8feecef HBASE-24505 Reimplement Hbck.setRegionStateInMeta (#1852) 
Signed-off-by: Wellington Ramos Chevreuil <wchevreuil@apache.org>
 2020-06-05 08:55:27 +08:00
Viraj Jasani cc4c28c575
HBASE-24456 : Create ImmutableScan and use it for CustomizedScanInfoBuilder (#1818) 
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-06-04 16:30:32 +05:30
Duo Zhang 4a84e28937 Revert "HBASE-24500 The behavior of RegionInfoBuilder.newBuilder(RegionInfo) is strange (#1840)" 
This reverts commit 8815215b3e.
 2020-06-04 12:17:15 +08:00
Duo Zhang 8815215b3e HBASE-24500 The behavior of RegionInfoBuilder.newBuilder(RegionInfo) is strange (#1840) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
 2020-06-04 11:18:25 +08:00
Bharath Vissapragada e59f39e09e
HBASE-21905: [FIFOCompaction] Don't compact already inflight store files (#1829) 
This one surfaced as a flake test but turns out to be a legit bug
in FIFOCompaction code. FifoCompaction does not check if an empty
store file is already being compacted by an in-flight compaction
request and still enqueues. It can potentially race with a running
compaction (as in this test case, see jira for the exact exception).

Fixes the edge case and cleans up the test code a bit.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2020-06-03 17:03:24 -07:00
Nick Dimiduk 4884773676 HBASE-24418 Consolidate Normalizer implementations 
Simplify our Normalizer story to have just a single, configurable
implementation.

* fold the features of `MergeNormalizer` into
  `SimpleRegionNormalizer`, removing the intermediate abstract class.
* configuration keys for merge-only features now share a common
  structure.
* add configuration to selectively disable normalizer split/merge
  operations.
* `RegionNormalizer` now extends `Configurable` instead of creating a
  new instance of `HBaseConfiguration` or snooping one off of other
  fields.
* avoid the extra RPCs by using `MasterServices` instead of
  `MasterRpcServices`.
* boost test coverage of all the various flags and feature
  combinations.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
 2020-06-03 16:43:40 -07:00
Wellington Ramos Chevreuil 11d093bc39 HBASE-21406 "status 'replication'" should not show SINK if the cluste… (#1761) 
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>

(Cherry picked from commit e5345b3a7c)
 2020-06-03 09:33:36 +01:00
Viraj Jasani b3c6af9500
HBASE-24492 : Remove infinite loop from ProtobufLogReader#readNext (#1831) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-06-03 13:44:19 +05:30
niuyulin 571ef7e811 HBASE-24412 Canary support check only one column family per RegionTask (#1817) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-06-03 09:16:11 +08:00
stack 9c8895cfbf HBASE-24474 Rename LocalRegion to MasterRegion (#1811) 
Addendum: HRegion changed size.
 2020-06-02 10:44:48 -07:00
Duo Zhang 6b43015fc9 HBASE-24474 Rename LocalRegion to MasterRegion (#1811) 
Signed-off-by: Michael Stack <stack@apache.org>
 2020-06-02 10:43:27 +08:00
Bharath Vissapragada a7a8afdf6a
HBASE-24479: Deflake TestCompaction#testStopStartCompaction (#1820) 
Polling of active compaction count is racy. Tightened the asserts
to be more reliable.

Signed-off-by: Reid Chan <reidchan@apache.org>
 2020-05-31 23:25:39 -07:00
Bharath Vissapragada d7f388518c
HBASE-24477: Move ConfigurationObserver and related classes to hbase-common 
This utility is useful for any module that wants to detect
dynamic config changes. Having it to hbase-common makes it
accessible to all the other modules.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-05-31 14:14:30 -07:00
bsglz 327d090e1b HBASE-24475 Clean up the master thread name getting in SplitLogManager and AssignmentManager (#1810) 2020-05-30 22:24:05 -07:00
jacob-leblanc 7b17e1cdaa HBASE-24454 - Read ioErrorStartTime to local temporary variable to avoid issue when it is set to -1 between greater than zero check and calculation of error duration (#1816) 
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
 2020-05-30 21:07:46 -07:00
Duo Zhang 4d5efec767 HBASE-24471 The way we bootstrap meta table is confusing (#1806) 
Signed-off-by: Michael Stack <stack@apache.org>
 2020-05-30 16:37:41 +08:00
Wellington Ramos Chevreuil 2dd160e4c7 HBASE-23716 MasterFileSystem should not create MasterProcWALs dir on … (#1078) 
Signed-off-by: Josh Elser <elserj@apache.org>
 2020-05-30 16:37:30 +08:00
Duo Zhang 1ac00e11cb Revert "HBASE-23716 MasterFileSystem should not create MasterProcWALs dir on … (#1078)" 
Wrong jira id

This reverts commit aca7f6fcf3.
 2020-05-30 16:37:19 +08:00
Viraj Jasani b2d24a1812
HBASE-23941 : FilterBy operator support in get_slowlog_responses API (#1793) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2020-05-30 13:57:32 +05:30
Wellington Ramos Chevreuil aca7f6fcf3 HBASE-23716 MasterFileSystem should not create MasterProcWALs dir on … (#1078) 
Signed-off-by: Josh Elser <elserj@apache.org>
 2020-05-30 16:11:32 +08:00
Michael Stack 6f1992f32f
HBASE-22287 inifinite retries on failed server in RSProcedureDispatcher (#1800) 
Adds backoff in place of retry every 100ms.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-05-29 10:00:53 -07:00
xincunSong 8edcbb4441
HBASE-24470 : Add store file info when log meessages in HStore. (#1804) 
Signed-off-by: binlijin <binlijin@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-05-29 19:01:12 +05:30
Viraj Jasani bce06627bb
HBASE-24428 : Fix for testSplitCompactWithPriority (ADDENDUM) 2020-05-29 18:24:12 +05:30
bsglz 22efe67758
HBASE-24451 Remove the HasThread because the related bug had been fixed since jdk7 (#1797) 
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-05-29 17:46:03 +05:30
Viraj Jasani 1e86ff09d7
HBASE-24428 : Update compaction priority for recently split daughter regions (#1784) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2020-05-27 20:58:57 +05:30
binlijin c94b946b31 HBASE-24371 Add more details when print CompactionConfiguration info (#1711) 
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
 2020-05-27 19:48:53 +08:00
Duo Zhang 441935a9d9 HBASE-24437 Addendum just start mini dfs cluster, and make the log roll more robust (#1787) 
Signed-off-by: stack <stack@apache.org>
 2020-05-27 11:26:35 +08:00
Michael Stack f8a5cc92af
HBASE-24425 Run hbck_chore_run and catalogjanitor_run on draw of 'HBCK Report' page (#1771) 
Signed-off-by: clarax
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
 2020-05-26 19:50:23 -07:00
Zheng Wang 1e16d385f8 HBASE-24423 No need to get lock in canSplit because hasReferences will get lock too 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2020-05-26 13:37:29 -07:00
stack 3ef44a4dcf HBASE-24437 Flaky test, TestLocalRegionOnTwoFileSystems#testFlushAndCompact 
Force another WAL roll just in case and make the check more loose (its
about whether the files are present, not their count)
 2020-05-26 11:55:09 -07:00
huaxiangsun 90bd19e55b
HBASE-24379 CatalogJanitor misreports region holes when there are actually over laps. (#1741) (#1777) 
Signed-off-by: stack <stack@apache.org>
 2020-05-26 10:06:13 -07:00
XinSun d285c6b681 HBASE-24416 RegionNormalizer spliting region should not be limited by hbase.normalizer.min.region.count (#1770) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-05-26 17:05:16 +08:00
huaxiangsun 51a84025cb
HBASE-24369 Provide more information about merged child regions in Hbck Overlaps section, which cannot be fixed immediately (#1756) (#1768) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Clara Xiong
Signed-off-by: stack <stack@apache.org>
 2020-05-25 16:00:39 -07:00
WenFeiYi b0fd3fc3e2
HBASE-24427 HStore.add log format error (#1773) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-05-25 18:30:00 +05:30
Viraj Jasani 9d364af796
HBASE-24426 : Provide regionName in warning during failure to move region from CLOSING to CLOSED (#1772) 
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
 2020-05-25 17:38:01 +05:30
wenbang 03929770fe HBASE-24401 Cell size limit check on append should consider 0 or less value to disable the check (#1742) 
Signed-off-by: Guanghao Zhang <zghao@apache.org>
 2020-05-25 10:05:18 +08:00
Duo Zhang 58eed9a4bb HBASE-24408 Introduce a general 'local region' to store data on master (#1753) 
Signed-off-by: stack <stack@apache.org>
 2020-05-23 16:40:39 +08:00
huaxiangsun be9e12d626
HBASE-24370 Avoid aggressive MergeRegion and GCMultipleMergedRegionsProcedure (#1719) (#1763) 
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
 2020-05-22 16:40:16 -07:00
huaxiangsun b144d17044
HBASE-24376 MergeNormalizer is merging non-adjacent regions and causing region overlaps/holes. (#1734) (#1758) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
 2020-05-22 08:09:36 -07:00
Viraj Jasani 6f5e5e4828
HBASE-23938 : System table hbase:slowlog to store complete slow/large… (#1681) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: ramkrish86 <ramkrishna@apache.org>
 2020-05-20 15:30:30 +05:30
niuyulin 3605db2fa0
HBASE-24386 TableSnapshotScanner support scan limit (#1724) 
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off by: Viraj Jasani <vjasani@apache.org>
 2020-05-19 21:10:03 +05:30
Mingliang Liu 5893c4f393
HBASE-23969 Meta browser should show all `info` columns (#1710) 
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-05-18 11:48:36 -07:00
ramkrish86 04cd6dc3f8 HBASE-24365 MetricsTableWrapperAggregateImpl runnable fails due to exception and never runs (#1729) 
Signed-off-by: binlijin <binlijin@gmail.com>
Signed-off by: Viraj Jasani <vjasani@apache.org>
 2020-05-18 15:13:41 +05:30
binlijin 9fd61f4774 HBASE-24377 MemStoreFlusher throw NullPointerException (#1721) 
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: Anoop Sam John <anoop.hbase@gmail.com>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-05-18 10:12:15 +08:00
Baiqiang Zhao b66d8edbd6 HBASE-24381 The Size metrics in Master Webui is wrong if the size is 0 (#1723) 2020-05-18 09:25:40 +08:00
Guanghao Zhang 692a597b63 HBASE-23702 hbase.hstore.flusher.count setting to 0 breaks HMaster (#1703) 
Co-authored-by: Ctest
Signed-off-by: Anoop Sam John <anoop.hbase@gmail.com>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2020-05-15 09:16:22 +08:00
Andrew Purtell a93d94cad3
Amend HBASE-24350: Extending and Fixing HBaseTable level replication metrics (#1704) 
- Rename WALEntryBatch#getWaEntriesWithSize -> getWalEntriesWithSize
 2020-05-14 13:42:35 -07:00
Sandeep Pal 1ff532678d
HBASE-24350: Extending and Fixing HBaseTable level replication metrics (#1704) 
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2020-05-14 13:36:11 -07:00
stack 2e4be3e77d HBASE-24368 Let HBCKSCP clear 'Unknown Servers', even if RegionStateNode has RegionLocation == null 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/GCMultipleMergedRegionsProcedure.java
 Edit a log.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/HBCKServerCrashProcedure.java
 Add override of isMatchingRegionLocation. Allow 'null' as a pass in
 HBCKSCP.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/ServerCrashProcedure.java
 Add a method for HBCKSCP to override and be less strict filtering
 assigns.

hbase-server/src/main/resources/hbase-webapps/master/hbck.jsp
 Some doc on what 'Unknown Servers' are.
 2020-05-14 13:07:59 -07:00
bsglz 8d4cb772b8 HBASE-24164 Retain the ReadRequests and WriteRequests of region on we… (#1500) 
Signed-off-by: stack <stack@apache.org>
 2020-05-13 22:30:24 -07:00
Viraj Jasani cb17d70226
Revert "HBASE-24327 : Flaky connection in TestMasterShutdown#testMasterShutdo… (#1690)" 
This reverts commit d9b60d3339.
 2020-05-13 22:12:58 +05:30
Duo Zhang dc2146069c
HBASE-24309 Avoid introducing log4j and slf4j-log4j dependencies for … (#1697) 
Signed-off-by: stack <stack@apache.org>
 2020-05-13 17:59:21 +08:00
Viraj Jasani d9b60d3339
HBASE-24327 : Flaky connection in TestMasterShutdown#testMasterShutdo… (#1690) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2020-05-13 02:40:36 +05:30
huaxiangsun b7f7326950
HBASE-24256 When fixOverlap hits the max region limit, it is possible to include the same region in multiple merge request (#1584) (#1694) 
Signed-off-by: stack <stack@apache.org>
 2020-05-12 09:54:12 -07:00
Gkkkk302 a7e3c03541
HBASE-24355 Fix typos in the HStore#compact annotation (#1699) 
Co-authored-by: guo.kangkang <guo.kangkang@immomo.com>

Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-05-12 21:08:16 +05:30
stack 5c08a8c04b HBASE-24338 [Flakey Tests] NPE in TestRaceBetweenSCPAndDTP Addendum2: make same fix for TestRaceBetweenSCPAndTRSP but in a second place (addendum 1 got first place). 2020-05-11 20:50:05 -07:00
stack cf786a2dfb HBASE-24338 [Flakey Tests] NPE in TestRaceBetweenSCPAndDTP 
Addendum: make same fix for TestRaceBetweenSCPAndTRSP which does same
thing.
 2020-05-11 14:08:50 -07:00
Viraj Jasani 60035721ca
Revert "HBASE-24327 : Handle shutdown() if master cannot be contacted (#1684)" 
This reverts commit a89967dfe8.
 2020-05-11 16:50:01 +05:30
Geoffrey Jacoby 2993a95ee2
HBASE-24321 - Add writable MinVersions and read-only Scan to coproc S… (#1655) 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Lars Hofhansl <larsh@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-05-11 12:55:45 +05:30
Viraj Jasani a89967dfe8
HBASE-24327 : Handle shutdown() if master cannot be contacted (#1684) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2020-05-10 17:14:32 +05:30
Duo Zhang f41c9038e7
HBASE-24333 Backport HBASE-24304 "Separate a hbase-asyncfs module" to branch-2.x (#1664) 
Signed-off-by: stack <stack@apache.org>
 2020-05-09 12:00:45 +08:00
stack 406a3e8178 HBASE-24338 [Flakey Tests] NPE in TestRaceBetweenSCPAndDTP #1669 
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-05-07 13:43:35 -07:00
stack 618dca5782 HBASE-24338 [Flakey Tests] NPE in TestRaceBetweenSCPAndDTP 2020-05-07 13:30:23 -07:00
stack ba85ea3eff HBASE-24342 [Flakey Tests] Disable TestClusterPortAssignment.testClusterPortAssignment as it can't pass 100% of the time 2020-05-07 13:20:54 -07:00
niuyulin 735aa8bf9f
HBASE-24328 skip duplicate GCMultipleMergedRegionsProcedure while previous finished (#1672) 
Co-authored-by: niuyulin <niuyulin@xiaomi.com>
 2020-05-07 10:25:25 -07:00
huaxiangsun fca0f3f810
HBASE-24316 GCMulitpleMergedRegionsProcedure is not idempotent (#1660) (#1671) 
It addresses couple issues:
   1. Make sure deleteMergeQualifiers() does not delete the row if there is no columns with "merge" keyword.
   2. GCMulitpleMergedRegionsProcedure now acquire an exclusive lock on the child region.

Signed-off-by: stack <stack@apache.org>
 2020-05-06 16:07:21 -07:00
huaxiangsun 04ac08d49b
HBASE-24273 HBCK's "Orphan Regions on FileSystem" reports regions with referenced HFiles (#1613) (#1661) 
Signed-off-by: stack <stack@apache.org>
 2020-05-06 09:50:52 -07:00
Michael Stack 89ae3c55c8
HBASE-24331 [Flakey Test] TestJMXListener rmi port clash (#1657) 
Add check that we can make jmx connector in setup. If we can't retry.
 2020-05-06 09:27:52 -07:00
Reid Chan 20a7a542ff HBASE-24336 [Metrics] FSDataInputStream's localBytesRead is wrong (#1667) 
Signed-off-by: binlijin <binlijin@gmail.com>
 2020-05-06 19:36:53 +08:00
Mohammad Arshad edd8829cfc
HBASE-24211: Create table is slow in large cluster when AccessController is enabled. (#1631) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-05-06 15:17:34 +05:30
Duo Zhang 7f4683bafa HBASE-24265 Remove hedged rpc call support, implement the logic in MaterRegistry … (#1593) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2020-05-06 16:11:28 +08:00
Nick Dimiduk e6b8ca9dcf HBASE-23976 Use nio for file manipulation in TestVerifyBucketCacheFile (#1342) 
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
 2020-05-05 22:05:25 -07:00
stack 99e96cca3b HBASE-24118 [Flakey Tests] TestCloseRegionWhileRSCrash 
Addendum; add back bit missing off end of test.
 2020-05-05 21:15:22 -07:00
Michael Stack 528e9d6454
HBASE-24307 [Flakey Tests] krb server for secure thrift tests throws BindException (#1656) 
Move the random free port generate back into hbasecommontestingutility
  from hbasetestingutility.

  Add a create simple kdc server utility that will start a kdc server and
  if a bindexception, create a new one on a new random port in hbase-common.

  Add new BoundSocketMaker helpful when trying to manufacture
  BindExceptions because of port clash.

  Change thrift and http kdc tests to use this new utility (removes
  code duplication around kdc server setup).
 2020-05-05 21:08:09 -07:00
Michael Stack 80561504d2
HBASE-24284 [h3/jdk11] REST server won't start Exclude transitive includes of jax-rs 1.x and then explicitly include jax-rs 2.x glassfish impl for REST context when hadoop3. (#1625) 2020-05-05 15:27:11 -07:00
Duo Zhang 64bfc99f05 HBASE-24314 Some classes still use log4j logger directly (#1637) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
 2020-05-05 20:24:58 +08:00
Anoop Sam John 5e5b13cb51
HBASE-24311 Add more details in MultiVersionConcurrencyControl STUCK log message (#1654) 2020-05-05 16:25:40 +05:30
Nick Dimiduk 47dca8eb45 HBASE-24260 Add a ClusterManager that issues commands via coprocessor 
Implements `ClusterManager` that relies on the new
`ShellExecEndpointCoprocessor` for remote shell command execution.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
 2020-05-04 10:52:28 -07:00
stack ffae8a570c HBASE-24246 Miscellaneous hbck2 fixMeta bulk merge fixes: better logging around merges/overlap-fixing, 'HBCK Report' overlap listing, and configuration (#1572) 
Addendum to address minor feedback on text
 2020-05-04 08:24:09 -07:00
Michael Stack 1dd4b13e17
HBASE-24246 Miscellaneous hbck2 fixMeta bulk merge fixes: better logging around merges/overlap-fixing, 'HBCK Report' overlap listing, and configuration (#1572) 
hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
 Fix weird brackets around each region name when logging.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetaFixer.java
  Log when we hit the max merge limit. Also up limit to 64.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MergeTableRegionsProcedure.java
 Make logs make more sense to operator.

hbase-server/src/main/resources/hbase-webapps/master/hbck.jsp
 Make RegionName show when you mouseover so long names don't mess up
 display of holes and overlaps.

Address Mingliang Liu liuml07 feedback

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
 2020-05-04 08:21:52 -07:00
niuyulin 0a428e8b48
HBASE-24221 Support bulkLoadHFile by family (#1627) 
Co-authored-by: niuyulin <niuyulin@xiaomi.com>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
 2020-05-03 13:46:53 +01:00
stack da92a12bf5 HBASE-24199 Procedure related metrics is not consumed in the JMX metric (#1616) 
Addendum; fix failing test
 2020-05-01 17:31:59 -07:00
xincunSong 0d2b959d20 [HBASE-24288]Allow admin user to create table and do bulkLoad (#1612) 
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
Signed-off-by: binlijin <binlijin@gmail.com>
 2020-05-01 11:58:06 -07:00
gkanade 1df3bee5f6 HBASE-24199 Procedure related metrics is not consumed in the JMX metric (#1616) 
Signed-off-by: stack <stack@apache.org>
 2020-05-01 11:49:32 -07:00
bsglz a6aea729c5
HBASE-22710 Wrong result in one case of scan that use raw and version… (#767) 
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-04-30 14:11:04 +05:30
xincunSong d59af05b0f HBASE-24282 'scanDetail' log message is missing when responseTooSlow happens on the first scan rpc call (#1604) 
Signed-off-by: Guangxu Cheng <gxcheng@apache.org>
Signed-off-by: stack <stack@apache.org>
 2020-04-29 16:04:40 -07:00
stack 93f0d4ce06 HBASE-24247 Failed multi-merge because two regions not adjacent (legitimately). 
Addendum: Restore old semantic where merge of non-adjacent required
'force'
 2020-04-29 14:34:37 -07:00
Duo Zhang 922921ee5f HBASE-24249 Move code in FSHDFSUtils to FSUtils and mark related clas… (#1586) 
Signed-off-by: stack <stack@apache.org>
 2020-04-29 11:31:32 +08:00
stack 3b0812d3e1 HBASE-24215 [Flakey Tests] [ERROR] TestSecureRESTServer java.lang.NoClassDefFoundError: com/sun/jersey/core/spi/factory/AbstractRuntimeDelegate 
Addendum #2: Add jersey-servlet to hadoop3 profile.
 2020-04-28 20:18:31 -07:00
Michael Stack 9e697d0ea6
HBASE-24247 Failed multi-merge because two regions not adjacent (legitimately) (#1570) 
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionInfo.java
 Add new isOverlap method that takes list of RegionInfos checking that
 current RegionInfo is overlapped by the passed in Regions.

    Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
    Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
 2020-04-28 13:22:00 -07:00
Duo Zhang 47348f5887 Revert "HBASE-24211: Create table is slow in large cluster when AccessController is enabled. (#1546)" 
This breaks TestAccessController.testAccessControllerUserPermsRegexHandling

This reverts commit 66e90dece2.
 2020-04-27 22:50:02 +08:00
Duo Zhang 5af807153a HBASE-24264 Disable TestNettyIPC.testHedgedAsyncEcho (#1589) 2020-04-27 09:43:09 +08:00
bsglz ee5283ee82
HBASE-23264 Resolve a TODO of BucketAllocator: "Why we add the extra … (#801) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-04-26 16:20:40 +05:30
Mohammad Arshad 66e90dece2
HBASE-24211: Create table is slow in large cluster when AccessController is enabled. (#1546) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Pankaj <pankajkumar@apache.org>
 2020-04-25 17:45:32 +05:30
niuyulin 90d738107f HBASE-24222 remove FSUtils.checkAccess and replace with FileSystem.access in HBCK (#1557) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-04-23 23:59:34 +08:00
Viraj Jasani b40381857a
HBASE-24024 : Reject multi() requests with rows higher than threshold (#1560) 
Signed-off-by: Reid Chan <reidchan@apache.org>
 2020-04-22 12:51:49 +05:30
Beata Sudi 74c033db83
HBASE-24139 : Balancer should avoid leaving idle region servers (#1511) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
 2020-04-22 10:11:16 +05:30
stack f8ce21f096 HBASE-24226 Address other hard references to '/tmp' found in Configuration 2020-04-21 17:39:30 -07:00
stack 3c1725f2f0 HBASE-24175 [Flakey Tests] TestSecureExportSnapshot FileNotFoundException Addendum #3; catch more '/tmp' references by hadoop. Fix broken build. 2020-04-21 17:39:20 -07:00
stack b262b9a013 HBASE-24175 [Flakey Tests] TestSecureExportSnapshot FileNotFoundException Addendum #3; catch more '/tmp' references by hadoop. 2020-04-21 17:39:04 -07:00
stack 8a1ffa6c43 HBASE-24220 Allow that zk NOTEMPTY multi exception is retryable by running in-series 
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapred/TableRecordReaderImpl.java
hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
 Cleanup checkstyle warnings. Don't depend on hbase-client
 ScannerCallable.

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
 Cut down on cluster resource usage.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/AbstractTestDLS.java
 Debug

hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController3.java
 Debug

hbase-thrift/src/test/java/org/apache/hadoop/hbase/thrift/TestThriftHttpServer.java
 Debug

hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Debug

hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java
 Allow that NONEMPTY is retryable by running in series.
 2020-04-20 15:43:09 -07:00
Guangxu Cheng 2762ddc6a8 HBASE-23896 Snapshot owner cannot delete snapshot when ACL is enabled and Kerberos is not enabled (#1211) 
Signed-off-by: binlijin <binlijin@gmail.com>
 2020-04-20 10:26:21 +08:00
Reid Chan eed39202e1
HBASE-24174 Fix findbugs warning on ServiceAuthorizationManager for master branch (#1537) (#1542) 
Signed-off-by: binlijin <binlijin@gmail.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
 2020-04-18 13:36:19 +08:00
XinSun 935c23312f HBASE-24166 Duplicate implementation for acquireLock between CreateTableProcedure and its parent class (#1486) 
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-04-18 10:31:46 +08:00
Joseph295 c775adfc06 HBASE-24182 log when the region is set to closing status (#1509) 
Signed-off-by: stack <stack@apache.org>
 2020-04-17 12:21:12 -07:00
Wei-Chiu Chuang 98ecd584c4
HBASE-23833. The relocated hadoop-thirdparty protobuf breaks HBase asyncwal (#1301) (#1534) 
* Use Reflection to access shaded Hadoop protobuf classes.

(cherry picked from commit a321e536989083ca3620bf2c53f12c07740bf5b0)

* Update to improve the code:

1. Added license.
2. Added more comments.
3. Wrap byte array instead of copy to make a ByteString.
4. Moved all reflection instantiation to static class loading time.

* Use LiteralByteString to wrap byte array instead of copying it.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
(cherry picked from commit 72727ff9be)
 2020-04-17 12:08:02 -07:00
gkanade 361e81e1f8 MetricsTable#updateFlushTime is wrong (#1539) 
Authored-by: Gaurav Kanade <gkanade@microsoft.com>
Signed-off-by: Ramkrishna <ramkrishna@apache.org>
 2020-04-17 19:24:52 +05:30
Viraj Jasani a0a769d96f
HBASE-24195 : Admin.getRegionServers() should return live servers exc… (#1523) 
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
 2020-04-16 20:16:30 +05:30
BukrosSzabolcs 8498fe7260
HBASE-24186: RegionMover ignores replicationId (#1512) 
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
 2020-04-16 19:56:52 +05:30
huaxiangsun efb5ff07ae
HBASE-24183 [flakey test] replication.TestAddToSerialReplicationPeer (#1514) (#1525) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
 2020-04-15 14:42:38 -07:00
stack 62c5bdcc08 HBASE-24175 [Flakey Tests] TestSecureExportSnapshot FileNotFoundException 
Addendum: add test to check for '/tmp' references.
 2020-04-15 10:55:08 -07:00
Reid Chan 4c83067e1e
HBASE-24112 [RSGroup] Support renaming rsgroup (#1520) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
 2020-04-15 19:04:40 +08:00
GeorryHuang f61aa0292d HBASE-23994: Add WebUI to Canary (#1292) 
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
 2020-04-14 16:09:47 -07:00
binlijin df3768cf2a HBASE-24181 Add region info when log meessages in HRegion. (#1506) 
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
 2020-04-14 19:45:19 +08:00
Jianfei Jiang 2f4ae522c3 HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7 (#1453) 
* HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7

* HBASE-24132 Upgrade to Apache ZooKeeper 3.5.7

Co-authored-by: 姜建飞 10222269 <jiang.jianfei@zte.com.cn>

Signed-off-by: Mate Szalay-Beko <szalay.beko.mate@gmail.com>
Signed-off-by: Norbert Kalmar <nkalmar@cloudera.com>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangdo@apache.org>
 2020-04-13 13:48:47 -07:00
Viraj Jasani 45622abe2f
HBASE-24140 : Move CandidateGenerator and their implementors out of StochasticLoadBalancer (#1458) 
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
 2020-04-12 19:21:06 +05:30
stack 9067dfe83c HBASE-24175 [Flakey Tests] TestSecureExportSnapshot FileNotFoundException 
Overwrite yarn configs that write /tmp dir by default.
 2020-04-11 21:23:17 -07:00
stack cc4aaefa52 HBASE-24167 [Flakey Tests] TestHRegionWithInMemoryFlush#testWritesWhileScanning gets stuck MVCC 2020-04-10 18:30:08 -07:00
huaxiangsun 3e4dfa9b31
HBASE-24161 [flakey test] locking.TestEntityLocks.testEntityLockTimeout (#1477) (#1480) 
Signed-off-by: stack <stack@apache.org>
 2020-04-10 09:42:47 -07:00
huaxiangsun 39187f87dd
HBASE-24159 [flakey test] regionserver.TestRegionMergeTransactionOnCluster (#1472) (#1474) 
Delete tables after each test cases to avoid polluting states.

Signed-off-by: stack <stack@apache.org>
 2020-04-09 21:09:34 -07:00
Pankaj 823cf97495 HBASE-24074 ConcurrentModificationException occured in ReplicationSourceManager while refreshing the peer (#1471) 
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
Signed-off-by: stack <stack@apache.org>
 2020-04-09 16:42:22 -07:00
stack 2d11796380 HBASE-24158 [Flakey Tests] TestAsyncTableGetMultiThreaded 2020-04-09 11:03:22 -07:00
Reid Chan 5012241817 HBASE-24121 [Authorization] ServiceAuthorizationManager isn't dynamically updatable. And it should be (#1439) 
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
 2020-04-09 16:30:00 +08:00
Sean Busbey 611c62f52a HBASE-24138 log more details about balancer decisions for StochasticLoadBalancer (#1455) 
- at DEBUG log messages about RegionCountSkewCostFunction region/server totals
- at DEBUG log messages about the decision to balance or not with total costs
- at TRACE log messages about region count on each server RegionCountSkewCostFunction sees
- at TRACE log message with the individual cost functions used in the decision to balance or not

Signed-off-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 2d78a286b6)
 2020-04-08 15:50:38 -05:00
huaxiangsun 6f00714ad3
HBASE-24120 Flakey Test: TestReplicationAdminWithClusters timeout (#1441) (#1456) 
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
 2020-04-08 12:23:50 -07:00