Commit Graph

17490 Commits

Author SHA1 Message Date
Duo Zhang 54454f8de6 HBASE-24949 Optimize FSTableDescriptors.get to not always go to fs when cache miss (#2317)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-08-29 22:50:39 +08:00
stack 85370f1443 HBASE-24953 Temporarily up the timeout for branch-2 from 9 hours to 12 for build
Addendum upping build time from 12 to 24 hours.
2020-08-26 21:23:49 -07:00
bsglz 08b0b85901
HBASE-24898 Addendum to deal with Daylight Saving Time (#2318)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-08-27 11:18:01 +08:00
WenFeiYi 3a85bdd482 HBASE-24892 config 'hbase.hregion.memstore.mslab.indexchunksize' not be used (#2265)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-08-27 08:53:37 +08:00
Bharath Vissapragada bb9121da77
HBASE-24765: Dynamic master discovery (#2314)
This patch adds the ability to discover newly added masters
dynamically on the master registry side. The trigger for the
re-fetch is either periodic (5 mins) or any registry RPC failure.
Master server information is cached in masters to avoid repeated
ZK lookups.

Updates the client side connection metrics to maintain a counter
per RPC type so that clients have visibility into counts grouped
by RPC method name.

I didn't add the method to ZK registry interface since there
is a design discussion going on in splittable meta doc. We can
add it later if needed.

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
(cherry picked from commit 275a38e153)
2020-08-26 09:13:34 -07:00
niuyulin a76773a402
HBASE-24928 balanceRSGroup should skip generating balance plan for disabled table and splitParent region (#2300)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-08-26 14:01:35 +08:00
stack 5c12b1e310 HBASE-24953 Temporarily up the timeout for branch-2 from 9 hours to 12 for build 2020-08-25 13:10:46 -07:00
Duo Zhang 0be2b6ce55 HBASE-24945 Remove MetaTableAccessor.getRegionCount (#2303)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-25 21:33:15 +08:00
ethanhur a88d118370
HBASE-14847 Add FIFO compaction section to HBase book
Closes #2298

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-25 18:50:11 +05:30
Duo Zhang 4e50269889 HBASE-24944 Remove MetaTableAccessor.getTableRegionsAndLocations in hbase-rest module (#2302)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2020-08-25 16:52:59 +08:00
Duo Zhang e61a3460a7 HBASE-24942 MergeTableRegionsProcedure should not call clean merge region (#2301)
Signed-off-by: stack <stack@apache.org>
2020-08-25 15:15:46 +08:00
Duo Zhang 4455856e9c HBASE-23834 HBase fails to run on Hadoop 3.3.0/3.2.2/3.1.4 due to jetty version mismatch (#2222)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-08-25 15:02:55 +08:00
Michael Stack 4243536b19
HBASE-24885 STUCK RIT by hbck2 assigns (#2283)
Adds region state check on hbck2 assigns/unassigns. Returns pid of -1
if in inappropriate state with logging explaination which suggests
passing override if operator wants to assign/unassign anyways. Here
is an example of what happens now if hbck2 tries an unassign and
Region already unassigned:

  2020-08-19 11:22:06,926 INFO  [RpcServer.default.FPBQ.Fifo.handler=1,queue=0,port=50086] assignment.AssignmentManager(820): Failed {ENCODED => d1112e553991e938b6852f87774c91ee, NAME => 'TestHbck,zzzzz,1597861310769.d1112e553991e938b6852f87774c91ee.', STARTKEY => 'zzzzz', ENDKEY => ''} unassign, override=false; set override to by-pass state checks.
  org.apache.hadoop.hbase.client.DoNotRetryRegionException: Unexpected state for state=CLOSED, location=null, table=TestHbck, region=d1112e553991e938b6852f87774c91ee
          at org.apache.hadoop.hbase.master.assignment.AssignmentManager.preTransitCheck(AssignmentManager.java:583)
          at org.apache.hadoop.hbase.master.assignment.AssignmentManager.createOneUnassignProcedure(AssignmentManager.java:812)
          at org.apache.hadoop.hbase.master.MasterRpcServices.unassigns(MasterRpcServices.java:2616)
          at org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$HbckService$2.callBlockingMethod(MasterProtos.java)
          at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:397)
          at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
          at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
          at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)

Previous it would just create the unassign anyways. Now must pass override
to queue the procedure regardless. Safer.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
 javadoc on assigns/unassigns. Minor refactor in assigns/unassigns to cater to
 case where procedure may come back null (if override not set and fails state checks).

hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
 checkstyle cleanups.
 Clarifying javadoc on how there is no state checking when bulk assigns creating/enabling
 tables.

 createOneAssignProcedure and createOneUnassignProcedure now handle exceptions which now
 can be thrown if no override and region state is not appropriate.

 Aggregation of createAssignProcedure and createUnassignProcedure instances adding in
 region state check invoked if override is NOT set.

hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateNode.java
 Change to setProcedure so it returns passed proc as result instead of void

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-08-24 09:19:43 -07:00
XinSun 0e63b12648 HBASE-24871 Replication may loss data when refresh recovered replication sources (#2249)
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Conflicts:
	hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java
2020-08-24 22:08:26 +08:00
Yechao Chen 836c04d356 HBASE-24811 Use class access static field or method (#2189)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
2020-08-23 11:26:38 +08:00
Zheng Wang 34de350cb3
HBASE-24898 Use EnvironmentEdge.currentTime() instead of System.currentTimeMillis() in CurrentHourProvider
Closes #2272

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-22 15:32:12 +05:30
Michael Stack 43494f6c73 HBASE-24918 Make RegionInfo#UNDEFINED IA.Private (#2289)
Mark RegionInfo#UNDEFINED IA.Private and deprecated;
it is for internal use only and likely to be removed in hbase4.

Move MutableRegionInfo out of RegionInfoBuilder and have it as a
stanadlone task; a nice-to-have.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-08-21 09:26:17 -07:00
Duo Zhang 3d6e64d248 HBASE-24926 Should call setFailure in MergeTableRegionsProcedure when isMergeable returns false (#2290)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-21 22:35:13 +08:00
mokai a914f2efeb
HBASE-24686 [LOG] Log improvement in Connection#close (#2219)
Signed-off-by: Pankaj Kumar <pankajkumar@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-08-21 18:58:05 +05:30
Guanghao Zhang 4667a971b1 HBASE-24912 Enlarge MemstoreFlusherChore/CompactionChecker period for unit test (#2285)
Signed-off-by: stack <stack@apache.org>
2020-08-21 12:42:11 +08:00
Nick Dimiduk acfa08cd87 HBASE-24627 Normalize one table at a time
Introduce an additional method to our Admin interface that allow an
operator to selectivly run the normalizer. The IPC protocol supports
general table name select via compound filter.

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-19 09:22:21 -07:00
Toshihiro Suzuki 00aa3bc9fc
HBASE-24884 BulkLoadHFilesTool/LoadIncrementalHFiles should accept -D options from command line parameters (#2275)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2020-08-19 19:55:34 +09:00
Zheng Wang 7335dbc834
HBASE-24854 Correct the help content of assign and unassign commands in hbase shell
Closes #2241

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-08-13 15:39:55 +05:30
Guanghao Zhang 4bf833418f HBASE-24876 Fix the flaky job url in hbase-personality.sh (#2253)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-08-13 16:52:29 +08:00
Nick Dimiduk 9c85f3641f HBASE-24583 Normalizer can't actually merge empty regions...
when neighbor is larger than average size

* add `testMergeEmptyRegions` to explicitly cover different
  interleaving of 0-sized regions.
* fix bug where merging a 0-size region is skipped due to large
  neighbor.
* remove unused `splitPoint` from `SplitNormalizationPlan`.
* generate `toString`, `hashCode`, and `equals` methods from Apache
  Commons Lang3 template on `SplitNormalizationPlan` and
  `MergeNormalizationPlan`.
* simplify test to use equality matching over `*NormalizationPlan`
  instances as plain pojos.
* test make use of this handy `TableNameTestRule`.
* fix line-length issues in `TestSimpleRegionNormalizer`

Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
Signed-off-by: Aman Poonia <aman.poonia.29@gmail.com>
2020-08-12 10:11:40 -07:00
WenFeiYi 28a05a0f87
HBASE-24844 RecoverableZookeeper#close followed by ExecutorService shutdown
* Resolves Exception on standalone (master) shutdown

Closes #2239

Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-12 17:10:56 +05:30
Viraj Jasani 8ccf643fdc
HBASE-24750 : All ExecutorService should use guava ThreadFactoryBuilder (#2214)
Closes #2196

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Ted Yu <tyu@apache.org>
Signed-off-by: niuyulin <nyl353@163.com>
2020-08-12 15:57:53 +05:30
Duo Zhang 96ea136a8e HBASE-24856 Fix error prone error in FlushTableSubprocedure (#2242)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2020-08-12 10:01:25 +08:00
Nick Dimiduk 8737afb01e HBASE-24843 Sort the constants in `hbase_constants.rb`
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2020-08-11 16:13:42 -07:00
Duo Zhang 7f5f5776b7 HBASE-24841 Change the jenkins job urls in our jenkinsfile (#2227)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-11 09:07:30 +08:00
Bharath Vissapragada 0533937b16
HBASE-24846: Address compaction races in TestFIFOCompactionPolicy (#2203)
Looped through the test 100 times and it passes. Without the patch it fails
every ~10 runs or so.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2020-08-10 17:21:27 -07:00
Bharath Vissapragada cd97d1a452
Revert "HBASE-21905 (addendum): Address compaction races in TestFIFOCompactionPolicy (#2203)"
This reverts commit c372ff19cf.

Reverting to apply as HBASE-24846.
2020-08-10 17:21:05 -07:00
Nick Dimiduk 17a0c2aabf HBASE-24835 Normalizer should log a successful run at INFO level
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-10 10:13:54 -07:00
huaxiangsun bd30555ac8
HBASE-24824 Add more stats in PE for read replica (#2205) (#2212)
Signed-off-by: stack <stack@apache.org>
2020-08-10 09:58:50 -07:00
Toshihiro Suzuki 22bf9a38c9
HBASE-24680 Refactor the checkAndMutate code on the server side (#2184)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Josh Elser <elserj@apache.org>
2020-08-10 18:57:17 +09:00
bsglz ebf493f075
HBASE-24694 Support flush a single column family of table (#2218)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-08-10 10:36:13 +01:00
Duo Zhang aab8ce9e01 HBASE-24838 The pre commit job fails to archive surefire reports (#2224)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-08-10 11:13:56 +08:00
Duo Zhang ef7b9eb36e HBASE-23157 WAL unflushed seqId tracking may wrong when Durability.ASYNC_WAL is used (#762)
Signed-off-by: stack <stack@apache.org>
2020-08-10 10:54:30 +08:00
chenglei 457234c695 HBASE-24625 AsyncFSWAL.getLogFileSizeIfBeingWritten does not return the expected synced file length(addendum) (#2055)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-08-10 10:04:05 +08:00
stack cee7431d0a HBASE-24830 Some tests involving RS crash fail with NullPointerException after HBASE-24632 in branch-2 2020-08-09 11:16:37 -07:00
Viraj Jasani 4a2f3f89f4
HBASE-24788 : TableOutputFormat source compatibility (ADDENDUM)
Brought back InterruptedException in throws clause of getRecordWriter and checkOutputSpecs of TableOutputFormat
2020-08-08 14:30:40 +05:30
Bo Cui 11cfb0f382 HBASE-21721 FSHLog : reduce write#syncs() times (#2217)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Anoop <anoopsamjohn@apache.org>
2020-08-08 09:41:36 +05:30
Josh Elser 303db63b76 HBASE-24779 Report on the WAL edit buffer usage/limit for replication
Closes #2193

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2020-08-07 14:33:30 -04:00
bsglz cb3dd990f7 HBASE-24826 Add some comments for processlist in hbase shell (#2207)
Signed-off-by: Elliot Miller <elliot_miller@apple.com>
Signed-off-by: stack <stack@apple.com>
2020-08-06 19:29:33 -07:00
niuyulin e099aa5e20 HBASE-24659 Calculate FIXED_OVERHEAD automatically (#2018)
Co-authored-by: niuyulin <niuyulin@xiaomi.com>

SIgned-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-08-06 14:52:20 -07:00
Bharath Vissapragada c372ff19cf
HBASE-21905 (addendum): Address compaction races in TestFIFOCompactionPolicy (#2203)
Looped through the test 100 times and it passes. Without the patch it fails
every ~10 runs or so.

Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2020-08-06 09:40:32 -07:00
Michael Stack 9a564dc2bf
HBASE-24817 Allow configuring WALEntry filters on ReplicationSource (#2198)
Allow specifying base WALEntry filter on construction of
ReplicationSource. Add means of being able to filter WALs by name.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java
 Add constructor that allows passing a predicate for filtering *in* WALs
 and a list of filters for filtering *out* WALEntries. The latter was
 hardcoded to filter out system-table WALEntries. The former did not
 exist but we'll need it if Replication takes in more than just the
 default Provider.

Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-06 09:29:08 -07:00
sguggilam 9f62e5645d
HBASE-24823 Port HBASE-22762 Print the delta between phases in the split/merge/compact/flush transaction journals
Closes #2204

Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-08-06 18:51:38 +05:30
gkanade bfa558a0e3 HBASE-24713 RS startup with FSHLog throws NPE after HBASE-21751
writer null check if not initialized yet during syncrunner run (#2201)

Signed-off-by: Ramkrishna <ramkrishna@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: AnoopSamJohn<anoopsamjohn@apache.org>
(cherry picked from commit b0863c5832)
2020-08-05 23:41:13 -05:00
Sean Busbey ec39d59161 Revert "writer null check if not initialized yet during syncrunner run (#2201)"
bad commit message

This reverts commit b0863c5832.
2020-08-05 23:39:18 -05:00