Commit Graph

7330 Commits

Author SHA1 Message Date
Andor Molnár 9a6494f02f HBASE-22759 Extended grant and revoke audit events with caller info - ADDENDUM
Added remote address to grant/revoke audit log messages
2019-08-06 11:44:13 +02:00
stack 94e35d8cdf HBASE-22771 [HBCK2] fixMeta method and server-side support; ADDENDUM 2019-08-05 22:56:21 -07:00
stack db440d3099 HBASE-22771 [HBCK2] fixMeta method and server-side support
This is a first cut at this patch. Implements hold fixing only
currently.

Add a fixMeta method to Hbck Interface.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
 Bug fix. If hole is on end of last table, I wasn't seeing it.

A hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetaFixer.java
 Add a general meta fixer class. Explains up top why this stuff doesn't
 belong inside MetaTableAccessor.

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
 Break out the filesystem messing so don't have to copy it nor do more
 than is needed doing fixup for Region holes.

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionFileSystem.java
 Change behavious slightly. If directory exists, don't fail as we did
 but try and keep going and create .regioninfo file if missing (or
 overwrite if in place). This should make it idempotent. Can rerun
 command. Lets see if any repercussions in test suite.

A hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMetaFixer.java
 Add test.

Signed-off-by: Zheng Hu <openinx@gmail.com>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2019-08-05 22:51:53 -07:00
Duo Zhang fe2409971c HBASE-22539 WAL corruption due to early DBBs re-use when Durability.ASYNC_WAL is used (#437)
Signed-off-by: Zheng Hu <openinx@gmail.com>
2019-08-05 16:52:07 +08:00
Guanghao Zhang e68b16a6c1 HBASE-22737 Add a new admin method and shell cmd to trigger the hbck chore to run (#425)
Signed-off-by: stack <stack@apache.org>
2019-08-01 10:43:57 +08:00
Biju Nair c8ab562338 HBASE-22363 Remove hardcoded number of read cache block buckets
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-07-31 09:26:32 +02:00
stack 9799e0b6b2 HBASE-22741 Show catalogjanitor consistency complaints in new 'HBCK Report' page
Signed-off-by: huzheng <openinx@gmail.com>
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2019-07-30 11:32:46 -07:00
stack 4587b39e63 Revert "Revert "HBASE-22723 Have CatalogJanitor report holes and overlaps; i.e. problems it sees when doing its regular scan of hbase:meta""
This reverts commit bf36bdf2b6.
2019-07-30 09:21:47 -07:00
openinx 892aa832ad HBASE-22758 Remove the unneccesary info cf deletion in DeleteTableProcedure#deleteFromMeta (#424) 2019-07-30 21:48:46 +08:00
Andor Molnár 29f8e195ef HBASE-22759. Extended grant and revoke audit events with caller info (#427) 2019-07-30 12:50:57 +02:00
stack efead31eb2 HBASE-22751 table.jsp fails if ugly regions in table 2019-07-30 10:30:27 +02:00
Peter Somogyi bf36bdf2b6 Revert "HBASE-22723 Have CatalogJanitor report holes and overlaps; i.e. problems it sees when doing its regular scan of hbase:meta"
This reverts commit 65a6b270dd.
2019-07-30 10:12:27 +02:00
Peter Somogyi dcc2a2b8d8 Revert "HBASE-22751 table.jsp fails if ugly regions in table"
This reverts commit d45b2d4469.
2019-07-30 10:12:03 +02:00
stack 65a6b270dd HBASE-22723 Have CatalogJanitor report holes and overlaps; i.e. problems it sees when doing its regular scan of hbase:meta
Refactor of CatalogJanitor so it generates a
Report on the state of hbase:meta when it runs. Also
refactor so CJ runs even if RIT (previous it would
punt on running if RIT) so it can generate a 'Report'
on the interval regardless. If RIT, it just doesn't
go on to do the merge/split GC as it used to.

If report finds an issue, dump as a WARN message
to the master log.

Follow-on is to make the Report actionable/available
for the Master to pull when it goes to draw the hbck
UI page (could also consider shipping the Report as
part of ClusterMetrics?)

Adds new, fatter Visitor to CJ, one that generates
Report on each run keeping around more findings as
it runs.

Moved some methods around so class reads better;
previous methods were randomly ordered in the class.

M hbase-client/src/main/java/org/apache/hadoop/hbase/MetaTableAccessor.java
 Make a few handy methods public.

M hbase-client/src/main/java/org/apache/hadoop/hbase/client/RegionInfo.java
 Add utility as defaults on the Inteface; i.e. is this the first region
 in table, is it last, does a passed region come next, or does passed
 region overlap this region (added tests for this new stuff).

M hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java
 Bugfix... handle case where buffer passed is null.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
 Lots of change, reorg., but mostly adding consistency checking
 to the visitor used scanning hbase:meta on a period and the
 generation of a Report on what the scan has found traversing
 hbase:meta. Added a main so could try the CatalogJanitor against
 a running cluster.

A hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitorCluster.java
 Fat ugly test for CatalogJanitor consistency checking.

M hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionInfo.java
 Add tests for new functionality in RI.

M hbase-shell/src/main/ruby/hbase/table.rb
 Bug fix for case where meta has a null regioninfo; scan was aborting.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2019-07-29 16:21:05 -07:00
stack d45b2d4469 HBASE-22751 table.jsp fails if ugly regions in table 2019-07-29 15:15:32 -07:00
Guanghao Zhang e54c4018e7 HBASE-22709 Add a chore thread in master to do hbck checking (#404)
Signed-off-by: stack <stack@apache.org>
2019-07-27 18:55:22 +08:00
Guanghao Zhang f8b592e9f8 HBASE-22742 [HBCK2] Add more log for hbck operations at master side
Signed-off-by: stack <stack@apache.org>
2019-07-27 12:41:54 +08:00
stack fd267168d8 HBASE-22382 Refactor tests in TestFromClientSide; ADDENDUM to fix tests 2019-07-26 13:42:45 -07:00
linkaline 100a625324 HBASE-22733 TestSplitTransactionOnCluster.testMasterRestartAtRegionSplitPendingCatalogJanitor is flakey (#408)
Signed-off-by: stack <stack@apache.org>
2019-07-25 11:16:16 -07:00
Andor Molnár 884843d085 HBASE-22382 Refactor tests in TestFromClientSide (#385)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-07-25 10:41:31 +08:00
Guanghao Zhang 56417284bf HBASE-22721 Refactor HBaseFsck: move the inner class out
Signed-off-by: stack <stack@apache.org>
2019-07-25 08:32:19 +08:00
Jeongdae Kim d88ee0447a
HBASE-22715 All scan requests should be handled by scan handler threads in RWQueueRpcExecutor (#393)
Signed-off-by: Andrew Purtell <apurtell@apache.org>

Conflicts:
	hbase-server/src/test/java/org/apache/hadoop/hbase/ipc/TestSimpleRpcScheduler.java
2019-07-24 11:56:48 -07:00
syedmurtazahassan 6a46e071c5
HBASE-22603 Javadoc Warnings related to @link tag
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2019-07-23 14:46:24 +02:00
syedmurtazahassan 49fcd92bdf HBASE-22610 [BucketCache] Rename hbase.offheapcache.minblocksize
Co-authored-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
2019-07-23 12:05:51 +08:00
Toshihiro Suzuki 03f9fc3aa8 HBASE-22690 Deprecate / Remove OfflineMetaRepair in hbase-2+
Signed-off-by: stack <stack@apache.org>
2019-07-23 10:22:54 +09:00
Guanghao Zhang 727ebbddac HBASE-22527 [hbck2] Add a master web ui to show the problematic regions 2019-07-18 08:11:06 -07:00
sreenivasulureddy df2cdc8e5d HBASE-22537 Split happened Replica region can not be deleted after deleting table successfully and restarting RegionServer
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2019-07-18 15:29:35 +01:00
Guanghao c5af92b644 HBASE-22704 Avoid NPE when access table.jsp and snapshot.jsp but master not finish initialization (#389) 2019-07-18 08:45:51 +08:00
stack fa35bbc0d2 HBASE-7191 HBCK - Add offline create/fix hbase.version and hbase.id
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-07-17 08:01:12 -07:00
Viraj Jasani 8b37958161
HBASE-22643 : Delete region without archiving only if regiondir is present
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Xu Cang <xucang@apache.org>
2019-07-16 17:52:56 -07:00
Duo Zhang 1092533308 HBASE-22684 The log rolling request maybe canceled immediately in LogRoller due to a race (#378)
Signed-off-by: Michael Stack <stack@apache.org>
2019-07-15 22:14:08 +08:00
syedmurtazahassan d26e9d04e1
HBASE-22586 Javadoc Warnings related to @param tag
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2019-07-13 17:24:57 +02:00
Duo Zhang 421ae2a688 HBASE-22681 The 'assert highestUnsyncedTxid < entry.getTxid();' in AbstractFWAL.append may fail when using AsyncFSWAL (#376)
Signed-off-by: Zheng Hu <openinx@gmail.com>
2019-07-13 22:16:46 +08:00
syedmurtazahassan 30e874ae58
HBASE-22571 Javadoc Warnings related to @return tag
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2019-07-13 16:01:32 +02:00
Andrew Purtell 5fc84844b3
HBASE-22686 ZkSplitLogWorkerCoordination doesn't allow a regionserver to pick up all of the split work it is capable of (#377)
Signed-off-by: Xu Cang <xcang@apache.org>

Conflicts:
	hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitLogWorker.java
2019-07-12 18:13:15 -07:00
Toshihiro Suzuki 8ec797d08e HBASE-19893 restore_snapshot is broken in master branch when region splits
Signed-off-by: Sakthi <sakthivel.azhaku@gmail.com>
Signed-off-by: stack <stack@apache.org>
2019-07-07 20:45:47 -07:00
Reid Chan 1fb3fa22fb HBASE-22656 [Metrics] Table metrics 'BatchPut' and 'BatchDelete' are never updated
Signed-off-by: Michael Stack <stack@apache.org>
2019-07-07 23:33:03 +08:00
openinx 6d5e120596 HBASE-22582 The Compaction writer may access the lastCell whose memory has been released when appending fileInfo in the final (#341) 2019-07-04 14:50:26 +08:00
Andor Molnár 35521d4a2f HBASE-13798 TestFromClientSide* don't close the Table 2019-07-02 13:00:20 +02:00
Mate Szalay-Beko 6205a6c8b3 HBASE-22637 fix flaky test in TestMetaTableMetrics
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Signed-off-by: Sakthi <sakthivel.azhaku@gmail.com>
2019-07-02 09:46:21 +02:00
Bing Xiao a2a929f488 HBASE-21751 WAL creation fails during region open may cause region assign forever fail
Signed-off-by: Allan Yang <allan163@apache.org>
    Signed-off-by: Michael Stack <stack@apache.org>
2019-07-01 21:54:00 -07:00
Bing Xiao 963cb45f1c HBASE-22169 Open region failed cause memory leak
Signed-off-by: stack <stack@apache.org>
2019-07-01 13:13:19 -07:00
haxiaolin 13d5a67094 HBASE-20368 Fix RIT stuck when a rsgroup has no online servers but AM's pendingAssginQueue is cleared 2019-07-01 13:01:36 -07:00
Duo Zhang 60097a6467 HBASE-22632 SplitTableRegionProcedure and MergeTableRegionsProcedure should skip store files for unknown column families (#344)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2019-06-27 22:16:46 +08:00
Duo Zhang a172b480fe HBASE-22617 Recovered WAL directories not getting cleaned up (#330)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2019-06-25 18:44:59 +08:00
Andrew Purtell a1aab9514f
HBASE-22616 responseTooXXX logging for Multi should characterize the component ops (#329)
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Michael Stack <stack@apache.org>
2019-06-24 18:05:51 -07:00
virajjasani 332575fa22
HBASE-22520 Avoid possible NPE while performing seekBefore in Hal… (#281)
HBASE-22520 Avoid possible NPE while performing seekBefore in HalfStoreFileReader
2019-06-20 15:02:34 -07:00
Reid Chan 060f52de5a HBASE-22596 [Chore] Separate the execution period between CompactionChecker and PeriodicMemStoreFlusher
Signed-off-by: Zach York <zyork@apache.org>
Signed-off-by: Xu Cang <xucang@apache.org>
2019-06-20 11:02:10 +08:00
Artem Ervits ec6638c902 HBASE-22561 modify HFilePrettyPrinter to accept non-rootdir directories
Signed-off-by: huzheng <openinx@gmail.com>
2019-06-19 15:35:18 +08:00
Jan Hentschel 3f62e481e2
HBASE-22344 Documented the deprecation of public and limited private APIs (#208)
Signed-off-by: stack <stack@apache.org>
2019-06-17 23:10:24 +02:00
Istvan Toth 02f9c8b3b4 HBASE-22581 user with "CREATE" permission can grant, but not revoke permissions on created table
Signed-off-by: Reid Chan <reidchan@apache.org>
2019-06-17 23:59:02 +08:00
Murtaza Hassan cf63fbac05
HBASE-22565 Javadoc Warnings: @see cannot be used in inline documentation 2019-06-14 11:50:33 +02:00
Andrew Purtell b355020224
HBASE-22530 The metrics of store files count of region are returned to clients incorrectly (Eungsop Yoo)
Signed-off-by: Xu Cang <xucang@apache.org>
2019-06-13 18:18:36 -07:00
Josh Elser 3d64cf1203 HBASE-22562 Remove dead code: skipControl
Signed-off-by: Reid Chan <reidchan@apache.org>
2019-06-13 15:53:29 -04:00
zhangduo 021141a941 HBASE-22551 TestMasterOperationsForRegionReplicas is flakey 2019-06-10 08:10:58 +08:00
WenFeiYi 5ae8bc4974 HBASE-22284 optimization StringBuilder.append of AbstractMemStore.toString #182 2019-06-09 14:28:27 -07:00
Artem Ervits b3a63fe962 HBASE-21536 Fix completebulkload usage instructions for branch-2.1
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-06-06 19:45:57 +02:00
stack d8438938cd HBASE-22546 TestRegionServerHostname#testRegionServerHostname fails reliably for me 2019-06-05 20:51:33 -07:00
Daisuke Kobayashi 5b3c954498 HBASE-22160 Add sorting functionality in regionserver web UI for user
Signed-off-by: Toshihiro Suzuki <brfrn169@gmail.com>
Signed-off-by: Wellington Chevreuil <wellington.chevreuil@gmail.com>
2019-06-05 11:08:39 +09:00
Sakthi 44c3a63160 HBASE-22536 TestForeignExceptionSerialization fails when run on JDK11
Signed-off-by: Sean Busbey <busbey@apache.org>
(cherry picked from commit 2280c8f4a88350278447515deb0811c42d0bb476)
2019-06-04 11:30:03 -05:00
Guanghao b64fdd956c HBASE-22523 Refactor RegionStates#getAssignmentsByTable to make it easy to understand 2019-06-02 22:11:25 +08:00
Josh Elser c5bf670562 HBASE-22511 Add some more rs-status paths to RS UI links
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-05-31 14:49:13 -04:00
Clay Baenziger 9029130ba0 HBASE-22487 getMostLoadedRegions is unused
Signed-off-by: Wellington Chevreuil <wellington.chevreuil@gmail.com>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-05-30 16:15:14 +02:00
Guanghao 3c848de784 HBASE-22488 Cleanup the explicit timeout value for test methods 2019-05-29 15:17:54 +08:00
Guanghao 21241d03d9 HBASE-22486 Fix flaky test TestLockManager 2019-05-29 15:16:21 +08:00
stack bbf01b6074 Preparing development version 2.1.6-SNAPSHOT 2019-05-28 22:07:03 +00:00
stack 0ba7cc01a1 Preparing HBase release 2.1.5RC1; tagging and updates to CHANGES.md and RELEASENOTES.md 2019-05-28 22:06:49 +00:00
Josh Elser ba9609afbd HBASE-22467 UI fixes to enable Knox proxying
Closes #261

Signed-off-by: Sean Busbey <busbey@apache.org>
2019-05-28 16:31:54 -04:00
binlijin 8e9b504ffe HBASE-22441 BucketCache NullPointerException in cacheBlock 2019-05-27 17:15:57 +08:00
Sakthi 6d1b4f7d35 HBASE-21991 [Addendum] Mark LossCounting as Private
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2019-05-25 23:48:21 +02:00
Puleya7 4b7e256cc6 HBASE-22440. Override getWalGroupsReplicationStatus to avoid NPE
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-05-23 13:42:14 +08:00
stack 96175c7f51 Preparing development version 2.1.6-SNAPSHOT 2019-05-22 22:27:50 +00:00
stack 5a13560d24 Preparing HBase release 2.1.5RC0; tagging and updates to CHANGES.md and RELEASENOTES.md 2019-05-22 22:27:35 +00:00
binlijin b4a08f5baa HBASE-22445 Add file info when throw exceptions in HFileReaderImpl 2019-05-22 20:24:03 +08:00
binlijin 81dc712779 HBASE-22447 Check refCount before free block in BucketCache 2019-05-22 16:26:43 +08:00
Andrew Purtell 5b24d4e1bc
HBASE-22449 https everywhere in Maven metadata (#247) 2019-05-21 12:38:26 -07:00
Sergey Soldatov 9a7293c174 HBASE-22289 WAL-based log splitting resubmit threshold may result in a task being stuck forever 2019-05-20 09:23:39 -07:00
stack c5d5cd3ff2 Revert "HBASE-22169 Open region failed cause memory leak"
This reverts commit 9363a4a6dfc0266d322c4c216c4c6c5cfbf8bb5.
2019-05-17 16:53:58 -07:00
Bing Xiao 9363a4a6df HBASE-22169 Open region failed cause memory leak
Signed-off-by: stack <stack@apache.org>
2019-05-17 13:43:52 -07:00
Wellington Chevreuil 696298bfac
HBASE-21784 Dump replication queue should show list of wal files ordered chronologically
Change-Id: I18c372406290e2b1e2b5503e2c87adcb9bf6fe91

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2019-05-14 14:40:03 -07:00
zhangduo e0f8f9b886 HBASE-21658 Addendum fix infinite wait when there are no meta locations yet 2019-05-14 08:13:53 +08:00
Caroline Zhou 00efac9361
HBASE-22378. HBase Canary fails with TableNotFoundException when table deleted during Canary run
Signed-off-by: Xu Cang <xucang@apache.org>
2019-05-13 12:36:40 -07:00
Puleya7 97db64d413
HBASE-22392. Remove extra/useless +
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2019-05-12 21:44:38 +02:00
zhangduo f5bd41b762 HBASE-21658 Should get the meta replica number from zk instead of config at client side 2019-05-11 09:31:58 +08:00
Xu Cang 0132407c3e
HBASE-22274 Cell size limit check on append considers cell's previous size 2019-05-09 23:24:23 -07:00
chenyang 4eab21b2f0 HBASE-22324 loss a mass of data when the sequenceId of cells greater than Integer.Max
Signed-off-by: huzheng <openinx@gmail.com>
2019-05-09 15:49:28 +08:00
Andrew Purtell ad35c224e1
HBASE-22375 Promote AccessChecker to LimitedPrivate(Coprocessor) 2019-05-08 18:16:54 -07:00
ramkrish86 363647243c HBASE-22072 High read/write intensive regions may cause long crash (#214)
* HBASE-22072 High read/write intensive regions may cause long crash
recovery

* Make the 'closing' variable as volatile and move the test case to
standlone class
2019-05-08 15:50:26 +05:30
zhangduo a4381802c6 HBASE-22190 SnapshotFileCache may fail to load the correct snapshot file list when there is an on-going snapshot operation 2019-05-04 13:15:20 +08:00
Sergey Shelukhin 7269aa3743 HBASE-22354 master never sets abortRequested, and thus abort timeout doesn't work for it
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-05-03 09:08:15 +02:00
zhangduo 61f4cd7406 HBASE-22350 Rewrite TestClientOperationTimeout so we do not timeout when creating table 2019-05-02 17:38:39 +02:00
Sakthi 2c9077ae51 HBASE-22054: Space Quota: Compaction is not working for super user in case of NO_WRITES_COMPACTIONS
Signed-off-by: Josh Elser <elserj@apache.org>
2019-04-30 14:40:51 -04:00
zhangduo 71ea3237ce HBASE-22236 AsyncNonMetaRegionLocator should not cache HRegionLocation with null location 2019-04-30 17:24:43 +08:00
chfeng e0cc651098
HBASE-21815 Make isTrackingMetrics and getMetrics of ScannerContext public
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2019-04-29 13:03:05 -07:00
Sakthi 2d78dfb2ed HBASE-22086: Space Quota issue: Deleting snapshot doesn't update the usage of table
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-04-29 11:47:06 -04:00
Zach York 11843b7010
HBASE-22291 Fix recovery of recovered.edits files under root dir
When trying to recover a recovered.edits file under the root directory,
it is possible to get a WrongFS exception.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2019-04-25 18:11:31 -07:00
Jan Hentschel 57cd5ae5aa HBASE-22231 Removed unused and '*' import 2019-04-24 09:09:11 +02:00
stack 899610e657 HBASE-22298 branch-2.2 nightly fails "[ForOverride] Method annotated @ForOverride must have protected or package-private visibility" 2019-04-23 14:02:15 -07:00
Wellington Chevreuil aaa2f50ae1 HBASE-22200 - WALSplitter.hasRecoveredEdits should use same FS instance from WAL region dir
Signed-off-by: Zach York <zyork@apache.org>
2019-04-22 15:19:29 -07:00
Andrew Purtell 6ac2437485
HBASE-22235 OperationStatus.{SUCCESS|FAILURE|NOT_RUN} are not visible to 3rd party coprocessors 2019-04-16 13:18:57 -07:00