Commit Graph

6225 Commits

Author SHA1 Message Date
Michael Stack 619dcf24f4 Revert "HBASE-18229: create new Async Split API to embrace AM v2"
TestShell is failing.

This reverts commit 290fb8965d.
2017-06-30 03:30:50 -07:00
Michael Stack 926781540c HBASE-16192 Fix the potential problems in TestAcidGuarantees (Colin Ma) 2017-06-30 03:17:35 -07:00
Yi Liang 290fb8965d HBASE-18229: create new Async Split API to embrace AM v2
Signed-off-by: Michael Stack <stack@apache.org>
2017-06-29 16:21:08 -07:00
张世彬10204932 f6dd43be70 HBASE-17982 correct spelling error of 'occured'
Signed-off-by: Michael Stack <stack@apache.org>
2017-06-29 15:10:08 -07:00
Umesh Agashe 7e7917ed6d HBASE-18292 Fixed flaky test hbase.master.locking.TestLockProcedure#testLocalMasterLockRecovery
Signed-off-by: Michael Stack <stack@apache.org>
2017-06-28 21:33:15 -07:00
Umesh Agashe 15bda5d5d4 HBASE-18278 Enable and Fix for unit test hbase.master.procedure.TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta
Signed-off-by: Michael Stack <stack@apache.org>
2017-06-28 16:02:19 -07:00
Ben-Epstein 1c9f9ed06a HBASE-18281 created private static pattern matcher for performance
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-06-28 11:32:57 -07:00
Kahlil Oppenheimer 812b383325 HBASE-18164 Fast locality computation in balancer - addendum handles NaN
Signed-off-by: tedyu <yuzhihong@gmail.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
2017-06-27 14:58:03 -05:00
Sean Busbey 0ad8f26662 Revert "HBASE-18164 Fast locality computation in balancer - addendum handles NaN"
This reverts commit b4c3fe9763.

early commit missed some review feedback.
2017-06-27 14:58:03 -05:00
tedyu d5e206dfa0 HBASE-18161 Incremental Load support for Multiple-Table HFileOutputFormat (Densel Santhmayor) 2017-06-27 12:33:39 -07:00
张世彬10204932 4746a1c92b HBASE-18265 Correct the link unuseful in regionServer's region state UI
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-06-27 10:39:43 +08:00
Kahlil Oppenheimer b4c3fe9763 HBASE-18164 Fast locality computation in balancer - addendum handles NaN
-Added new LocalityCostFunction and LocalityCandidateGenerator that
cache localities of every region/rack combination and mappings of every
region to its most local server and to its most local rack.

-Made LocalityCostFunction incremental so that it only computes locality
based on most recent region moves/swaps, rather than recomputing the
locality of every region in the cluster at every iteration of the
balancer

-Changed locality cost function to reflect the ratio of:
(Current locality) / (Best locality possible given current cluster)

Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-06-26 12:39:51 -07:00
Guanghao Zhang 289938337b HBASE-18234 Revisit the async admin api 2017-06-26 17:30:47 +08:00
David Harju d9dd319390 HBASE-18023 Log multi-* requests for more than threshold number of rows
Signed-off-by: Josh Elser <elserj@apache.org>
2017-06-24 15:34:02 -04:00
张世彬10204932 a4c26526c0 HBASE-18263 Resolve NPE in backup Master UI when accessing procedures.jsp
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-06-24 07:16:27 -07:00
Ramkrishna 49825aec6b HBASE-18221 Switch from pread to stream should happen under HStore's
reentrant lock (Ram)
2017-06-23 10:34:10 +05:30
Umesh Agashe 0357c2266a HBASE-18254 ServerCrashProcedure checks and waits for meta initialized, instead should check and wait for meta loaded
After enabling test hbase.master.procedure.TestServerCrashProcedure#testRecoveryAndDoubleExecutionOnRsWithMeta, this bug is found in ServerCrashProcedure

Signed-off-by: Michael Stack <stack@apache.org>
2017-06-21 21:58:15 -07:00
Andrew Purtell ff5d497310 HBASE-18235 LoadBalancer.BOGUS_SERVER_NAME should not have a bogus hostname
We deliberately use 'localhost' instead of a bogus hostname for
LoadBalancer.BOGUS_SERVER_NAME so the operation will fail fast.
2017-06-21 14:38:38 -07:00
QilinCao af25940b25 HBASE-18252 Resolve BaseLoadBalancer bad practice warnings
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-06-21 10:12:57 -07:00
tedyu 3ff84a2da9 HBASE-18226 Disable reverse DNS lookup at HMaster and use the hostname provided by RegionServer (Duo Xu) 2017-06-21 08:15:27 -07:00
Ashish Singhi 0f5a250ac2 HBASE-18212 reduce log level for unbuffer warning.
In Standalone mode with local filesystem HBase logs Warning message:Failed to invoke 'unbuffer' method in class org.apache.hadoop.fs.FSDataInputStream

Signed-off-by: Umesh Agashe <uagashe@cloudera.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
2017-06-20 01:11:17 -05:00
Kahlil Oppenheimer 3a79590bb4 HBASE-18164 Fast locality computation in balancer
-Added new LocalityCostFunction and LocalityCandidateGenerator that
cache localities of every region/rack combination and mappings of every
region to its most local server and to its most local rack.

-Made LocalityCostFunction incremental so that it only computes locality
based on most recent region moves/swaps, rather than recomputing the
locality of every region in the cluster at every iteration of the
balancer

-Changed locality cost function to reflect the ratio of:
(Current locality) / (Best locality possible given current cluster)

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-06-20 01:11:08 -05:00
Pankaj Kumar cfa1bd3265 HBASE-18180 Possible connection leak while closing BufferedMutator in TableOutputFormat
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-06-19 02:02:54 -07:00
Guanghao Zhang 74c6f44877 HBASE-18170 Refactor ReplicationSourceWALReaderThread 2017-06-19 09:34:23 +08:00
Umesh Agashe fa93c0f59a HBASE-18104 AMv2: Enabled aggregation of RPCs
Unit test (TestAssignmentManager) uses mock which always aggregates. So added trace level log message and verified manually on a single node cluster.

Signed-off-by: Michael Stack <stack@apache.org>
2017-06-16 23:54:11 -07:00
Umesh Agashe 5db63014b0 HBASE-18227 Fixed unit test hbase.coprocessor.TestCoprocessorMetrics#testRegionObserverAfterRegionClosed
Calling closeRegion() directly on remote server is not supported post-AMv2. Calling unassign() on master

Signed-off-by: Michael Stack <stack@apache.org>
2017-06-16 20:19:59 -07:00
Josh Elser 2ef80ab39a HBASE-18225 Avoid toString() on an array 2017-06-16 08:55:14 -07:00
Ramkrishna 7c15e57005 HBASE-18220 - Addendum as per Duo suggestion for readability (Ram) 2017-06-16 12:35:38 +05:30
Ramkrishna ac2018e276 HBASE-18220 Compaction scanners need not reopen storefile scanners while
trying to switch over from pread to stream (Ram)
2017-06-16 11:04:44 +05:30
Michael Stack 4184ae7563 HBASE-18004 getRegionLocations needs to be called once in
ScannerCallableWithReplicas#call() (Huaxiang Sun)
2017-06-15 13:41:49 -07:00
Michael Stack 8c7bf7b0a9 HBASE-18166 [AMv2] We are splitting already-split files v2 Address Stephen Jiang reivew comments; ADDENDUM TO FIX COMPILE 2017-06-15 11:40:38 -07:00
Michael Stack c02a142143 HBASE-18166 [AMv2] We are splitting already-split files v2 Address Stephen Jiang reivew comments 2017-06-15 10:27:49 -07:00
tedyu 299850ea70 HBASE-18209 Include httpclient / httpcore jars in build artifacts 2017-06-14 21:00:12 -07:00
Andrew Purtell 6af3ee44c0 HBASE-18219 Fix typo in constant HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT 2017-06-14 16:03:50 -07:00
Vincent 385b792446 HBASE-18137 Replication gets stuck for empty WALs
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-06-10 12:45:40 -07:00
Ashu Pachauri 1aedc07b52 HBASE-18192: Replication drops recovered queues on region server shutdown
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-06-09 19:53:31 -07:00
Ashu Pachauri 1e4f8491f7 HBASE-18092: Removing a peer does not properly clean up the ReplicationSourceManager state and metrics
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-06-09 08:23:41 -07:00
Chia-Ping Tsai 922894c96e HBASE-18193 Master web UI presents the incorrect number of regions 2017-06-09 14:45:20 +08:00
Umesh Agashe a81577d827 HBASE-18195 Removed redundant single quote from start message for HMaster and HRegionServer
Signed-off-by: Michael Stack <stack@apache.org>
2017-06-08 22:21:25 -07:00
Gary Helmling 17966525e9 HBASE-18141 Regionserver fails to shutdown when abort triggered during RPC call 2017-06-08 17:30:20 -07:00
Yi Liang ea7d51e129 HBASE-18109: Assign system tables first
This issue adds comments and a sort so system tables are queued first
(which will ensure they go out first). This should be good enough
along w/ existing scheduling mechanisms to ensure system/meta get
assigned first.

Signed-off-by: Michael Stack <stack@apache.org>
2017-06-08 13:26:42 -07:00
Chia-Ping Tsai 1e7804634c HBASE-18008 Any HColumnDescriptor we give out should be immutable 2017-06-08 23:28:33 +08:00
Alex Araujo c830a0f47f HBASE-18184 Add hbase-hadoop2-compat jar as MapReduce dependency
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-06-07 21:58:17 -07:00
Michael Stack adfb48eeb8 HBASE-18190 Set version in branch-2 to 2.0.0-alpha-1 2017-06-07 21:10:01 -07:00
Chia-Ping Tsai 9c8c749cd3 HBASE-18158 Two running in-memory compaction threads may lose data for flushing 2017-06-07 18:04:30 +08:00
Ramkrishna 9329a18c2d HBASE-17849 PE tool random read is not totally random (Ram) 2017-06-07 11:31:30 +05:30
huzheng 0d0c330401 HBASE-17678 FilterList with MUST_PASS_ONE may lead to redundant cells returned
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-06-06 21:08:12 -07:00
Phil Yang 2f1923a823 HBASE-15576 Scanning cursor to prevent blocking long time on ResultScanner.next() 2017-06-07 11:32:04 +08:00
tedyu 80e15aac21 HBASE-16392 Backup delete fault tolerance (Vladimir Rodionov) 2017-06-06 20:29:13 -07:00
Chia-Ping Tsai da3c023635 HBASE-18145 The flush may cause the corrupt data for reading
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-06-06 18:02:43 -07:00
Andrew Purtell 858bccfcb8 HBASE-18132 Low replication should be checked in period in case of datanode rolling upgrade (Allan Yang) 2017-06-06 17:21:21 -07:00
Ashish Singhi 1950acc67a HBASE-9393 Hbase does not closing a closed socket resulting in many CLOSE_WAIT
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-06-06 12:52:46 -07:00
zhangduo ee0f148c73 HBASE-18038 Rename StoreFile to HStoreFile and add a StoreFile interface for CP 2017-06-06 20:36:38 +08:00
Umesh Agashe 07c38e7165 HBASE-16549 Added new metrics for AMv2 procedures
Following AMv2 procedures are modified to override onSubmit(), onFinish() hooks provided by HBASE-17888 to do
metrics calculations when procedures are submitted and finshed:
* AssignProcedure
* UnassignProcedure
* MergeTableRegionProcedure
* SplitTableRegionProcedure
* ServerCrashProcedure

Following metrics is collected for each of the above procedure during lifetime of a process:
* Total number of requests submitted for a type of procedure
* Histogram of runtime in milliseconds for successfully completed procedures
* Total number of failed procedures

As we are moving away from Hadoop's metric2, hbase-metrics-api module is used for newly added metrics.

Modified existing tests to verify count of procedures.

Signed-off-by: Michael Stack <stack@apache.org>
2017-06-05 17:14:14 -07:00
Michael Stack e65d8653e5 HBASE-18155 TestMasterProcedureWalLease is flakey 2017-06-03 12:55:18 -07:00
Enis Soztutar 118429cbac HBASE-15160 Put back HFile's HDFS op latency sampling code and add metrics for monitoring (Yu Li and Enis Soztutar) 2017-06-02 17:41:53 -07:00
tedyu ef46debde8 HBASE-18005 read replica: handle the case that region server hosting both primary replica and meta region is down (huaxiang sun) 2017-06-02 09:29:51 -07:00
Guanghao Zhang 549171465d HBASE-18111 Replication stuck when cluster connection is closed
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-06-01 15:12:15 -07:00
Michael Stack e1f3c89b3b HBASE-18143 [AMv2] Backoff on failed report of region transition quickly goes to astronomical time scale
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
  Rather than compound the pause time, just have backoff multiple the
  original INIT_PAUSE_TIME_MS so we go 1, 2, 5, 10, ... etc. rather than
  1, 2, 30, 600... and so on.

  Minor fixup around logging so report of failed transition is no longer
  reported as trace-level.
2017-06-01 13:01:36 -07:00
Jerry He c7a7f880dd HBASE-16261 MultiHFileOutputFormat Enhancement (Yi Liang) 2017-06-01 10:44:17 -07:00
Guanghao Zhang 123086edad HBASE-18130 Refactor ReplicationSource 2017-06-01 14:50:45 +08:00
Chinmay Kulkarni db8ce0566d HBASE 17959 Canary timeout should be configurable on a per-table basis
Added support for configuring read/write timeouts on a per-table basis
when in region mode.
Added unit test for per-table timeout checks.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-05-31 17:58:43 -07:00
Michael Stack 3975bbd008 HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi) Move to a new AssignmentManager, one that describes Assignment using a State Machine built on top of ProcedureV2 facility.
This doc. keeps state on where we are at w/ the new AM:
https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.vfdoxqut9lqn
Includes list of tests disabled by this patch with reasons why.

Based on patches from Matteos' repository and then fix up to get it all to pass cluster
tests, filling in some missing functionality, fix of findbugs, fixing bugs, etc..
including:

    1. HBASE-14616 Procedure v2 - Replace the old AM with the new AM.
    The basis comes from Matteo's repo here:
    689227fcbf

    Patch replaces old AM with the new under subpackage master.assignment.
    Mostly just updating classes to use new AM -- import changes -- rather
    than the old. It also removes old AM and supporting classes.
    See below for more detail.

    2. HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi)
    3622cba4e3

    Adds running of remote procedure. Adds batching of remote calls.
    Adds support for assign/unassign in procedures. Adds version info
    reporting in rpc. Adds start of an AMv2.

    3. Reporting of remote RS version is from here:
    ddb4df3964.patch

    4. And remote dispatch of procedures is from:
    186b9e7c4d

    5. The split merge patches from here are also melded in:
    9a3a95a2c2
    and d6289307a0

We add testing util for new AM and new sets of tests.

Does a bunch of fixup on logging so its possible to follow a procedures' narrative by grepping
procedure id. We spewed loads of log too on big transitions such as master fail; fixed.

Fix CatalogTracker. Make it use Procedures doing clean up of Region data on split/merge.
Without these changes, ITBLL was failing at larger scale (3-4hours 5B rows) because we were
splitting split Regions among other things (CJ would run but wasn't
taking lock on Regions so havoc).

    Added a bunch of doc. on Procedure primitives.

    Added new region-based state machine base class. Moved region-based
    state machines on to it.

    Found bugs in the way procedure locking was doing in a few of the
    region-based Procedures. Having them all have same subclass helps here.

    Added isSplittable and isMergeable to the Region Interface.

    Master would split/merge even though the Regions still had
    references. Fixed it so Master asks RegionServer if Region
    is splittable.

    Messing more w/ logging. Made all procedures log the same and report
    the state the same; helps when logging is regular.

    Rewrote TestCatalogTracker. Enabled TestMergeTableRegionProcedure.

    Added more functionality to MockMasterServices so can use it doing
    standalone testing of Procedures (made TestCatalogTracker use it
    instead of its own version).

    Add to MasterServices ability to wait on Master being up -- makes
    it so can Mock Master and start to implement standalone split testing.
    Start in on a Split region standalone test in TestAM.

    Fix bug where a Split can fail because it comes in in the middle of
    a Move (by holding lock for duration of a Move).

    Breaks CPs that were watching merge/split. These are run by Master now
    so you need to observe on Master, not on RegionServer.

    Details:

    M hbase-client/src/main/java/org/apache/hadoop/hbase/ClusterStatus.java
    Takes List of regionstates on construction rather than a Set.
    NOTE!!!!! This is a change in a public class.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
    Add utility getShortNameToLog

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ShortCircuitMasterConnection.java
    Add support for dispatching assign, split and merge processes.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java
    Purge old overlapping states: PENDING_OPEN, PENDING_CLOSE, etc.

    M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java
    Lots of doc on its inner workings. Bug fixes.

    M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
    Log and doc on workings. Bug fixes.

    A hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java
    Dispatch remote procedures every 150ms or 32 items -- which ever
    happens first (configurable). Runs a timeout thread. This facility is
    not on yet; will come in as part of a later fix. Currently works a
    region at a time. This class carries notion of a remote procedure and of a buffer full of these.
    "hbase.procedure.remote.dispatcher.threadpool.size" with default = 128
    "hbase.procedure.remote.dispatcher.delay.msec" with default = 150ms
    "hbase.procedure.remote.dispatcher.max.queue.size" with default = 32

    M hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java
    Add in support for merge. Remove no-longer used methods.

    M hbase-protocol-shaded/src/main/protobuf/Admin.proto b/hbase-protocol-shaded/src/main/protobuf/Admin.proto
    Add execute procedures call ExecuteProcedures.

    M hbase-protocol-shaded/src/main/protobuf/MasterProcedure.proto
    Add assign and unassign state support for procedures.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/client/VersionInfoUtil.java
    Adds getting RS version out of RPC
    Examples: (1.3.4 is 0x0103004, 2.1.0 is 0x0201000)

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
    Remove periodic metrics chore. This is done over in new AM now.
    Replace AM with the new. Host the procedures executor.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java
    Have AMv2 handle assigning meta.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
    Extract version number of the server making rpc.

    A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignProcedure.java
    Add new assign procedure. Runs assign via Procedure Dispatch.
    There can only be one RegionTransitionProcedure per region running at the time,
    since each procedure takes a lock on the region.

    D hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignCallable.java
    D hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
    D hbase-server/src/main/java/org/apache/hadoop/hbase/master/BulkAssigner.java
    D hbase-server/src/main/java/org/apache/hadoop/hbase/master/GeneralBulkAssigner.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/GeneralBulkAssigner.java
    Remove these hacky classes that were never supposed to live longer than
    a month or so to be replaced with real assigners.

    D hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStateStore.java
    D hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java
    D hbase-server/src/main/java/org/apache/hadoop/hbase/master/UnAssignCallable.java

    A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
    A procedure-based AM (AMv2).

    TODO
     - handle region migration
     - handle meta assignment first
     - handle sys table assignment first (e.g. acl, namespace)
     - handle table priorities
      "hbase.assignment.bootstrap.thread.pool.size"; default size is 16.
      "hbase.assignment.dispatch.wait.msec"; default wait is 150
      "hbase.assignment.dispatch.wait.queue.max.size"; wait max default is 100
      "hbase.assignment.rit.chore.interval.msec"; default is 5 * 1000;
      "hbase.assignment.maximum.attempts"; default is 10;

     A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MoveRegionProcedure.java
     Procedure that runs subprocedure to unassign and then assign to new location

     A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
     Manage store of region state (in hbase:meta by default).

     A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
     In-memory state of all regions. Used by AMv2.

     A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java
     Base RIT procedure for Assign and Unassign.

     A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/UnassignProcedure.java
     Unassign procedure.

     A hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
     Run region assignement in a manner that pays attention to target server version.
     Adds "hbase.regionserver.rpc.startup.waittime"; defaults 60 seconds.
2017-05-31 17:49:11 -07:00
Phil Yang 9cf1a08c53 HBASE-18122 Scanner id should include ServerName of region server 2017-05-31 13:57:05 +08:00
Andrew Purtell d547feac6b HBASE-18027 HBaseInterClusterReplicationEndpoint should respect RPC limits when batching edits 2017-05-30 14:24:51 -07:00
zhangduo 6846b03944 HBASE-18042 Client Compatibility breaks between versions 1.2 and 1.3 2017-05-27 17:55:49 +08:00
zhangduo efc7edc81a HBASE-18115 Move SaslServer creation to HBaseSaslRpcServer 2017-05-27 11:38:41 +08:00
Guanghao Zhang 97484f2aaf HBASE-18114 Update the config of TestAsync*AdminApi to make test stable 2017-05-27 11:11:40 +08:00
Ramkrishna 8b5c161cbf HBASE-17777 TestMemstoreLAB#testLABThreading runs too long for a small
test (Ram)
2017-05-26 17:15:04 +05:30
huzheng b076b8e794 HBASE-18120 (addendum) Fix TestAsyncRegionAdminApi
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2017-05-26 16:39:42 +08:00
huzheng 712beb305e HBASE-18120 Fix TestAsyncRegionAdminApi
Signed-off-by: Michael Stack <stack@apache.org>
2017-05-25 23:10:16 -07:00
huzheng f441ca0458 HBASE-16011 TableSnapshotScanner and TableSnapshotInputFormat can produce duplicate rows if split table.
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-05-25 12:47:43 -07:00
tedyu 3e426b2f85 HBASE-18099 FlushSnapshotSubprocedure should wait for concurrent Region#flush() to finish 2017-05-25 04:41:29 -07:00
Michael Stack a3c5a74487 Revert "HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi)"
Revert a mistaken commit!!!

This reverts commit dc1065a85d.
2017-05-24 23:31:36 -07:00
Michael Stack dc1065a85d HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi)
Move to a new AssignmentManager, one that describes Assignment using
a State Machine built on top of ProcedureV2 facility.

This doc. keeps state on where we are at w/ the new AM:
https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.vfdoxqut9lqn
Includes list of tests disabled by this patch with reasons why.

Based on patches from Matteos' repository and then fix up to get it all to pass cluster
tests, filling in some missing functionality, fix of findbugs, fixing bugs, etc..
including:

1. HBASE-14616 Procedure v2 - Replace the old AM with the new AM.
The basis comes from Matteo's repo here:
689227fcbf

Patch replaces old AM with the new under subpackage master.assignment.
Mostly just updating classes to use new AM -- import changes -- rather
than the old. It also removes old AM and supporting classes.
See below for more detail.

2. HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi)
3622cba4e3

Adds running of remote procedure. Adds batching of remote calls.
Adds support for assign/unassign in procedures. Adds version info
reporting in rpc. Adds start of an AMv2.

3. Reporting of remote RS version is from here:
ddb4df3964.patch

4. And remote dispatch of procedures is from:
186b9e7c4d

5. The split merge patches from here are also melded in:
9a3a95a2c2
and d6289307a0

We add testing util for new AM and new sets of tests.

Does a bunch of fixup on logging so its possible to follow a procedures' narrative by grepping
procedure id. We spewed loads of log too on big transitions such as master fail; fixed.

Fix CatalogTracker. Make it use Procedures doing clean up of Region data on split/merge.
Without these changes, ITBLL was failing at larger scale (3-4hours 5B rows) because we were
splitting split Regions among other things (CJ would run but wasn't
taking lock on Regions so havoc).

Added a bunch of doc. on Procedure primitives.

Added new region-based state machine base class. Moved region-based
state machines on to it.

Found bugs in the way procedure locking was doing in a few of the
region-based Procedures. Having them all have same subclass helps here.

Added isSplittable and isMergeable to the Region Interface.

Master would split/merge even though the Regions still had
references. Fixed it so Master asks RegionServer if Region
is splittable.

Messing more w/ logging. Made all procedures log the same and report
the state the same; helps when logging is regular.

Rewrote TestCatalogTracker. Enabled TestMergeTableRegionProcedure.

Added more functionality to MockMasterServices so can use it doing
standalone testing of Procedures (made TestCatalogTracker use it
instead of its own version).

Add to MasterServices ability to wait on Master being up -- makes
it so can Mock Master and start to implement standalone split testing.
Start in on a Split region standalone test in TestAM.

Fix bug where a Split can fail because it comes in in the middle of
a Move (by holding lock for duration of a Move).

Breaks CPs that were watching merge/split. These are run by Master now
so you need to observe on Master, not on RegionServer.

Details:

M hbase-client/src/main/java/org/apache/hadoop/hbase/ClusterStatus.java
Takes List of regionstates on construction rather than a Set.
NOTE!!!!! This is a change in a public class.

M hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
Add utility getShortNameToLog

M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ShortCircuitMasterConnection.java
Add support for dispatching assign, split and merge processes.

M hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java
Purge old overlapping states: PENDING_OPEN, PENDING_CLOSE, etc.

M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java
Lots of doc on its inner workings. Bug fixes.

M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
Log and doc on workings. Bug fixes.

A hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java
Dispatch remote procedures every 150ms or 32 items -- which ever
happens first (configurable). Runs a timeout thread. This facility is
not on yet; will come in as part of a later fix. Currently works a
region at a time. This class carries notion of a remote procedure and of a buffer full of these.
"hbase.procedure.remote.dispatcher.threadpool.size" with default = 128
"hbase.procedure.remote.dispatcher.delay.msec" with default = 150ms
"hbase.procedure.remote.dispatcher.max.queue.size" with default = 32

M hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java
Add in support for merge. Remove no-longer used methods.

M hbase-protocol-shaded/src/main/protobuf/Admin.proto b/hbase-protocol-shaded/src/main/protobuf/Admin.proto
Add execute procedures call ExecuteProcedures.

M hbase-protocol-shaded/src/main/protobuf/MasterProcedure.proto
Add assign and unassign state support for procedures.

M hbase-server/src/main/java/org/apache/hadoop/hbase/client/VersionInfoUtil.java
Adds getting RS version out of RPC
Examples: (1.3.4 is 0x0103004, 2.1.0 is 0x0201000)

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
Remove periodic metrics chore. This is done over in new AM now.
Replace AM with the new. Host the procedures executor.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java
Have AMv2 handle assigning meta.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
Extract version number of the server making rpc.

A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignProcedure.java
Add new assign procedure. Runs assign via Procedure Dispatch.
There can only be one RegionTransitionProcedure per region running at the time,
since each procedure takes a lock on the region.

D hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignCallable.java
D hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
D hbase-server/src/main/java/org/apache/hadoop/hbase/master/BulkAssigner.java
D hbase-server/src/main/java/org/apache/hadoop/hbase/master/GeneralBulkAssigner.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/GeneralBulkAssigner.java
Remove these hacky classes that were never supposed to live longer than
a month or so to be replaced with real assigners.

D hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStateStore.java
D hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java
D hbase-server/src/main/java/org/apache/hadoop/hbase/master/UnAssignCallable.java

A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
A procedure-based AM (AMv2).

TODO
 - handle region migration
 - handle meta assignment first
 - handle sys table assignment first (e.g. acl, namespace)
 - handle table priorities
  "hbase.assignment.bootstrap.thread.pool.size"; default size is 16.
  "hbase.assignment.dispatch.wait.msec"; default wait is 150
  "hbase.assignment.dispatch.wait.queue.max.size"; wait max default is 100
  "hbase.assignment.rit.chore.interval.msec"; default is 5 * 1000;
  "hbase.assignment.maximum.attempts"; default is 10;

 A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MoveRegionProcedure.java
 Procedure that runs subprocedure to unassign and then assign to new location

 A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
 Manage store of region state (in hbase:meta by default).

 A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
 In-memory state of all regions. Used by AMv2.

 A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java
 Base RIT procedure for Assign and Unassign.

 A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/UnassignProcedure.java
 Unassign procedure.

 A hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
 Run region assignement in a manner that pays attention to target server version.
 Adds "hbase.regionserver.rpc.startup.waittime"; defaults 60 seconds.
2017-05-24 20:47:25 -07:00
Amit Patel 8b75e9ed91 HBASE-18101 Fix type mismatch on container access
Signed-off-by: Michael Stack <stack@apache.org>
2017-05-24 19:59:16 -07:00
Umesh Agashe 837bb9ece7 HBASE-18091 Added API for getting who currently holds a lock on namespace/ table/ region/ server and log messages when procedure needs to wait to acquire lock
Signed-off-by: Michael Stack <stack@apache.org>
2017-05-24 14:56:22 -07:00
Yu Li 998bd5f90e HBASE-18084 Improve CleanerChore to clean from directory which consumes more disk space 2017-05-24 16:41:04 +08:00
Balazs Meszaros 80dd8bf51b HBASE-18096 Limit HFileUtil visibility and add missing annotations
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-05-24 16:34:59 +08:00
Stephen Yuan Jiang 1d0295f4e2 HBASE-18093 Overloading the meaning of 'enabled' in Quota Manager to indicate either quota disabled or quota manager not ready is not good (Stephen Yuan Jiang) 2017-05-23 06:40:33 -07:00
zhangduo 3f75ba195c HBASE-18013 Write response directly instead of creating a fake call when setup connection 2017-05-23 15:09:08 +08:00
tedyu 28d619b22b HBASE-17850 Backup system repair utility (Vladimir Rodionov) 2017-05-22 16:25:59 -07:00
Josh Elser f1a9990328 HBASE-17977 Enable the MasterSpaceQuotaObserver by default
It should be the normal case that HBase automatically deletes
quotas for deleted tables. Switch the Observer to be on by
default and add an option to instead prevent it from being added.
2017-05-22 13:41:36 -04:00
Josh Elser b971b449e8 HBASE-17978 Ensure superusers can circumvent actions restricted by space quota violations 2017-05-22 13:41:36 -04:00
Josh Elser ed618da906 HBASE-17981 Consolidate the space quota shell commands 2017-05-22 13:41:36 -04:00
Josh Elser d671a1dbc6 HBASE-17955 Various reviewboard improvements to space quota work
Most notable change is to cache SpaceViolationPolicyEnforcement objects
in the write path. When a table has no quota or there is not SpaceQuotaSnapshot
for that table (yet), we want to avoid creating lots of
SpaceViolationPolicyEnforcement instances, caching one instance
instead. This will help reduce GC pressure.
2017-05-22 13:41:36 -04:00
Josh Elser 98ace3d586 HBASE-17447 Implement a MasterObserver for automatically deleting space quotas
When a table or namespace is deleted, it would be nice to automatically
delete the quota on said table/NS. It's possible that not all people
would want this functionality so we can leave it up to the user to
configure this Observer.
2017-05-22 13:41:35 -04:00
Josh Elser a8460b8bad HBASE-17794 Swap "violation" for "snapshot" where appropriate
A couple of variables and comments in which violation is incorrectly
used to describe what the code is doing. This was a hold over from early
implementation -- need to scrub these out for clarity.
2017-05-22 13:41:35 -04:00
Josh Elser 13af7f8ac6 HBASE-17002 JMX metrics and some UI additions for space quotas 2017-05-22 13:41:35 -04:00
Josh Elser 91b4d2e827 HBASE-17568 Better handle stale/missing region size reports
* Expire region reports in the master after a timeout.
* Move regions in violation out of violation when insufficient
    region size reports are observed.
2017-05-22 13:41:35 -04:00
Josh Elser 8159eae781 HBASE-17602 Reduce some quota chore periods/delays 2017-05-22 13:41:35 -04:00
Josh Elser f031b69969 HBASE-17516 Correctly handle case where table and NS quotas both apply
The logic surrounding when a table and namespace quota both apply
to a table was incorrect, leading to a case where a table quota
violation which should have fired did not because of the less-strict
namespace quota.
2017-05-22 13:41:35 -04:00
Josh Elser 80a1f8fa2a HBASE-17428 Implement informational RPCs for space quotas
Create some RPCs that can expose the in-memory state that the
RegionServers and Master hold to drive the space quota "state machine".
Then, create some hbase shell commands to interact with those.
2017-05-22 13:41:35 -04:00
Josh Elser 4ad49bc3ac HBASE-17478 Avoid reporting FS use when quotas are disabled
Also, gracefully produce responses when quotas are disabled.
2017-05-22 13:41:35 -04:00
Josh Elser 6c9082fe16 HBASE-17259 API to remove space quotas on a table/namespace 2017-05-22 13:41:35 -04:00
Josh Elser 34ba143fc8 HBASE-17001 Enforce quota violation policies in the RegionServer
The nuts-and-bolts of filesystem quotas. The Master must inform
RegionServers of the violation of a quota by a table. The RegionServer
must apply the violation policy as configured. Need to ensure
that the proper interfaces exist to satisfy all necessary policies.

This required a massive rewrite of the internal tracking by
the general space quota feature. Instead of tracking "violations",
we need to start tracking "usage". This allows us to make the decision
at the RegionServer level as to when the files in a bulk load request
should be accept or rejected which ultimately lets us avoid bulk loads
dramatically exceeding a configured space quota.
2017-05-22 13:41:35 -04:00
Josh Elser 98b4181f43 HBASE-16999 Implement master and regionserver synchronization of quota state
* Implement the RegionServer reading violation from the quota table
* Implement the Master reporting violations to the quota table
* RegionServers need to track its enforced policies
2017-05-22 13:41:35 -04:00
Josh Elser 533470f8c8 HBASE-16998 Implement Master-side analysis of region space reports
Adds a new Chore to the Master that analyzes the reports that are
sent by RegionServers. The Master must then, for all tables with
quotas, determine the tables that are violating quotas and move
those tables into violation. Similarly, tables no longer violating
the quota can be moved out of violation.

The Chore is the "stateful" bit, managing which tables are and
are not in violation. Everything else is just performing
computation and informing the Chore on the updated state.

Added InterfaceAudience annotations and clean up the QuotaObserverChore
constructor. Cleaned up some javadoc and QuotaObserverChore. Reuse
the QuotaViolationStore impl objects.
2017-05-22 13:41:35 -04:00
tedyu 7fb0ac26e3 HBASE-17557 HRegionServer#reportRegionSizesForQuotas() should respond to UnsupportedOperationException 2017-05-22 13:41:35 -04:00
Josh Elser 6b334cd817 HBASE-17000 Implement computation of online region sizes and report to the Master
Includes a trivial implementation of the Master-side collection to
avoid. Only enough to write a test to verify RS collection.
2017-05-22 13:41:35 -04:00
tedyu f74e051bce HBASE-16996 Implement storage/retrieval of filesystem-use quotas into quota table (Josh Elser) 2017-05-22 13:41:35 -04:00
Apekshit Sharma 23ea2c36f5 HBASE-18068 Fix flaky test TestAsyncSnapshotAdminApi
- internalRestoreSnapshot() returns future which completes by just getting proc_id from master. Changed it to wait for the procedure to complete.
- Refactor TestAsyncSnapshotAdminApi: Add cleanup() which deletes all tables and snapshots after every test run. Simplifies individual tests.

Change-Id: Idc30fb699db32d58fd0f60da220953a430f1d3cc
2017-05-22 09:20:37 -07:00
Guanghao Zhang 3aac047a4f HBASE-18069 Fix flaky test TestReplicationAdminWithClusters#testDisableAndEnableReplication 2017-05-22 17:17:25 +08:00
Josh Elser 709f5a1980 HBASE-18075 Support non-latin table names and namespaces 2017-05-21 22:24:12 -04:00
zhangduo 1ceb25cf09 HBASE-18081 The way we process connection preamble in SimpleRpcServer is broken 2017-05-21 20:36:33 +08:00
anastas 1520c8fd4d HBASE-18056 Make the default behavior of CompactionPipeline to merge it segments into one, due to better read performance in this case 2017-05-21 12:27:57 +03:00
Umesh Agashe 8b70d043e4 HBASE-18071 Fix flaky test TestStochasticLoadBalancer#testBalanceCluster
Test was failing on clusters with large number of servers or regions. Using commonly using config settings like some other tests seems to work.

Signed-off-by: Michael Stack <stack@apache.org>
2017-05-19 11:09:28 -07:00
Guanghao Zhang 3fe4b28bb0 HBASE-15616 Allow null qualifier for all table operations 2017-05-19 17:47:08 +08:00
tedyu 958cd2d1b7 HBASE-18035 Meta replica does not give any primaryOperationTimeout to primary meta region (huaxiang sun) 2017-05-18 15:56:41 -07:00
Jingcheng Du 6dc4190c07 HBASE-18049 It is not necessary to re-open the region when MOB files cannot be found 2017-05-18 18:54:58 +08:00
huzheng 37dd8ff722 HBASE-11013: Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2017-05-18 17:39:50 +08:00
Chia-Ping Tsai 32d2062b5c HBASE-18019 Close redundant memstore scanners 2017-05-18 16:07:21 +08:00
Guanghao Zhang 62d7323023 HBASE-18053 AsyncTableResultScanner will hang when scan wrong column family 2017-05-17 12:16:51 +08:00
Umesh Agashe c1b45a2c45 HBASE-18016 Changes to inherit default behavior of abort from StateMachineProcedure making TruncateTableProcedure abortable
This will allow abort and manual cleanup of stuck instances of TruncateTableProcedure.

Signed-off-by: Michael Stack <stack@apache.org>
2017-05-16 18:57:54 -07:00
Umesh Agashe 5eb1b7b96c HBASE-18018 Changes to support abort for all procedures by default
The default behavior for abort() method of StateMachineProcedure is changed to support aborting all procedures irrespective of if rollback is supported or not. Currently its observed that sometimes procedures may fail on a step which will be considered as retryable error as abort is not supported. As a result procedure may stuck in a endless loop repeating same step again.User should have an option to abort any stuck procedure and do clean up manually. Please refer to HBASE-18016 and discussion there.

Signed-off-by: Michael Stack <stack@apache.org>
2017-05-16 18:56:32 -07:00
anoopsamjohn 67d1358311 HBASE-18043 Institute a hard limit for individual cell size that cannot be overridden by clients. - Addendum to fix test issue in TestMobStoreScanner. 2017-05-16 18:53:27 +05:30
zhangduo ad9ffaaafd HBASE-18055 Releasing L2 cache HFileBlocks before shipped() when switching from pread to stream causes result corruption 2017-05-16 21:16:36 +08:00
Andrew Purtell 6b60ba8ade HBASE-18043 Institute a hard limit for individual cell size that cannot be overridden by clients 2017-05-15 18:03:33 -07:00
zhangduo 341223d86c HBASE-18012 Move RpcServer.Connection to a separated file 2017-05-15 18:07:38 +08:00
anastas 5cdaca5c00 HBASE-16436 Adding CellChunkMap code, its tests and fixes to all code review comments 2017-05-14 16:04:36 +03:00
tedyu 305ffcb040 HBASE-17938 General fault - tolerance framework for backup/restore operations (Vladimir Rodionov) 2017-05-12 09:27:58 -07:00
Umesh Agashe da68537ae6 HBASE-17786 Create LoadBalancer perf-test tool to benchmark Load Balancer algorithm performance
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
2017-05-12 11:22:15 -05:00
Chia-Ping Tsai b34ab5980e HBASE-17887 Row-level consistency is broken for read 2017-05-12 19:45:07 +08:00
tedyu 5e046151d6 HBASE-11013: Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions - revert, pending work in snapshot descriptor 2017-05-11 18:53:14 -07:00
tedyu b3dcfb659e HBASE-17928 Shell tool to clear compaction queues (Guangxu Cheng) 2017-05-11 18:47:12 -07:00
tedyu d8d4ba7c59 HBASE-17928 Shell tool to clear compaction queues - revert pending work in snapshot descriptor 2017-05-11 18:43:59 -07:00
Andrew Purtell c833473619 Amend HBASE-18000 Make sure we always return the scanner id with ScanResponse (Karan Mehta and Duo Zhang)
Move getRegionScanner() call below builder.setScannerId() to handle a
corner case.
2017-05-11 14:35:23 -07:00
zhangduo 0ae0edcd63 HBASE-17917 Use pread by default for all user scan 2017-05-11 11:28:22 +08:00
anastas c5cc81d8e3 HBASE-17343 Changing the default type of CompactingMemStore to be BASIC 2017-05-10 09:02:26 +03:00
Roman Nikitchenko 3bcc475ee2 HBASE-8758 Error in RegionCoprocessorHost class preScanner method documentation
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-05-10 11:35:45 +08:00
tedyu 815b0f853b HBASE-17928 Shell tool to clear compaction queues (Guangxu Cheng) 2017-05-09 18:32:38 -07:00
Andrew Purtell c38bf12444 HBASE-18017 Reduce frequency of setStoragePolicy failure warnings 2017-05-09 12:08:05 -07:00
huzheng 951b23a44c HBASE-11013: Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-05-09 09:32:48 -07:00
zhangduo 51d4c68b7c HBASE-18009 Move RpcServer.Call to a separated file 2017-05-09 14:56:03 +08:00
Andrew Purtell 959deb0e5c HBASE-17924 Consider sorting the row order when processing multi() ops before taking rowlocks
(Allan Yang)
2017-05-08 16:23:13 -07:00
Andrew Purtell 3dc38a4ff1 HBASE-18015 Storage class aware block placement for procedure v2 WALs 2017-05-08 14:11:27 -07:00
zhangduo e99ed99bee HBASE-18000 Make sure we always return the scanner id with ScanResponse
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-05-05 11:38:34 -07:00
huzheng 2026540ea3 HBASE-17667: Implement async flush/compact region methods
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2017-05-05 21:12:49 +08:00
Zach York 01af27061e HBASE-17990 Refactor TestFSUtils to use Before
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-05-04 14:40:40 -07:00
Sean Busbey 0fb1fd1200 HBASE-17964 ensure hbase-metrics-api is included in mapreduce job classpaths
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2017-05-04 10:39:14 -05:00
tedyu 52119200bc HBASE-17991 Add more details about compaction queue on /dump (Guangxu Cheng) 2017-05-04 08:24:58 -07:00
Sukumar Maddineni 2de6b051f6 HBASE-16466 Snapshots support in VerifyReplication tool
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-05-03 14:22:18 -07:00
Chinmay Kulkarni 78f6799f4d Addendum patch for HBASE-12870. Added check for null pointer.
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-05-03 14:03:16 -07:00
huzheng ff998ef74f HBASE-17867: Implement async procedure RPC API(list/exec/abort/isFinished)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2017-05-03 21:46:37 +08:00
Guanghao Zhang 91995749c2 HBASE-17958 Avoid passing unexpected cell to ScanQueryMatcher when optimize SEEK to SKIP 2017-05-03 21:28:43 +08:00
binlijin 4dbd025ccf HBASE-17263 Netty based rpc server impl
Signed-off-by: zhangduo <zhangduo@apache.org>
2017-05-03 17:23:26 +08:00
ckulkarni 30d0c3fdde HBASE-12870 "Major compaction triggered" and "Skipping major compaction" messages lack the region information
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-05-02 10:54:15 -07:00
Jan Hentschel 17007685c1 HBASE-13074 Cleaned up usage of hbase.master.lease.thread.wakefrequency
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-05-02 23:12:02 +08:00
Gábor Lipták c8a7e80e0a HBASE-14286 Correct typo in argument name for WALSplitter.writeRegionSequenceIdFile
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-05-02 10:01:55 +08:00
yoshida.shinya ef94de3eb7 HBASE-17957 Custom metrics of replicate endpoints don't prepend "source." to global metrics
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-05-01 08:11:59 -07:00
Abhishek Kumar 1848353fd6 HBASE-17879: Avoid NPE in snapshot.jsp when accessing without any request parameter
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-04-29 10:51:46 +08:00
Sean Busbey ba12cdf138 HBASE-17975 TokenUtil should throw remote exception rather than squash it.
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Ted Yu <tedyu@apache.org>
Signed-off-by: Umesh Agashe <uagashe@cloudera.com>
2017-04-28 13:19:33 -05:00
zhangduo 73d80bb416 HBASE-17970 Set yarn.app.mapreduce.am.staging-dir when starting MiniMRCluster
Signed-off-by: Sean Busbey <busbey@apache.org>
2017-04-28 12:37:00 -05:00
QilinCao 03e8f6b56e HBASE-17835 Spelling mistakes in the Java source
Signed-off-by: Sean Busbey <busbey@apache.org>
2017-04-28 11:25:57 -05:00
Jonathan M Hsieh 43f3fccb7b HBASE-17920 TestFSHDFSUtils always fails against hadoop 3.0.0-alpha2
Signed-off-by: Sean Busbey <busbey@apache.org>
2017-04-28 11:25:14 -05:00
Steen Manniche 635c9db815 HBASE-17817 add table name to output (if available) when removing coprocessors
Amending-Author: Sean Busbey <busbey@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
2017-04-28 11:24:00 -05:00
tedyu 5411d3ecb1 HBASE-17972 Remove mergePool from CompactSplitThread (Guangxu Cheng) 2017-04-28 06:52:10 -07:00
Ramkrishna b401a35fdc HBASE-17950 Write the chunkId also as Int instead of long into the first
byte of the chunk (Ram)
2017-04-28 14:44:46 +05:30
Allan Yang c4cbb419a3 HBASE-17757 Unify blocksize after encoding to decrease memory fragment
Signed-off-by: anoopsamjohn <anoopsamjohn@gmail.com>
2017-04-28 07:25:36 +05:30
huzheng 4bc0eb31c3 HBASE-17865: Implement async listSnapshot/deleteSnapshot methods.
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2017-04-28 09:09:13 +08:00
Karan Mehta 880db3eee4 HBASE-17965 Canary tool should print the regionserver name on failure
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-04-26 18:30:13 -07:00
Thiruvel Thirumoolan 6bad35e728 HBASE-16942 Add FavoredStochasticLoadBalancer and FN Candidate generators
Signed-off-by: Francis Liu <toffer@apache.org>
2017-04-26 18:11:45 -07:00
Chia-Ping Tsai 053e61541e HBASE-15583 Any HTableDescriptor we give out should be immutable 2017-04-27 03:22:29 +08:00
zhangduo 8973582bc6 HBASE-17956 Raw scan should ignore TTL 2017-04-26 15:03:53 +08:00
Balazs Meszaros 2557506415 HBASE-15143 Procedure v2 - Web UI displaying queues
Signed-off-by: Michael Stack <stack@apache.org>
2017-04-25 09:39:28 -07:00
zhangduo 72fac37981 HBASE-17952 The new options for PE tool do not work 2017-04-25 09:34:49 +08:00
tedyu 435104af70 HBASE-17302 The region flush request disappeared from flushQueue - addendum (Guangxu Cheng) 2017-04-23 07:37:34 -07:00
Chia-Ping Tsai 9053ec6fe6 HBASE-17943 Addendum increases the threshold value of in-memory compaction for TestWalAndCompactingMemStoreFlush 2017-04-22 20:47:55 +08:00
tedyu e95cf479c7 HBASE-16314 Retry on table snapshot failure during full backup (Vladimir Rodionov) 2017-04-21 16:15:07 -07:00
huzheng d39f40e787 HBASE-17864: Implement async snapshot/cloneSnapshot/restoreSnapshot methods
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2017-04-21 18:57:43 +08:00
s9514171 33dadc1a94 HBASE-17941 CellArrayMap#getCell may throw IndexOutOfBoundsException
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-04-21 11:35:39 +08:00
Chia-Ping Tsai ea3a27b18d HBASE-17943 The in-memory flush size is different for each CompactingMemStore located in the same region 2017-04-21 11:20:20 +08:00
Abhishek Singh Chouhan 49cba2c237 HBASE-17937 Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
Signed-off-by: zhangduo <zhangduo@apache.org>
2017-04-21 08:50:09 +08:00
Guanghao Zhang 40cc666ac9 HBASE-17915 Implement async replication admin methods 2017-04-20 18:13:03 +08:00
tedyu b1ef8dd43a HBASE-17542 Move backup system table into separate namespace 2017-04-20 02:57:24 -07:00
zhangduo 0953c14470 HBASE-17940 HMaster can not start due to Jasper related classes conflict 2017-04-20 16:06:50 +08:00
zhangduo 3acd8e4644 HBASE-17929 addendum fix typo and modify printUsage 2017-04-19 21:03:09 +08:00
Ramkrishna 972e8c8c29 HBASE-16438 Create a cell type so that chunk id is embedded in it (Ram) 2017-04-19 15:28:03 +05:30
zhangduo 6e962d6f65 HBASE-17929 addendum add scan read type option 2017-04-19 11:58:44 +08:00
zhangduo 66b616d7a3 HBASE-17914 Create a new reader instead of cloning a new StoreFile when compaction 2017-04-19 09:26:33 +08:00
zhangduo 75d1e0361a HBASE-17930 Avoid using Canary.sniff in HBaseTestingUtility 2017-04-18 18:07:25 +08:00
gjacoby 5eda5fb9d7 HBASE-17912 - Avoid major compactions on region server startup
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-04-17 19:41:19 -07:00
zhangduo 3c32032f5c HBASE-17929 Add more options for PE tool 2017-04-18 09:52:34 +08:00
Ramkrishna ecdfb82326 Revert "HBASE-16438 Create a cell type so that chunk id is embedded in it (Ram)"
This reverts commit c2c2178b2e.
2017-04-18 00:00:12 +05:30
Ramkrishna c2c2178b2e HBASE-16438 Create a cell type so that chunk id is embedded in it (Ram) 2017-04-17 09:28:24 +05:30
huaxiang sun c1ac3f7739 HBASE-17366 Run TestHFile#testReaderWithoutBlockCache failes
Signed-off-by: CHIA-PING TSAI <chia7712@gmail.com>
2017-04-17 10:34:17 +08:00
huzheng 8db9760363 HBASE-17866: Implement async setQuota/getQuota methods
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2017-04-17 09:49:30 +08:00
Michael Stack 7678855fac HBASE-17904 Get runs into NoSuchElementException when using Read Replica, with hbase. ipc.client.specificThreadForWriting
to be true and hbase.rpc.client.impl to be org.apache.hadoop.hbase.ipc.RpcClientImpl (Huaxiang Sun)
2017-04-16 11:01:06 -07:00
Umesh Agashe c8461456d0 HBASE-17888: Added generic methods for updating metrics on submit and finish of a procedure execution
Signed-off-by: Michael Stack <stack@apache.org>
2017-04-14 11:51:08 -07:00
Apekshit Sharma da5fb27eab HBASE-16775 Fix flaky TestExportSnapshot#testExportRetry.
Reason for flakyness: Current test is probability based fault injection and triggers failure 3% of the time. Earlier when test used LocalJobRunner which didn't honor "mapreduce.map.maxattempts", it'd pass 97% time (when no fault is injected) and fail 3% time (when fault was injected). Point being, even when the test was complete wrong, we couldn't catch it because it was probability based.

This change will inject fault in a deterministic manner.
On design side, it encapsulates all testing hooks in ExportSnapshot.java into single inner class.

Change-Id: Icba866e1d56a5281748df89f4dd374bc45bad249
2017-04-12 11:11:31 -07:00
CHIA-PING TSAI cf3215d343 HBASE-17897 StripeStoreFileManager#nonOpenRowCompare use the wrong comparison function 2017-04-12 11:54:42 +08:00
CHIA-PING TSAI 3aadc675b0 HBASE-17896 The FIXED_OVERHEAD of Segment is incorrect 2017-04-12 11:26:43 +08:00
Nemo Chen 0b5bd78d6e HBASE-16469 Several log refactoring/improvement suggestions
Signed-off-by: Sean Busbey <busbey@apache.org>
2017-04-11 14:16:12 -05:00
Enis Soztutar 82d554e378 HBASE-16477 Remove Writable interface and related code from WALEdit/WALKey 2017-04-10 02:31:42 -07:00
CHIA-PING TSAI df96d328fb HBASE-17872 The MSLABImpl generates the invaild cells when unsafe is not availble 2017-04-09 23:28:34 +08:00
Chia-Ping Tsai 18c5ecf6ed HBASE-17881 Remove the ByteBufferCellImpl 2017-04-07 21:14:19 +08:00
Chia-Ping Tsai 1a701ce444 HBASE-17836 CellUtil#estimatedSerializedSizeOf is slow when input is ByteBufferCell 2017-04-07 09:30:15 +08:00
Jerry He 48b2502a5f HBASE-17816 HRegion#mutateRowWithLocks should update writeRequestCount metric (Weizhan Zeng) 2017-04-06 16:45:45 -07:00
Umesh Agashe 9109803891 HBASE-17863: Procedure V2: Some cleanup around Procedure.isFinished() and procedure executor
Signed-off-by: Michael Stack <stack@apache.org>
2017-04-06 12:05:23 -07:00
Tomu Tsuruhara ec5188df30 HBASE-17871 scan#setBatch(int) call leads wrong result of VerifyReplication
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-04-06 07:00:13 -07:00
Yu Li cbcbcf4dcd HBASE-17854 Use StealJobQueue in HFileCleaner after HBASE-17215 2017-04-05 17:53:21 +08:00
zhangduo a66d491892 HBASE-17857 Remove IS annotations from IA.Public classes 2017-04-05 15:34:06 +08:00
tedyu 910b68082c HBASE-14141 HBase Backup/Restore Phase 3: Filter WALs on backup to include only edits from backed up tables (Vladimir Rodionov) 2017-04-04 18:20:11 -07:00
Yu Li 9facfa550f HBASE-17215 Separate small/large file delete threads in HFileCleaner to accelerate archived hfile cleanup speed 2017-04-01 10:59:11 +08:00
huzheng 5f98ad2053 HBASE-17668: Implement async assgin/offline/move/unassign methods
Signed-off-by: zhangduo <zhangduo@apache.org>
2017-04-01 08:19:14 +08:00
Karan Mehta 80381f3944 HBASE-17698 ReplicationEndpoint choosing sinks
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-03-31 16:12:10 -07:00
Michael Stack 7700a7fac1 HBASE-16780 Since move to protobuf3.1, Cells are limited to 64MB where previous they had no limit Update internal pb to 3.2 from 3.1. 2017-03-31 12:44:59 -07:00
zhangduo 0ec1459467 HBASE-17855 Fix typo in async client implementation 2017-03-31 13:00:27 +08:00
Michael Stack d033cbb715 HBASE-17844 Subset of HBASE-14614, Procedure v2: Core Assignment Manager (non-critical changes)
Minor changes related to HBASE-14614. Added comments. Changed logging.
Added toString formatting. Removed imports. Removed unused code.
2017-03-30 10:31:04 -07:00
Guanghao Zhang 752b258b7c HBASE-17520 Implement isTableEnabled/Disabled/Available methods 2017-03-30 13:37:17 +08:00
tedyu f159557ede HBASE-17287 Master becomes a zombie if filesystem object closes 2017-03-28 20:41:46 -07:00
tedyu 0345fc8775 HBASE-14417 Incremental backup and bulk loading 2017-03-28 16:23:36 -07:00
anastas cb4fac1d18 HBASE-17765 Reviving the merge of the compacting pipeline
making the limit on the number of the segments in the pipeline configurable, adding merge test, fixing bug in sizes counting
2017-03-28 10:54:57 -05:00
Sean Busbey 046d4e183b Revert "Reviving the merge of the compacting pipeline: making the limit on the number of the segments in the pipeline configurable, adding merge test, fixing bug in sizes counting"
This reverts commit c77e2135db.

Bad commit message
2017-03-28 10:53:41 -05:00
Ashu Pachauri 4b62a52ebc HBASE-16755 Honor flush policy under global memstore pressure
Signed-off-by: Gary Helmling <garyh@apache.org>
2017-03-27 11:47:20 -07:00
anastas c77e2135db Reviving the merge of the compacting pipeline: making the limit on the number of the segments in the pipeline configurable, adding merge test, fixing bug in sizes counting 2017-03-27 15:41:32 +03:00
Chia-Ping Tsai 6bd3109062 HBASE-17623 Reuse the bytes array when building the hfile block 2017-03-25 23:49:32 +08:00
huzheng faf81d5133 HBASE-17669: Implement async mergeRegion/splitRegion methods
Signed-off-by: zhangduo <zhangduo@apache.org>
2017-03-24 21:25:24 +08:00
zhangduo f1c1f258e5 HBASE-17595 addendum fix the problem for mayHaveMoreCellsInRow 2017-03-23 20:34:10 +08:00
CHIA-PING TSAI fe3c32ebd5 HBASE-17809 cleanup unused class 2017-03-23 14:15:28 +08:00
tedyu 1cfd22bf43 HBASE-17798 RpcServer.Listener.Reader can abort due to CancelledKeyException (Guangxu Cheng) 2017-03-21 06:59:29 -07:00
eshcar 8f4ae0a0dc HBASE-17655 Removing MemStoreScanner and SnapshotScanner 2017-03-21 12:35:47 +02:00
Apekshit Sharma e39e0e634a HBASE-17582 Fix broken drop page cache hint (broken by HBASE-15236).
Change-Id: I2947ab979979f977db7b0c282c4aaf4eb1f26482
2017-03-20 11:19:51 -07:00
Jan Hentschel 55d6dcaf87 HBASE-16084 Cleaned up the stale references in Javadoc
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-03-20 10:55:36 -07:00
zhangduo 5b4bb8217d HBASE-17691 Add ScanMetrics support for async scan 2017-03-20 20:54:04 +08:00
Chia-Ping Tsai 261aa9445c HBASE-17803 Addendum fix NPE 2017-03-19 19:05:25 +08:00
CHIA-PING TSAI 23abc90068 HBASE-17803 PE always re-creates table when we specify the split policy 2017-03-19 18:27:54 +08:00
tedyu 75d0f49dcd HBASE-14123 HBase Backup/Restore Phase 2 (Vladimir Rodionov) 2017-03-18 03:04:19 -07:00
tedyu 0ed50f42e7 HBASE-17707 New More Accurate Table Skew cost function/generator - revert due to test failure 2017-03-16 19:08:45 -07:00
tedyu a69c23abfe HBASE-17706 TableSkewCostFunction improperly computes max skew - revert due to test failure 2017-03-16 19:07:59 -07:00
Andrew Purtell 7f0e6f1c9e HBASE-17758 [RSGROUP] Add shell command to move servers and tables at the same time (Guangxu Cheng) 2017-03-16 18:37:40 -07:00
Michael Stack 7c19490bac HBASE-17792 Use a shared thread pool for AtomicityWriter, AtomicGetReader, AtomicScanReader's connections in TestAcidGuarantees (Huaxiang Sun) 2017-03-16 15:15:28 -07:00
Michael Stack 6fb44f7eb8 HBASE-17501 Revert "Revert "guard against NPE while reading FileTrailer and HFileBlock""
This reverts commit 9a4068dcf8.

This is a revert of a revert -- i.e. a restore -- just so I can add the
JIRA issue to the commit message.
2017-03-16 14:54:17 -07:00
Michael Stack 9a4068dcf8 Revert "guard against NPE while reading FileTrailer and HFileBlock"
This reverts commit 201c838250.

Reverted because missing JIRA number. Fixing...
2017-03-16 14:53:25 -07:00
Kahlil Oppenheimer edbd0e494d HBASE-17706 TableSkewCostFunction improperly computes max skew
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-03-16 11:57:25 -07:00
Ramkrishna e67eb6c424 HBSE-15314 Allow more than one backing file in bucketcache (Chunhui Shen) 2017-03-16 16:28:58 +05:30
Guanghao Zhang 6a6fff103e HBASE-17790 Mark ReplicationAdmin's peerAdded and listReplicationPeers as Deprecated 2017-03-16 16:55:18 +08:00
Kahlil Oppenheimer 53e9a1c43a HBASE-17707 New More Accurate Table Skew cost function/generator - re-enable with test fix
This reverts commit 9214ad69af.

Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-03-15 20:42:40 -07:00
zhangduo 1849e8a5a7 HBASE-17740 Correct the semantic of batch and partial for async client 2017-03-16 09:44:23 +08:00
Chia-Ping Tsai 0ecb678259 HBASE-17723 ClientAsyncPrefetchScanner may end prematurely when the size of the cache is one 2017-03-16 03:07:20 +08:00
zhangduo a49bc58a54 HBASE-17584 Expose ScanMetrics with ResultScanner rather than Scan 2017-03-15 17:48:58 +08:00
Yu Li aace02a230 HBASE-17782 Extend IdReadWriteLock to support using both weak and soft reference 2017-03-15 11:07:42 +08:00
tedyu 777fea552e HBASE-17779 disable_table_replication returns misleading message and does not turn off replication (Janos Gub) 2017-03-14 12:13:34 -07:00
Yu Li 44b255889c HBASE-17747 Support both weak and soft object pool 2017-03-14 11:07:52 +08:00
James Moore 201c838250 guard against NPE while reading FileTrailer and HFileBlock
guard against NPE from FSInputStream#seek

Signed-off-by: Michael Stack <stack@apache.org>
2017-03-13 14:53:35 -07:00
Tom Tsuruhara fee67bcf14 HBASE-17773 VerifyReplication tool wrongly emits Invalid arguments error
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2017-03-13 17:46:58 +08:00
Chia-Ping Tsai 31bc94ae60 HBASE-17746 TestSimpleRpcScheduler.testCoDelScheduling is broken
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2017-03-13 10:13:42 +08:00
zhangduo e0365df9d6 HBASE-17712 Addendum modify log message 2017-03-10 16:01:35 +08:00
zhangduo ed6e5d6999 HBASE-17763 IPCUtil.wrapException will wrap DoNotRetryIOException with IOException 2017-03-10 15:37:59 +08:00
anoopsamjohn ab5970773a HBASE-17338 Treat Cell data size under global memstore heap size only when that Cell can not be copied to MSLAB. 2017-03-10 11:12:59 +05:30
Andrew Purtell 7da0feea8d HBASE-17731 Fractional latency reporting in MultiThreadedAction 2017-03-09 16:54:23 -08:00
Ashu Pachauri d4cbff58cc HBASE-16977 VerifyReplication should log a printable representation of the row keys
Signed-off-by: Gary Helmling <garyh@apache.org>
2017-03-09 16:31:02 -08:00
Esteban Gutierrez 1160315e2f HBASE-15941 HBCK repair should not unsplit healthy splitted region
Signed-off-by: Michael Stack <stack@apache.org>
2017-03-08 22:02:14 -08:00
Umesh Agashe 2b89748432 HBASE-17761: Test TestRemoveRegionMetrics.testMoveRegion fails intermittently because of race condition
Signed-off-by: Esteban Gutierrez <esteban@apache.org>
2017-03-08 21:18:05 -08:00
Guanghao Zhang 4ff838df13 HBASE-17600 Implement get/create/modify/delete/list namespace admin operations 2017-03-09 10:32:46 +08:00
tedyu 9214ad69af HBASE-17707 New More Accurate Table Skew cost function/generator - revert due to test failure 2017-03-08 16:09:34 -08:00
Michael Stack 6a57050c24 HBASE-17718 Difference between RS's servername and its ephemeral node cause SSH stop working; AMENDMENT.
Make test tighter by extending ServerListener so can find when Master
is in the waiting-on-regionservers state and making more assertions
about state.

Fix error where I would move on from waiting-on-regionservers if
we had waited max time.
2017-03-08 08:14:24 -08:00
zhangduo 58c76192bd HBASE-17712 Remove/Simplify the logic of RegionScannerImpl.handleFileNotFound 2017-03-08 14:39:29 +08:00
Kahlil Oppenheimer ee1549cc97 HBASE-17707 New More Accurate Table Skew cost function/generator - adendum ensure testTableSkewConvergesToZero finishes before timeout
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-03-07 12:37:16 -08:00
Jan Hentschel b53f354763 HBASE-17532 Replaced explicit type with diamond operator
Signed-off-by: Michael Stack <stack@apache.org>
2017-03-07 11:22:51 -08:00
Karan Mehta fee86ebd7b HBASE-17716 Formalize Scan Metric names
Signed-off-by: Michael Stack <stack@apache.org>
2017-03-07 07:41:54 -08:00
Phil Yang b635414e83 HBASE-15484 Correct the semantic of batch and partial - amend to fix bug and revise the JavaDoc for related APIs. 2017-03-07 22:23:56 +08:00
Kahlil Oppenheimer 93b0cdeaaf HBASE-17707 New More Accurate Table Skew cost function/generator
This reverts commit 3b914df949.

Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-03-06 20:27:48 -08:00
tedyu dfc6cf3076 HBASE-17707 New More Accurate Table Skew cost function/generator - revert due to missing JIRA number 2017-03-06 20:26:59 -08:00
Phil Yang 0d3e986f7e HBASE-15484 Correct the semantic of batch and partial 2017-03-07 11:52:04 +08:00
ChiaPing Tsai 81cb298014 HBASE-17734 Guard against possibly copying the qualifier in the ScanDeleteTracker
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-03-06 18:01:06 -08:00
Michael Stack 7fa7156f2c HBASE-17718 Difference between RS's servername and its ephemeral node cause SSH stop working
This patch reverts HBASE-9593 -- i.e. registering in zk before we
register with master putting it back to how it was where we register
in zk AFTER we report for duty with the master (because then we'll
register in zk with the name the master gave us). It then fixes the
problem reported in HBASE-9593 in an alternate fashion by checking
for a RS znode if we failed a connect on assign; if none found, we
remove a server from online servers list.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
 Make move method available to tests.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java
 Correct method name changing moveFromOnelineToDeadServers to
 moveFromOnlineToDeadServers

 Add actual fix which is call to checkForRSznode if exception trying to
 open a region; if none found, call expire on the server so it gets
 removed from the list of online servers.

 This patch exposes sloppyness in the waitForRegionServers around our
 current case where Master is hosting regions but ONLY hbase:meta;
 in this case we need to wait on at least another server to report
 in beyond Master (we weren't but stuff was 'working' because of the
 early registration of RS nodes in zk).

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
 Make 'killed' available to tests.

 Put registry of ephemeral node back to where it was originally,
 so it is AFTER we get response from Master on registering for duty
 so we can put our znode up in zk with the name the Master gave us
 rather than local name (which could be unknown to the Master).
   private boolean stopping = false;

M hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRSKilledWhenInitializing.java
 Cleanup and test of new cleanup.
2017-03-06 13:14:23 -08:00
Kahlil Oppenheimer de25fcbd93 Fixed flaky test failure for TestStochasticLoadBalancer
This reverts commit 3b914df949.

Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-03-06 12:15:33 -08:00
tedyu 3b914df949 HBASE-17707 New More Accurate Table Skew cost function/generator - revert due to TestDefaultLoadBalancer failure 2017-03-05 15:03:29 -08:00
Andrew Purtell 404a2883f2 HBASE-17722 Metrics subsystem stop/start messages add a lot of useless bulk to operational logging 2017-03-03 12:40:06 -08:00
tedyu 06e984b086 HBASE-17707 New More Accurate Table Skew cost function/generator (Kahlil Oppenheimer) 2017-03-02 09:46:38 -08:00
tedyu 0b3ecc5ee7 HBASE-17710 HBase in standalone mode creates directories with 777 permission 2017-03-02 08:11:53 -08:00
huzheng 697a55a878 HBASE-17646: Implement Async getRegion method
Signed-off-by: zhangduo <zhangduo@apache.org>
2017-03-02 11:52:05 +08:00
anastas 613bcb3622 HBASE-17662 Disable in-memory flush when replaying from WAL
Signed-off-by: anoopsamjohn <anoopsamjohn@gmail.com>
2017-03-01 20:37:54 +05:30
tedyu 4a5eba5e59 HBASE-17673 Monitored RPC Handler not shown in the WebUI (Allan Yang) 2017-02-28 07:43:32 -08:00
Jingcheng Du f4e0ea24e6 HBASE-17688 MultiRowRangeFilter not working correctly if given same start and stop RowKey 2017-02-28 11:38:35 +08:00
Jingcheng Du 5a8f1e8aaa HBASE-17690 Clean up MOB code 2017-02-28 09:57:34 +08:00
Jan Hentschel 84a9eb3f5c HBASE-17634 Cleaned up the usage of Result.isPartial
Signed-off-by: zhangduo <zhangduo@apache.org>
2017-02-27 20:44:35 +08:00
Apekshit Sharma 4d90425031 HBASE-17699 Fix TestLockProcedure.
Earlier when queues had locks, clearQueue() also cleaned up old locks when AbstractProcedureScheduler.clear() was called to reset scheduler for testing failure and recovery.
Now with locks decoupled from queues, they need to be separately cleaned up.
We can't have clearLocks() as abstract method in AbstractProcedureScheduler because at that level, a procedure scheduler is just a queue. It's only in MasterProcedureScheduler that locks come into picture. So directly overriding clear() method in MPS.

Earlier when queues had locks, clearQueue() also cleaned up old locks when AbstractProcedureScheduler.clear() was called.
Now with locks decoupled from queues, they need to be separately cleaned up.
We can't have clearLocks() as abstract method in AbstractProcedureScheduler because at that level, a procedure scheduler is just a queue. It's only in MasterProcedureScheduler that locks come into picture. So directly overriding clear() method in MPS.

Change-Id: If1a0acb418a79f98ce6155541edb0c1e621638e3
2017-02-26 15:32:31 -08:00
tedyu f4bf375ea1 HBASE-17674 Major compaction may be cancelled in CompactionChecker (Guangxu Cheng) 2017-02-25 09:45:11 -08:00
Apekshit Sharma f444b3b542 HBASE-17312 Use 'default' keyword in coprocessor Observer interfaces to get rid of 'Base...Observer' implementations. Some javadoc improvements too.
Reason for refactor:
In cases where one might need to use multiple observers, say region, master and regionserver; and the fact that only one class can be extended, it gives rise to following pattern:

public class BaseMasterAndRegionObserver
  extends BaseRegionObserver
  implements MasterObserver

class AccessController
  extends BaseMasterAndRegionObserver
  implements RegionServerObserver

were BaseMasterAndRegionObserver is full copy of BaseMasterObserver.

There is an example of simple case too where the current design fails.
Say only one observer is needed by the coprocessor, but the design doesn't permit extending even that single observer (see RSGroupAdminEndpoint), that leads to copy of full Bas
e...Observer class into coprocessor class leading to 1000s of lines of code and this ugly mix of 5 main functions with 100 useless functions.

Javadocs changes:
- Adds class comments on 'default' methods and expectations.
- Adds explanaiton of Exception handling in Observers' class comment. Removes redundant @throws before each function.
- Improves javadocs for a bunch of functions
- deletes empty @params in a bunch of places

Change-Id: I265738d47e8554e7b4678e88bb916a0cc7d00ab3
2017-02-25 02:49:01 -08:00
Ashu Pachauri e7d16db2ac HBASE-17057 Minor compactions should also drop page cache (Ashu Pachauri)
Signed-off-by: Gary Helmling <garyh@apache.org>
2017-02-24 14:42:10 -08:00
dvdreddy a8fd1119ce HBASE-16630 Handle Fragmentation in bucket cache
Currently whenever a compaction/bulkload happen and the
blocks are evicted from theirs buckets the buckets become
fragmented and are not available to be used by other
BucketSizes

Bug Fix : Added Memory block type also to the list of
evictions that need to happen when there is a needForExtra

Improvement : Inorder to fix the non availabilty of Buckets and force
the movement of buckets to transformed sizes, whenever we encounter a
situation where an allocation cant be made for a BucketSize, we will
forcefully free the entire buckets that have least occupancy ratio. This
is the same strategy used by MemCached when they encounter a similar
issue going by the name 'Slab Calcification'. Only improvement is that
we use a heuristic to evict from the buckets that are least occupied
and also avoid the BucketSizes where there is a single Bucket

Change-Id: I9e3b4deb8d893953003ddf5f1e66312ed97ea9cb

Signed-off-by: Ramkrishna <ramkrishna.s.vasudevan@intel.com>
2017-02-24 15:02:07 +05:30
zhangduo 62de29e6f2 HBASE-16991 Make the initialization of AsyncConnection asynchronous 2017-02-24 15:36:54 +08:00
Abhishek Singh Chouhan c90d484f61 HBASE-17682 Region stuck in merging_new state indefinitely
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-02-23 15:30:17 -08:00
zhangduo 8fb44fae35 HBASE-17595 Add partial result support for small/limited scan 2017-02-23 16:03:42 +08:00
anoopsamjohn ff045cab84 HBASE-17647 OffheapKeyValue#heapSize() implementation is wrong. 2017-02-23 11:29:10 +05:30
Sean Busbey 040b2f186a HBASE-17677 ServerName parsing from directory name should be more robust to errors from guava's HostAndPort
Signed-off-by: Michael Stack <stack@apache.org>
2017-02-22 20:53:57 -06:00
Ashu Pachauri e4c06c120a HBASE-17590 Drop cache hint should work on store file write path (Ashu Pachauri)
Signed-off-by: Gary Helmling <garyh@apache.org>
2017-02-22 12:39:10 -08:00
zhangduo f037f230fd HBASE-17608 Add suspend support for RawScanResultConsumer 2017-02-22 20:32:03 +08:00
Yu Li 93e60153b0 HBASE-17676 Get class name once for all in AbstractFSWAL 2017-02-22 15:50:09 +08:00
tedyu c579cf67b0 HBASE-17657 TestZKAsyncRegistry is flaky 2017-02-21 21:17:15 -08:00
Phil Yang a0da66dc36 HBASE-17210 Set timeout on trying rowlock according to client's RPC timeout 2017-02-22 11:09:55 +08:00
rgidwani 3c0750de54 HBASE-17675 ReplicationEndpoint should choose new sinks if a SaslException occurs
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-02-21 18:44:54 -08:00
Sean Busbey d7ffa0013b HBASE-15328 sanity check the redirect used to send master info requests to the embedded regionserver.
Signed-off-by: Esteban Gutierrez <esteban@apache.org>
2017-02-20 22:03:17 -06:00
huzheng 22fa1cd3df HBASE-17472: Correct the semantic of permission grant
Signed-off-by: zhangduo <zhangduo@apache.org>
2017-02-20 20:13:43 +08:00
ChiaPing Tsai d08bafad1a HBASE-17661 fix the queue length passed to FastPathBalancedQueueRpcExecutor
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-02-19 13:25:32 -08:00
Sean Busbey 350904e90f HBASE-17561 table status page should escape values that may contain arbitrary characters.
Signed-off-by: Esteban Gutierrez <esteban@apache.org>
2017-02-18 15:46:18 -06:00
tedyu 0e05537dec HBASE-17460 enable_table_replication can not perform cyclic replication of a table (NITIN VERMA) 2017-02-17 17:56:53 -08:00
Huaxiang Sun d7325185ad HBASE-17172 Optimize major mob compaction with _del files
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-02-17 14:22:31 -08:00
Tim Brown 938aef772d HBASE-17658 Fix bookkeeping error with max regions for a table
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-02-17 07:40:16 -08:00
Yi Liang a2d2196e89 HBASE-17648: HBase Table-level synchronization fails between two secured(kerberized) cluster
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-02-17 07:03:53 -08:00
anoopsamjohn 7763dd6688 HBASE-17644 Always create ByteBufferCells after copying to MSLAB. 2017-02-17 17:50:41 +05:30
huzheng e129e6b65d HBASE-17619: Add async admin Impl which connect to RegionServer and implement close region methods.
Signed-off-by: zhangduo <zhangduo@apache.org>
2017-02-17 09:32:45 +08:00
Michael Stack ae840c0ccd HBASE-17656 Move new Address class from util to net package 2017-02-16 12:17:50 -08:00
Abhishek Singh Chouhan 0446ed0f0c HBASE-17069 RegionServer writes invalid META entries for split daughters in some circumstances
Signed-off-by: zhangduo <zhangduo@apache.org>
2017-02-16 15:27:44 +08:00
Huaxiang Sun 33e9a8c775 HBASE-17640 Unittest error in TestMobCompactor with different timezone (Huaxiang Sun)
Signed-off-by: Jingcheng Du <jingchengdu@apache.org>
2017-02-15 11:23:04 +08:00
zhangduo c8f3c8630b HBASE-17583 Add inclusive/exclusive support for startRow and endRow of scan for sync client 2017-02-15 10:18:41 +08:00
Michael Stack e019961150 HBASE-17624 Address late review of HBASE-6721, rsgroups feature
Addresses review comments by Sean Busbey and Appy that happened
to come in long after the commit of HBASE-6721, the original
rsgroup issue.

Also includes subsequent accommodation of Duo Zhang review.

Adds a new type to hold hostname and port. It is called
Address. It is a facade over Guava's HostAndPort. Replace
all instances of HostAndPort with Address. In particular,
those places where HostAndPort was part of the rsgroup
public API.

Fix licenses. Add audience annotations.

Cleanup and note concurrency expectation on a few core classes.
In particular, all access on RSGroupInfoManager is made
synchronized.

M hbase-client/src/main/java/org/apache/hadoop/hbase/ServerName.java
 Host the hostname and port in an instance of the new type Address.
 Add a bunch of deprecation of exotic string parses that should never
 have been public.

M hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupAdmin.java
 Make this an Interface rather than abstract class. Creation was a
 static internal method that only chose one type.... Let it be free
 as a true Interface instead.
2017-02-14 09:29:17 -08:00
Phil Yang b2217d1d8f HBASE-17639 Do not stop server if ReplicationSourceManager's waitUntilCanBePushed throws InterruptedException 2017-02-14 17:46:13 +08:00
Sean Busbey 024ae356ac HBASE-17558 ZK dumping jsp should escape HTML.
Signed-off-by: Enis Soztutar <enis@apache.org>
2017-02-13 23:37:59 -05:00
Jan Hentschel 55c2e2d484 HBASE-9702 Changed unit tests to use method names for tables 2017-02-13 13:27:55 -08:00
Yu Li d40bca4365 HBASE-17638 Remove duplicated initialization of CacheConfig in HRegionServer 2017-02-13 23:23:10 +08:00
Apekshit Sharma 826b9436fb HBASE-17605 Changes
- Moved locks out of MasterProcedureScheduler#Queue. One Queue object is used for each namespace/table, which aren't more than 100. So we don't need complexity arising from all functionalities being in one place. SchemaLocking now owns locks and locking implementaion has been moved to procedure2 package.
- Removed NamespaceQueue because it wasn't being used as Queue (add,peek,poll,etc functions threw UnsupportedOperationException). It's was only used for locks on namespaces. Now that locks have been moved out of Queue class, it's not needed anymore.
- Remoed RegionEvent which was there only for locking on regions. Tables/namespaces used locking from Queue class and regions couldn't (there are no separate proc queue at region level), hence the redundance. Now that locking is separate, we can use the same for regions too.
- Removed QueueInterface class. No declarations, except one implementaion, which makes the point of having an interface moot.
- Removed QueueImpl, which was the only concrete implementation of abstract Queue class. Moved functions to Queue class itself to avoid unnecessary level in inheritance hierarchy.
- Removed ProcedureEventQueue class which was just a wrapper around ArrayDeque class. But we now have ProcedureWaitQueue as 'Type class'.
- Encapsulated table priority related stuff in a single class.
- Removed some unused functions.
Change-Id: I6a60424cb41e280bc111703053aa179d9071ba17
2017-02-11 14:31:43 -08:00
tedyu a05abd83ef HBASE-17628 Local mode of mini cluster shouldn't use hdfs (Devaraj Das) 2017-02-10 17:52:33 -08:00
Esteban Gutierrez 054acec840 HBASE-17622 Add hbase-metrics package to TableMapReduceUtil 2017-02-10 09:32:50 -08:00
Jingcheng Du 9d5d25c88d HBASE-17618 Refactor the implementation of modify table and delete column in MOB 2017-02-10 21:26:53 +08:00
rgidwani 1b041a4fc7 HBASE-17609 Allow for region merging in the UI
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-02-09 14:06:56 -08:00
ChiaPing Tsai 489c8872c1 HBASE-17613 avoid copy of family when initializing the FSWALEntry
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-02-09 08:18:47 -08:00
zhangduo 712fe69e4d HBASE-17599 Use mayHaveMoreCellsInRow instead of isPartial 2017-02-09 15:38:02 +08:00
Enis Soztutar b23890157c HBASE-17105 Annotate RegionServerObserver 2017-02-08 13:35:22 -08:00
eshcar edbf2bb8de HBASE-17575: Run critical tests with each of the Inmemory Compaction Policies enabled
Signed-off-by: Michael Stack <stack@apache.org>
2017-02-08 08:39:37 -08:00
huzheng d8f3c6cff9 HBASE-17381 ReplicationSourceWorkerThread can die due to unhandled exceptions (huzheng)
Signed-off-by: Gary Helmling <garyh@apache.org>
2017-02-07 16:39:14 -08:00
Vincent c55fce00f3 HBASE-15995 Separate replication WAL reading from shipping
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-02-07 12:51:15 -08:00
tedyu d0498d979c HBASE-17565 StochasticLoadBalancer may incorrectly skip balancing due to skewed multiplier sum - addendum resets min cost 2017-02-07 12:24:20 -08:00
Ramkrishna 6c5eec249c HBASE-17484 Add non cached version of OffheapKV for write path (Ram) 2017-02-07 21:27:33 +05:30
tedyu 9d8de85fa5 HBASE-17565 StochasticLoadBalancer may incorrectly skip balancing due to skewed multiplier sum 2017-02-07 06:22:47 -08:00
Guanghao Zhang 9a78d00884 HBASE-17389 Convert all internal usages from ReplicationAdmin to Admin 2017-02-07 10:18:59 +08:00
zhangduo af9d359b8e HBASE-17402 TestAsyncTableScan sometimes hangs 2017-02-07 09:42:42 +08:00
Michael Stack 9ec0ec4922 HBASE-17350 Fixup of regionserver group-based assignment
Renamed move_rsgroup_servers as move_servers_rsgroup
Renamed move_rsgroup_tables as move_tables_rsgroup

Minor changes to help text in rsgroup commands making them all same.
Made LOG from RSGroupAdminServer all talk of 'rsgroup' rather than
'group' to be consistent.

Fix for table.jsp where it would fail to display regions because no
type for the protobuf record specified.

Fix it so that move of an offline server to 'default' rsgroup is like
moving the reference to the server to trash (keeps the 'default' group
consistently 'dynamic' regards its server-list).

Fixed another issue where we were stuck in a loop because regions
were in FAILED_OPEN state because no server to assign too so we'd
never recover (a vagary of the current state of Master assignement
but no less a possibility in real world deploys).

Make it so servers are sorted when we list them; its what operator
would expect.
2017-02-06 13:09:57 -08:00
Enis Soztutar 4d730244a9 HBASE-17187 DoNotRetryExceptions from coprocessors should bubble up to the application 2017-02-06 12:00:42 -08:00
tedyu 19e1450b0f HBASE-17601 close() in TableRecordReaderImpl assumes the split has started (Michael Axiak) 2017-02-06 10:32:20 -08:00
Guanghao Zhang 5093a49e0b HBASE-17596 Implement add/delete/modify column family methods 2017-02-06 13:45:01 +08:00
Guanghao Zhang 26a94844f5 HBASE-17511 Implement enable/disable table methods 2017-02-05 14:18:40 +08:00
zhangduo ffa0cea2a3 HBASE-17508 Unify the implementation of small scan and regular scan for sync client 2017-02-05 08:49:39 +08:00
Huaxiang Sun 1159296541 HBASE-16981 Expand Mob Compaction Partition policy from daily to weekly, monthly
Support weekly and monthly mob compact partition policies in addition to the existing
daily partition policy.

Signed-off-by: Jingcheng Du <jingchengdu@apache.org>
2017-02-03 15:52:43 +08:00
Zach York 63c819efbe HBASE-17588 Remove unused imports brought in by HBASE-17437
Signed-off-by: Michael Stack <stack@apache.org>
2017-02-02 22:05:50 -08:00
Zach York 537580bc51 HBASE-17587 Do not Rethrow DoNotRetryIOException as UnknownScannerException
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-02-02 17:12:18 -08:00
Ajay Jadhav f8b1f57b05 HBASE-17280 Add mechanism to control hbase cleaner behavior
Signed-off-by: anoopsamjohn <anoopsamjohn@gmail.com>
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-02-01 22:16:01 -06:00
Sean Busbey 2c799fb70a Revert "Add mechanism to control hbase cleaner behavior"
This reverts commit ef052521cd.

Bad commit message.
2017-02-01 22:11:48 -06:00
Ajay Jadhav ef052521cd Add mechanism to control hbase cleaner behavior
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-02-01 19:33:06 -08:00
Thiruvel Thirumoolan c6250ecc83 HBASE-17281 FN should use datanode port from hdfs configuration
Signed-off-by: Francis Liu <toffer@apache.org>
2017-02-01 10:43:58 -08:00
Balazs Meszaros 81956aa446 HBASE-17197 hfile does not work in 2.0
Files can be specified with -f option or without it.

Signed-off-by: Michael Stack <stack@apache.org>
2017-01-31 22:37:31 -08:00
Balazs Meszaros ccd5b9f873 HBASE-3462 Fix table.jsp in regards to splitting a region/table with an optional splitkey
Now compact/split requires row keys instead of region names (as UI suggests).

Signed-off-by: Michael Stack <stack@apache.org>
2017-01-31 16:34:41 -08:00
tedyu 34ffca1357 HBASE-16621 HBCK should have -fixHFileLinks (Janos Gub) 2017-01-31 14:32:45 -08:00
Geoffrey 5ebaadf1a6 HBASE-17543 - Create additional ReplicationEndpoint WALEntryFilters by configuration
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-01-31 12:58:37 -08:00
Zach York ae21797305 HBASE-17437 Support specifying a WAL directory outside of the root directory (Yishan Yang and Zach York)
Signed-off-by: Enis Soztutar <enis@apache.org>
2017-01-31 11:43:33 -08:00
Thiruvel Thirumoolan bd7c9581f2 HBASE-17198 Remove redundant FN updates to merged region
Signed-off-by: Francis Liu <toffer@apache.org>
2017-01-31 11:17:58 -08:00
Thiruvel Thirumoolan 680289d67d HBASE-17101: FavoredNodes should not apply to system tables
Signed-off-by: Francis Liu <toffer@apache.org>
2017-01-31 10:58:50 -08:00
Sean Busbey 6791828698 HBASE-17522 Handle JVM throwing runtime exceptions when we ask for details on heap usage the same as a correctly returned 'undefined'.
Signed-off-by: Michael Stack <stack@apache.org>
2017-01-31 09:56:45 -06:00
Jingcheng Du 47ce72d78c HBASE-16812 Clean up the locks in MOB 2017-01-31 12:15:09 +08:00
Balazs Meszaros e68ab09d5e HBASE-17566 Jetty upgrade fixes
Fix issues caused by HBASE-12894 Upgrade Jetty to 9.2.6 commit.

- removed Jetty 6.x dependencies (org.mortbay.jetty.*)
- corrected @Ignore-d unit tests

Signed-off-by: Michael Stack <stack@apache.org>
2017-01-30 11:56:21 -08:00
Michael Stack ffe7dac53d Revert "Revert "HBASE-12894 Upgrade to Jetty 9 for REST / Info Server / Thrift Http Server""
This reverts commit 0ac5d4a717.
This is a revert of a revert; i.e. a reapplication!
Fixes for breakage that comes in with this patch is in a follow-on.
2017-01-30 11:54:54 -08:00
Michael Stack 52ad310c80 HBASE-17471 Region Seqid will be out of order in WAL if using mvccPreAssign (Allan Yang) 2017-01-30 09:19:38 -08:00
Jan Hentschel aff8de8397 HBASE-17555 Changed calls to deprecated getHBaseAdmin to getAdmin
Signed-off-by: Michael Stack <stack@apache.org>
2017-01-28 21:41:25 -08:00
Yu Li 3693d1306d HBASE-17526 Procedure v2 - cleanup isSuspended from MasterProcedureScheduler#Queue (addendum) 2017-01-29 00:14:29 +08:00
Michael Stack c725d4d334 HBASE-17538 HDFS.setStoragePolicy() logs errors on local fs Policy is set in a number of places each with its own 'implementation'.
A setStoragePolicy was added by:

    commit 629b04f44f
    Author: Yu Li <liyu@apache.org>
    Date:   Fri Jan 6 18:35:38 2017 +0800

      HBASE-15172 Support setting storage policy in bulkload

..for hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java

And in *FileSystem files and StoreFileWriter by

    commit f92a14ade6
    Author: Yu Li <liyu@apache.org>
    Date:   Mon Jan 9 09:52:58 2017 +0800

      HBASE-14061 Support CF-level Storage Policy

This patch has all instances call the FSUtils#setStoragePolicy added here:

    commit eafc07a06d
    Author: tedyu <yuzhihong@gmail.com>
    Date:   Thu Jan 15 08:52:30 2015 -0800

      HBASE-12848 Utilize Flash storage for WAL

It does right thing when local vs distributed calling setStoragePolicy.

Fixed bug in the above FSUtils#setStoragePolicy where the check for a
distributed filesystem was failing when passed an HFileSystem -- though
it was backed by a DistributedFileSystem.

Cleanups.
2017-01-27 23:01:42 -08:00
Sukumar Maddineni bd8a4d5fad HBASE-17540 Change SASL server GSSAPI callback log line from DEBUG to TRACE in RegionServer
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-01-27 15:14:00 -08:00
tedyu 07c175b10a HBASE-17515 Reduce memory footprint of RegionLoads kept by StochasticLoadBalancer (Tim Brown) 2017-01-26 11:04:49 -08:00
tedyu e8979c67aa HBASE-17515 Reduce memory footprint of RegionLoads kept by StochasticLoadBalancer - revert due to pending discussion 2017-01-26 10:44:54 -08:00
Michael Stack 0ac5d4a717 Revert "HBASE-12894 Upgrade to Jetty 9 for REST / Info Server / Thrift Http Server"
This reverts commit 59fd6eb7f6.
2017-01-25 22:55:18 -08:00
tedyu a8bb27b2e0 HBASE-17515 Reduce memory footprint of RegionLoads kept by StochasticLoadBalancer (Tim Brown) 2017-01-25 17:59:55 -08:00
Apekshit Sharma 4254d7edf3 HBASE-17526 Queue#setSuspended() is not used anywhere, probably because when queue wait/wakes on an event, it gets removed or added back to the fairq.
Removing state, functions, and uses of isSuspended().

Change-Id: Id56eba81eaa66518bb0cba98070f0b78f8941c4f
2017-01-25 12:00:54 -08:00
Enis Soztutar c64a1d1994 HBASE-9774 HBase native metrics and metric collection for coprocessors 2017-01-25 11:47:35 -08:00
Yang Guang 59fd6eb7f6 HBASE-12894 Upgrade to Jetty 9 for REST / Info Server / Thrift Http Server 2017-01-24 23:37:12 -08:00
Guanghao Zhang 8aea84d723 HBASE-17500 Implement getTable/creatTable/deleteTable/truncateTable methods 2017-01-25 11:30:11 +08:00
zhangduo 85d701892e HBASE-17045 Unify the implementation of small scan and regular scan 2017-01-25 09:53:06 +08:00
Guanghao Zhang 843ba9e545 HBASE-17443 Move listReplicated/enableTableRep/disableTableRep methods from ReplicationAdmin to Admin 2017-01-24 13:40:29 +08:00
anastas aa5d9a9ad3 HBASE-17492 Fix the compacting memstore part in hbase shell ruby script
Signed-off-by: Michael Stack <stack@apache.org>
2017-01-23 21:27:44 -08:00
Michael Stack 980c8c2047 HBASE-17067 Procedure v2 - remove zklock/tryLock and use wait/wake (Matteo Bertozzi)
This is an amalgam of https://reviews.apache.org/r/54435/ and
9c14863594

Removes notion of suspend/resume from procedure. Instead have the below lock states
and just unschedule if lock is not yet available

 LOCK_ACQUIRED should be returned when the proc has the lock and the proc is ready to execute.
 LOCK_YIELD_WAIT should be returned when the proc has not the lock and the framework
   should take care of readding the procedure back to the runnable set for retry
 LOCK_EVENT_WAIT should be returned when the proc has not the lock and someone will take care of
  readding the procedure back to the runnable set when the lock is available.

Side benefit is being able to undo a bunch of synchronization around
procedure management.

Signed-off-by: Michael Stack <stack@apache.org>
2017-01-23 09:29:16 -08:00
Yu Li ba4a926b62 HBASE-17367 Make HTable#getBufferedMutator thread safe 2017-01-23 18:23:24 +08:00
tedyu 7754a9620e HBASE-17462 Use sliding window for read/write request costs in StochasticLoadBalancer (Tim Brown) 2017-01-22 18:35:38 -08:00
eshcar f254e278ec HBASE-17407: Correct update of maxFlushedSeqId in HRegion
Signed-off-by: zhangduo <zhangduo@apache.org>
2017-01-23 09:22:51 +08:00
zhangduo 3abd13dacb HBASE-17489 ClientScanner may send a next request to a RegionScanner which has been exhausted 2017-01-22 16:39:00 +08:00
Jan Hentschel 55a1aa1e73 HBASE-10699 Set capacity on ArrayList where possible and use isEmpty instead of size() == 0
Signed-off-by: Michael Stack <stack@apache.org>
2017-01-20 22:58:20 -08:00
Dave Navarro c74cf12925 HBASE-17357 FIX PerformanceEvaluation parameters parsing triggers NPE.
check command name is not null, if null print usage and exit

Signed-off-by: Michael Stack <stack@apache.org>
2017-01-20 22:51:57 -08:00
Guanghao Zhang 79018056f5 HBASE-17498 Implement listTables and listTableNames methods
Signed-off-by: Michael Stack <stack@apache.org>
2017-01-20 14:12:13 -08:00
Michael Stack 2285c57a35 HBASE-17488 WALEdit should be lazily instantiated (ChiaPing Tsai) 2017-01-20 09:37:48 -08:00
Guanghao Zhang 2ee3c73f76 HBASE-17497 Add first async MetaTableAccessor impl and Implement tableExists method
Signed-off-by: Michael Stack <stack@apache.org>
2017-01-20 09:24:06 -08:00
Michael Stack 4fdd6ff9ae HBASE-16831 Procedure V2 - Remove org.apache.hadoop.hbase.zookeeper.lock
(Appy)
2017-01-19 16:51:44 -08:00
Apekshit Sharma 558a6bb9d7 HBASE-16867 Procedure V2 - Check ACLs for MasterRpcServices' queueLock() and lockHeartbeat().
Change-Id: I03773059c169022318cf7953110bb022b6ad216d

Signed-off-by: Michael Stack <stack@apache.org>
2017-01-19 16:42:57 -08:00
Michael Stack 76dc957f64 HBASE-16786 Procedure V2 - Move ZK-lock's uses to Procedure framework locks (LockProcedure) - Matteo Bertozzi
Locks are no longer hosted up in zookeeper but instead by the Master.
2017-01-19 09:34:17 -08:00
Stephen Yuan Jiang bff7c4f1fd HBASE-17480 Remove split region code from Region Server (Stephen Yuan Jiang) 2017-01-19 09:05:19 -08:00
Guanghao Zhang cb9ce2ceaf HBASE-17396 Add first async admin impl and implement balance methods 2017-01-19 10:15:12 +08:00
tedyu 6cbc375aa4 HBASE-17482 mvcc mechanism fails when using mvccPreAssign (Allan Yang) 2017-01-18 07:50:41 -08:00
Ramkrishna b779143fdc HBASE-17081 Flush the entire CompactingMemStore content to disk - recommit
(Anastasia)
2017-01-18 14:40:47 +05:30
Stephen Yuan Jiang 805d39fca6 HBASE-17470 Remove merge region code from region server (Stephen Yuan Jiang) 2017-01-17 15:39:51 -08:00
Yu Li 287f95a579 HBASE-17474 Reduce frequency of NoSuchMethodException when calling setStoragePolicy() 2017-01-18 07:30:21 +08:00
tedyu faa9f735ca HBASE-17469 Properly handle empty TableName in TablePermission#readFields and #write (Manjunath Anand) 2017-01-17 09:36:59 -08:00
zhangduo 4ab95ebbce HBASE-17372 Make AsyncTable thread safe 2017-01-17 14:33:28 +08:00
Michael Stack 4cb09a494c HBASE-16744 Procedure V2 - Lock procedures to allow clients to acquire
locks on tables/namespaces/regions (Matteo Bertozzi)

Incorporates review comments from
    https://reviews.apache.org/r/52589/
    https://reviews.apache.org/r/54388/

M hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncTableBase.java
 Fix for eclipse complaint (from Duo Zhang)

M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java
M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java
 Log formatting

M hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/ProcedureTestingUtility.java
 Added wait procedures utility.

A hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/LockServiceProtos.java
A hbase-protocol-shaded/src/main/protobuf/LockService.proto b/hbase-protocol-shaded/src/main/protobuf/LockService.proto
 Implement new locking CP overrides.

A hbase-server/src/main/java/org/apache/hadoop/hbase/client/locking/EntityLock.java
 New hbase entity lock (ns, table, or regions)

A hbase-server/src/main/java/org/apache/hadoop/hbase/client/locking/LockServiceClient.java
 Client that can use the new internal locking service.
2017-01-13 21:07:03 -08:00
Zach York 9fd5dab1a8 HBASE-17464 Fix getNewDataTestDirOnTestFS to always return a unique path
Signed-off-by: Michael Stack <stack@apache.org>
2017-01-13 17:48:22 -08:00
tedyu cdee1a7034 HBASE-17450 TablePermission#equals throws NPE after namespace support was added (huzheng) 2017-01-13 09:37:08 -08:00
tedyu 6d98c48699 HBASE-17452 Failed taking snapshot - region Manifest proto-message too large (huaxiang sun) 2017-01-13 07:59:31 -08:00
eshcar 2f8ddf6fc5 HBASE-17434: New synchronization scheme for compaction pipeline
Signed-off-by: Michael Stack <stack@apache.org>
2017-01-12 06:35:58 -08:00
Mike Grimes f7d0f15c99 HBASE-17165 Add retry to LoadIncrementalHFiles tool
Signed-off-by: Michael Stack <stack@apache.org>
2017-01-11 15:20:48 -08:00
tedyu 471cf13b1a HBASE-17445 Count size of serialized exceptions in checking max result size quota 2017-01-11 07:00:27 -08:00
Yu Li 953416eb34 HBASE-14061 Support CF-level Storage Policy (addendum2)
Resolve TestPartitionedMobCompactor UT failure
2017-01-11 18:23:37 +08:00
Yu Li 36eeb2c569 HBASE-14061 Support CF-level Storage Policy (addendum)
Addendum to resolve compatible issue with Hadoop 2.8.0+ / 3.0.0-alpha1+, meantime added
a util method in ReflectionUtils for invoking method with reflection
2017-01-11 10:14:55 +08:00
Guanghao Zhang ac3b1c9aa9 HBASE-17337 list replication peers request should be routed through master 2017-01-10 08:57:26 +08:00
Michael Stack bd157ffe9a Revert "HBASE-17434 New Synchronization Scheme for Compaction Pipeline (Eshcar Hillel)"
Undo premature commit

This reverts commit 1576269123.
2017-01-09 15:40:01 -08:00
tedyu 8dd35631ca HBASE-17429 HBase bulkload cannot support HDFS viewFs (shenxianqiang) 2017-01-09 14:48:51 -08:00
Michael Stack dd1ae37148 HBASE-12148 Remove TimeRangeTracker as point of contention when many threads writing a Store (Huaxiang Sun) 2017-01-09 13:24:53 -08:00
Michael Stack 1576269123 HBASE-17434 New Synchronization Scheme for Compaction Pipeline (Eshcar Hillel) 2017-01-09 10:46:34 -08:00
tedyu 9cbeba6c3d HBASE-17435 Call to preCommitStoreFile() hook encounters SaslException in secure deployment 2017-01-09 08:51:00 -08:00
Ramkrishna f65a439f01 HBASE-17291 Remove ImmutableSegment#getKeyValueScanner (Ram) 2017-01-09 12:06:25 +05:30
Yu Li f92a14ade6 HBASE-14061 Support CF-level Storage Policy 2017-01-09 09:52:58 +08:00
tedyu b2a9be02ac HBASE-17408 Introduce per request limit by number of mutations (ChiaPing Tsai) 2017-01-06 13:07:15 -08:00
Ashish Singhi 5f631b9653 HBASE-17290 Potential loss of data for replication of bulk loaded hfiles 2017-01-06 16:18:20 +05:30
Yu Li 629b04f44f HBASE-15172 Support setting storage policy in bulkload 2017-01-06 18:35:38 +08:00
Guanghao Zhang e02ae7724d HBASE-17388 Move ReplicationPeer and other replication related PB messages to the replication.proto 2017-01-06 10:01:22 +08:00
rahul gidwani 0f6c79eb12 HBASE-16710 Add ZStandard Codec to Compression.java
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-01-05 10:46:55 -08:00
Jan Hentschel dba103e1b6 HBASE-17390 Added master and backup masters to online update of configuration
Signed-off-by: Jerry He <jerryjch@apache.org>
2017-01-04 22:08:05 -08:00
Josh Elser 20a7ae2865 HBASE-17409 Limit jsonp callback name to prevent xss 2017-01-04 17:05:07 -05:00
tedyu 63bd8be576 HBASE-17403 ClientAsyncPrefetchScanner doesn’t load any data if the MaxResultSize is too small (ChiaPing Tsai) 2017-01-04 08:21:25 -08:00
eshcar 69ce5967fd HBASE-17373: Fixing bug in moving segments from compaction pipeline to snapshot
Signed-off-by: Michael Stack <stack@apache.org>
2017-01-03 19:13:52 -08:00
Guanghao Zhang 0e48665641 HBASE-17336 get/update replication peer config requests should be routed through master 2016-12-30 10:12:47 +08:00
tedyu 1c477b2df9 HBASE-17387 Reduce the overhead of exception report in RegionActionResult for multi() 2016-12-29 17:54:02 -08:00
Jan Hentschel 7572e96e3a HBASE-17385 Change usage documentation from bin/hbase to hbase in various tools
Signed-off-by: Enis Soztutar <enis@apache.org>
2016-12-29 15:20:43 -08:00
tedyu 001a26d404 HBASE-17374 ZKPermissionWatcher crashed when grant after region close (Liu Junhong) 2016-12-28 19:54:01 -08:00
zhangduo 05b1d918b0 HBASE-17320 Add inclusive/exclusive support for startRow and endRow of scan 2016-12-29 09:43:31 +08:00
Michael Stack a3e0e0df0d HBASE-17149 Procedure V2 - Fix nonce submission to avoid unnecessary calling coprocessor multiple times; ADDENDUM by Stephen Yuan Jiang 2016-12-28 11:10:19 -08:00
tedyu 79e5efd35c HBASE-17081 Flush the entire CompactingMemStore content to disk - revert due to failure in TestHRegionWithInMemoryFlush 2016-12-28 10:53:07 -08:00
Michael Stack 306ef83c9c HBASE-17068 Procedure v2 - inherit region locks (Matteo Bertozzi) 2016-12-27 16:17:45 -08:00
tedyu e18e9a22da HBASE-17376 ClientAsyncPrefetchScanner may fail due to too many rows (ChiaPing Tsai) 2016-12-26 15:55:22 -08:00
Ramkrishna 463ffa792a HBASE-17081 [Recommit]Flush the entire CompactingMemStore content to disk
(Anastasia)
2016-12-26 22:05:13 +05:30
zhangduo 8fa5b0b946 HBASE-17345 Implement batch 2016-12-25 20:36:52 +08:00
ChiaPing Tsai 8cb55c4080 HBASE-17174 Refactor the AsyncProcess, BufferedMutatorImpl, and HTable
Signed-off-by: zhangduo <zhangduo@apache.org>
2016-12-24 12:02:05 +08:00
Jerry He 992e5717d4 HBASE-16010 Put draining function through Admin API (Matt Warhaftig) 2016-12-23 13:41:36 -08:00
Phil Yang 8fb9a91d44 HBASE-17314 Limit total buffered size for all replication sources 2016-12-23 11:48:06 +08:00
Guanghao Zhang b3f2bec099 HBASE-17335 enable/disable replication peer requests should be routed through master 2016-12-23 09:27:12 +08:00
zhangduo 09bb428763 HBASE-17334 Add locate row before/after support for AsyncRegionLocator 2016-12-22 20:39:01 +08:00
tedyu 66781864aa HBASE-17330 SnapshotFileCache will always refresh the file cache (Jianwei Cui) 2016-12-22 02:29:27 -08:00
binlijin fc93de51af HBASE-17262 Refactor RpcServer so as to make it extendable and/or pluggable 2016-12-22 14:49:56 +08:00
Michael Stack d787155fd2 HBASE-5401 PerformanceEvaluation generates 10x the number of expected mappers (Yi Liang) 2016-12-21 11:55:48 -08:00
Michael Stack a1d2ff4646 Revert "HBASE-17314 Limit total buffered size for all replication sources"
This reverts commit 3826e63967.
2016-12-21 11:17:28 -08:00
tedyu cac0904c16 HBASE-17341 Add a timeout during replication endpoint termination (Vincent Poon) 2016-12-21 08:26:22 -08:00
Guanghao Zhang e1f4aaeacd HBASE-11392 add/remove peer requests should be routed through master 2016-12-21 13:27:13 +08:00
Phil Yang 3826e63967 HBASE-17314 Limit total buffered size for all replication sources 2016-12-21 13:16:17 +08:00
Thiruvel Thirumoolan 3599716dff HBASE-17107: FN info should be cleaned up on region/table cleanup
Signed-off-by: Francis Liu <toffer@apache.org>
2016-12-20 18:52:49 -08:00
Vincent f8474c8d4d HBASE-17328 Properly dispose of looped replication peers
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2016-12-20 18:03:11 -08:00
tedyu 5ebb25d601 HBASE-17347 ExportSnapshot may write snapshot info file to wrong directory when specifying target name (Jianwei Cui) 2016-12-20 13:02:51 -08:00
tedyu e75dee3a21 HBASE-17344 The regionserver web UIs miss the coprocessors of RegionServerCoprocessorHost (ChiaPing Tsai) 2016-12-20 08:52:45 -08:00
zhangduo a2e967d92f HBASE-17142 Implement multi get 2016-12-20 20:41:55 +08:00
Ramkrishna db5953c6fe HBASE-17333 HBASE-17294 always ensures CompactingMemstore is default (Ram) 2016-12-20 11:01:27 +05:30
binlijin b7bde350a1 HBASE-17332 Replace HashMap to Array for DataBlockEncoding.idToEncoding 2016-12-20 09:32:28 +08:00
zhangduo d53f012bb7 Revert "HBASE-17081 Flush the entire CompactingMemStore content to disk (Anastasia"
Revert to address tests failure.

This reverts commit a2a7618d26.
2016-12-16 18:49:48 +08:00
tedyu 665fe3eef1 HBASE-17331 Avoid busy waiting in ThrottledInputStream (ChiaPing Tsai) 2016-12-18 08:15:15 -08:00
zhangduo f041306cda HBASE-17282 Reduce the redundant requests to meta table 2016-12-18 08:52:06 +08:00
Michael Stack da356069f2 HBASE-17149 Procedure v2 - Fix nonce submission
Signed-off-by: Michael Stack <stack@apache.org>
2016-12-16 21:41:18 -08:00
tedyu 2c107e4d08 HBASE-15432 TableInputFormat - support multi column family scan (Xuesen Liang) 2016-12-16 18:04:55 -08:00
tedyu 49b0bab504 HBASE-17292 Add observer notification before bulk loaded hfile is moved - addendum properly handles copied hfile 2016-12-16 15:31:30 -08:00
Michael Stack c3ce02d592 HBASE-17326 Fix findbugs warning in BufferedMutatorParams Fix above and the other hbase-server findbugs complaint about null-check 2016-12-16 13:47:58 -08:00
binlijin 6acbee179f HBASE-16398 optimize HRegion computeHDFSBlocksDistribution 2016-12-16 11:02:27 +08:00
tedyu 35f0718a41 HBASE-17292 Add observer notification before bulk loaded hfile is moved to region directory 2016-12-15 09:43:22 -08:00
Ramkrishna d1147eeb7e HBASE-15787 Change the flush related heuristics to work with offheap size
configured (Ram)
2016-12-15 22:49:52 +05:30
Ramkrishna a2a7618d26 HBASE-17081 Flush the entire CompactingMemStore content to disk (Anastasia
Braginsky)
2016-12-15 22:02:05 +05:30
tedyu 401e83cee3 HBASE-17318 Increment does not add new column if the increment amount is zero at first time writing (Guangxu Cheng) 2016-12-15 07:06:37 -08:00
anoopsamjohn f9750f2122 HBASE-17298 remove unused code in HRegion#doMiniBatchMutation. (huaxiang sun) 2016-12-15 15:54:49 +05:30
tedyu a5a6036883 HBASE-17309 Fix connection leaks in TestAcidGuarantees (huaxiang sun) 2016-12-14 20:03:44 -08:00
eshcar 691f266fc2 HBASE-17316: Addendum of HBASE-17294
Signed-off-by: Michael Stack <stack@apache.org>
2016-12-14 08:17:57 -08:00
eshcar a9310436d5 HBASE-17294: External configuration for memory compaction
Signed-off-by: Michael Stack <stack@apache.org>
2016-12-13 17:16:00 -08:00
tedyu de98f68408 HBASE-17302 The region flush request disappeared from flushQueue (Guangxu Cheng) 2016-12-13 12:45:09 -08:00
tedyu 2333596279 HBASE-17296 Provide per peer throttling for replication (Guanghao Zhang) 2016-12-13 04:20:20 -08:00
tedyu adb319f5c2 HBASE-17297 Single Filter in parenthesis cannot be parsed correctly (Xuesen Liang) 2016-12-12 12:07:34 -08:00
Jingcheng Du 1615f45b31 HBASE-17237 Override the correct compact method in HMobStore 2016-12-12 11:54:21 +08:00
tedyu 67420fe21d HBASE-16336 Removing peers seems to be leaving spare queues (Guanghao Zhang) 2016-12-10 04:33:40 -08:00
Josh Elser b554e05410 HBASE-17276 Only log stacktraces for exceptions once for updates in a batch
For large batches of updates, repeatedly logging WrongRegionExceptions,
FailedSanityCheckExceptions, and/or NoSuchColumnFamilyExceptions can
easily dominate the contents of a RegionServer log. After the first
occurence of logging the full exception, switch to logging only the
message on the exception.
2016-12-09 14:28:00 -08:00
zhangduo b3ae87bd7d HBASE-17251 Add a timeout parameter when locating region 2016-12-08 09:55:29 +08:00
Thiruvel Thirumoolan c1293cc91e HBASE-16941: FavoredNodes - Split/Merge code paths
Signed-off-by: Francis Liu <toffer@apache.org>
2016-12-07 16:38:48 -08:00
Jerry He 75567f828c HBASE-15437 Response size calculated in RPCServer for warning tooLarge responses does NOT count CellScanner payload 2016-12-07 14:47:10 -08:00
Matteo Bertozzi 1eb24e4e8e HBASE-17260 Procedure v2 - Add setOwner() overload taking a User instance 2016-12-06 10:32:43 -08:00
anoopsamjohn f112427425 HBASE-17207 Arrays.asList() with too few arguments. (John Leach) 2016-12-06 14:42:14 +05:30
Clay Baenziger c7b8b63cd1 HBASE-16700 Allow for coprocessor whitelisting
Signed-off-by: Enis Soztutar <enis@apache.org>
2016-12-05 14:46:15 -08:00
tedyu 3190605801 HBASE-17256 Rpc handler monitoring will be removed when the task queue is full (Guangxu Cheng) 2016-12-05 11:32:17 -08:00
Jerry He 39653862a4 HBASE-17221 Abstract out an interface for RpcServer.Call 2016-12-05 10:21:55 -08:00
Ramkrishna 94302a3d26 HBASE-17239 Add UnsafeByteOperations#wrap(ByteInput, int offset, int len)
API (Ram)
2016-12-05 15:13:04 +05:30
Jingcheng Du 26c2d93f77 HBASE-16841 Data loss in MOB files after cloning a snapshot and deleting that snapshot 2016-12-05 16:22:10 +08:00
anoopsamjohn e10baacd3e HBASE-17245 Replace HTableDescriptor#htd.getColumnFamilies().length with a low-cost implementation. (huaxiang sun) 2016-12-05 13:28:57 +05:30
tedyu 0d6b872d9d HBASE-17241 Avoid compacting already compacted mob files with _del files (huaxiang sun) 2016-12-04 20:23:06 -08:00
John Leach 42ff4a7164 HBASE-17231 Region#getCellCompartor -> Region#getCellComparator
Signed-off-by: Michael Stack <stack@apache.org>
2016-12-03 16:07:42 -08:00
Huaxiang Sun 03423fec6b HBASE-17243 Reuse CompactionPartitionId and avoid creating MobFileName in PartitionedMobCompactor to avoid unnecessary new objects (addendum)
Signed-off-by: Matteo Bertozzi <matteo.bertozzi@cloudera.com>
2016-12-03 07:19:02 -08:00
Huaxiang Sun efcd15bf67 HBASE-17243 Reuse CompactionPartitionId and avoid creating MobFileName in PartitionedMobCompactor to avoid unnecessary new objects
Signed-off-by: Matteo Bertozzi <matteo.bertozzi@cloudera.com>
2016-12-03 03:48:23 -08:00
Michael Stack 7775feda05 HBASE-17194 Assign the new region to the idle server after splitting (ChiaPing Tsai) 2016-12-02 11:34:14 -08:00
Michael Stack 4a20361f1a HBASE-17184 Revert "Code cleanup of LruBlockCache"
Revert "Revert "Code cleanup of LruBlockCache""
This is a revert of a revert; i.e. it is putting the change
back only this time we add the missing JIRA number

This reverts commit f3524a06bb.
2016-12-02 11:19:07 -08:00
Michael Stack f3524a06bb Revert "Code cleanup of LruBlockCache"
Missing JIRA number

This reverts commit ad857d1b77.
2016-12-02 11:18:42 -08:00
binlijin 9c13219f0d HBASE-17232 Replace HashSet with ArrayList to accumulate delayed scanners in KVHeap and StoreScanner. 2016-12-02 21:10:37 +08:00
anoopsamjohn 4b3ffffa09 HBASE-17161 MOB : Make ref cell creation more efficient. 2016-12-02 17:30:34 +05:30
Ramkrishna cb5c4c146f HBASE-17235 Improvement in creation of CIS for onheap buffer cases (Ram) 2016-12-02 14:07:16 +05:30
Ramkrishna 3b72b69d0e HBASE-17191 Make use of UnsafeByteOperations#unsafeWrap(ByteBuffer buffer)
in PBUtil#toCell(Cell cell) (Ram)
2016-12-02 12:48:06 +05:30
Stephen Yuan Jiang 0a24077841 HBASE-16119 Procedure v2 - Reimplement Merge region (Stephen Yuan Jiang) 2016-12-01 22:41:15 -08:00
Apekshit Sharma 37b30d93d8 HBASE-17111 Use Apache CLI in SnapshotInfo tool.
Change-Id: Ifd4b0200ba58737c5583dfbcf23f2549c4f233d0
2016-12-01 17:53:01 -08:00
Guanghao Zhang b3d8d06703 HBASE-17205 Add a metric for the duration of region in transition 2016-12-01 10:13:37 -08:00
Jonathan M Hsieh eeaea4aea3 HBASE-17224 Fix lots of spelling errors in HBase logging and exception messages (Grant Sohn) 2016-12-01 08:57:16 -08:00
zhangduo 540ede376b HBASE-16648 [JDK8] Use computeIfAbsent instead of get and putIfAbsent 2016-12-01 21:17:55 +08:00
zhangduo fb789b340c HBASE-17206 FSHLog may roll a new writer successfully with unflushed entries 2016-12-01 14:15:03 +08:00
John Leach 37d7b57128 HBASE-17216 Fields that can safetly be made static 2016-11-30 22:03:35 -08:00
Charlie Qiangeng Xu b2086873a9 HBASE-17110 Improve SimpleLoadBalancer to always take server-level balance into account
Signed-off-by: Yu Li <liyu@apache.org>
2016-12-01 12:12:16 +08:00
Enis Soztutar b6f5d5b85f HBASE-17151 New API to create HFile.Reader without instantiating block cache (Vladimir Rodionov) 2016-11-30 17:04:31 -08:00
Lars Francke ad857d1b77 Code cleanup of LruBlockCache
Signed-off-by: Michael Stack <stack@apache.org>
2016-11-30 12:57:14 -08:00