Commit Graph

11796 Commits

Author SHA1 Message Date
Jerry He 7c93098210 HBASE-14256 Flush task message may be confusing when region is recovered (Gabor Liptak) 2016-03-24 10:06:38 -07:00
Elliott Clark e71bc71c84 HBASE-15475 Addendum 2016-03-24 09:11:19 -07:00
tedyu 52fd70500e HBASE-15486 Avoid multiple disable/enable balancer calls while running rolling-restart.sh --graceful (Samir Ahmic) 2016-03-24 06:55:45 -07:00
Mikhail Antonov db3ba652f8 HBASE-15524 Fix NPE in client-side metrics 2016-03-23 21:23:54 -07:00
zhangduo fd5c0934b6 HBASE-15520 Fix broken TestAsyncIPC 2016-03-24 10:46:44 +08:00
Elliott Clark 925c185969 HBASE-15475 Allow TimestampsFilter to provide a seek hint
Summary:
Allow TestTimestampFilterSeekHint to provide a seek next hint.
This can be incorrect as it might skip deletes. However it can
make things much much faster.

Test Plan: Added a unit test.

Differential Revision: https://reviews.facebook.net/D55617
2016-03-23 18:56:52 -07:00
stack d7a4499dfc HBASE-15392 Single Cell Get reads two HFileBlocks
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
moreRowsMayExistAfterCell Exploit the fact a Scan is a Get Scan. Also save compares
if no non-default stopRow.

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
optimize Add doc on what is being optimized. Also, if a Get Scan, do not
optimize else we'll keep going after our row is DONE.
Another place to make use of the Get Scan fact is when we are DONE.. if
Get Scan, we can close out the scan.

M hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
Add tests for Get Scans and optimize around block loading.
2016-03-23 14:37:14 -07:00
tedyu 95e6d2276b HBASE-15300 Upgrade to zookeeper 3.4.8 2016-03-23 13:05:17 -07:00
Matteo Bertozzi 7177e4be64 HBASE-15508 Add command for exporting snapshot in hbase command script (Yufeng Jiang) 2016-03-23 10:26:32 -07:00
anoopsjohn cadfb21f4b HBASE-15064 BufferUnderflowException after last Cell fetched from an HFile Block served from L2 offheap cache - Addendum. 2016-03-23 15:03:50 +05:30
zhangduo 5fcadb86ab HBASE-15495 Connection leak in FanOutOneBlockAsyncDFSOutputHelper 2016-03-23 09:56:40 +08:00
stack 12f66e3060 HBASE-15477 Purge 'next block header' from cached blocks
When we read from HDFS, we overread to pick up the next blocks header.
Doing this saves a seek as we move through the hfile; we save having to
do an explicit seek just to read the block header every time we need to
read the body.  We used to read in the next header as part of the
current blocks buffer. This buffer was then what got persisted to
blockcache; so we were over-persisting: our block plus the next blocks'
header (33 bytes).

This patch undoes this over-persisting.

Removes support for version 1 blocks (0.2 was added in hbase-0.92.0).
Not needed any more.

There is an open question on whether checksums should be persisted
when caching. The code seems to say no but if cache is SSD backed or
backed by anything that does not do error correction, we'll want
checksums.

Adds loads of documentation.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java
  (write) Add writing from a ByteBuff.

M hbase-common/src/main/java/org/apache/hadoop/hbase/nio/ByteBuff.java
  (toString) Add one so ByteBuff looks like ByteBuffer when you click on
  it in IDE

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
  Remove support for version 1 blocks.

  Cleaned up handling of metadata added when we serialize a block to
  caches. Metadata is smaller now.

  When we serialize (used when caching), do not persist the next blocks
  header if present.

  Removed a bunch of methods, a few of which had overlapping
  functionality and others that exposed too much of our internals.
  Also removed a bunch of constructors and unified the constructors we
  had left over making them share a common init method.
  Shutdown access to defines that should only be used internally here.

  Renamed all to do w/ 'EXTRA' and 'extraSerialization' to instead talk
  about metadata saved to caches; was unclear previously what EXTRA was
  about.

  Renamed static final declarations as all uppercase.

  (readBlockDataInternal): Redid. Couldn't make sense of it previously.
  Undid heavy-duty parse of header by constructing HFileBlock. Other
  cleanups. Its 1/3rd the length it used to be. More to do in here.
2016-03-22 18:45:17 -07:00
stack a13d6e000d Revert "HBASE-15392 Single Cell Get reads two HFileBlocks M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java moreRowsMayExistAfterCell Exploit the fact a Scan is a Get Scan. Also save compares if no non-default stopRow."
Revert mistaken commit

This reverts commit 7073f69993.
2016-03-22 18:39:22 -07:00
stack c49d0caf53 Revert "HBASE-15477 Purge 'next block header' from cached blocks"
Overcommit. Revert.

This reverts commit ef94b55239.
2016-03-22 18:38:29 -07:00
stack 54a543de22 Revert "HBASE-15477 Purge 'next block header' from cached blocks"
Overcommit. Revert to fix.

This reverts commit 000117ad9f.
2016-03-22 18:37:25 -07:00
Enis Soztutar 3f3613a234 HBASE-15212 RRCServer should enforce max request size 2016-03-22 16:23:15 -07:00
stack 8af9ed7433 HBASE-15416 TestHFileBackedByBucketCache is flakey since it went in 2016-03-22 15:44:25 -07:00
Enis Soztutar b3fe4ed16c HBASE-15412 Add average region size metric (Alicia Ying Shu) 2016-03-22 14:46:27 -07:00
zhangduo 2348478506 HBASE-15360 addendum fix testCoDelScheduling
Signed-off-by: stack <stack@apache.org>
2016-03-22 13:52:26 -07:00
stack 000117ad9f HBASE-15477 Purge 'next block header' from cached blocks
When we read from HDFS, we overread to pick up the next blocks header.
Doing this saves a seek as we move through the hfile; we save having to
do an explicit seek just to read the block header every time we need to
read the body.  We used to read in the next header as part of the
current blocks buffer. This buffer was then what got persisted to
blockcache; so we were over-persisting: our block plus the next blocks'
header (33 bytes).

This patch undoes this over-persisting.

Removes support for version 1 blocks (0.2 was added in hbase-0.92.0).
Not needed any more.

There is an open question on whether checksums should be persisted
when caching. The code seems to say no but if cache is SSD backed or
backed by anything that does not do error correction, we'll want
checksums.

Adds loads of documentation.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java
  (write) Add writing from a ByteBuff.

M hbase-common/src/main/java/org/apache/hadoop/hbase/nio/ByteBuff.java
  (toString) Add one so ByteBuff looks like ByteBuffer when you click on
  it in IDE

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
  Remove support for version 1 blocks.

  Cleaned up handling of metadata added when we serialize a block to
  caches. Metadata is smaller now.

  When we serialize (used when caching), do not persist the next blocks
  header if present.

  Removed a bunch of methods, a few of which had overlapping
  functionality and others that exposed too much of our internals.
  Also removed a bunch of constructors and unified the constructors we
  had left over making them share a common init method.
  Shutdown access to defines that should only be used internally here.

  Renamed all to do w/ 'EXTRA' and 'extraSerialization' to instead talk
  about metadata saved to caches; was unclear previously what EXTRA was
  about.

  Renamed static final declarations as all uppercase.

  (readBlockDataInternal): Redid. Couldn't make sense of it previously.
  Undid heavy-duty parse of header by constructing HFileBlock. Other
  cleanups. Its 1/3rd the length it used to be. More to do in here.
2016-03-22 13:42:38 -07:00
stack ef94b55239 HBASE-15477 Purge 'next block header' from cached blocks
When we read from HDFS, we overread to pick up the next blocks header.
Doing this saves a seek as we move through the hfile; we save having to
do an explicit seek just to read the block header every time we need to
read the body.  We used to read in the next header as part of the
current blocks buffer. This buffer was then what got persisted to
blockcache; so we were over-persisting: our block plus the next blocks'
header (33 bytes).

This patch undoes this over-persisting.

Removes support for version 1 blocks (0.2 was added in hbase-0.92.0).
Not needed any more.

There is an open question on whether checksums should be persisted
when caching. The code seems to say no but if cache is SSD backed or
backed by anything that does not do error correction, we'll want
checksums.

Adds loads of documentation.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java
  (write) Add writing from a ByteBuff.

M hbase-common/src/main/java/org/apache/hadoop/hbase/nio/ByteBuff.java
  (toString) Add one so ByteBuff looks like ByteBuffer when you click on
  it in IDE

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
  Remove support for version 1 blocks.

  Cleaned up handling of metadata added when we serialize a block to
  caches. Metadata is smaller now.

  When we serialize (used when caching), do not persist the next blocks
  header if present.

  Removed a bunch of methods, a few of which had overlapping
  functionality and others that exposed too much of our internals.
  Also removed a bunch of constructors and unified the constructors we
  had left over making them share a common init method.
  Shutdown access to defines that should only be used internally here.

  Renamed all to do w/ 'EXTRA' and 'extraSerialization' to instead talk
  about metadata saved to caches; was unclear previously what EXTRA was
  about.

  Renamed static final declarations as all uppercase.

  (readBlockDataInternal): Redid. Couldn't make sense of it previously.
  Undid heavy-duty parse of header by constructing HFileBlock. Other
  cleanups. Its 1/3rd the length it used to be. More to do in here.
2016-03-22 13:34:51 -07:00
stack 891569bbd9 Add javadoc on how BBIOEngine works (by Anoop Sam John) 2016-03-22 13:01:13 -07:00
stack 7073f69993 HBASE-15392 Single Cell Get reads two HFileBlocks M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java moreRowsMayExistAfterCell Exploit the fact a Scan is a Get Scan. Also save compares if no non-default stopRow.
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
optimize Add doc on what is being optimized. Also, if a Get Scan, do not
optimize else we'll keep going after our row is DONE.
Another place to make use of the Get Scan fact is when we are DONE.. if
Get Scan, we can close out the scan.

M hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
Add tests for Get Scans and optimize around block loading.
2016-03-22 13:00:41 -07:00
Misty Stanley-Jones b6e1f63074 HBASE-15447 Improve javadocs description for Delete methods <Wellington Chevreuil> 2016-03-22 12:56:30 -07:00
tedyu 47471c35e3 HBASE-15488 Add ACL for setting split merge switch 2016-03-22 10:38:13 -07:00
Enis Soztutar 797562e6c3 HBASE-15464 Flush / Compaction metrics revisited 2016-03-21 17:50:02 -07:00
Elliott Clark 75252af3a9 HBASE-15460 Fix infer issues in hbase-common
Summary: Fix issues found by static analysis

Test Plan: Unit test pass

Differential Revision: https://reviews.facebook.net/D55551
2016-03-21 11:51:50 -07:00
stack 1123be9181 HBASE-15502 Skeleton unit test to copy/paste 2016-03-21 11:27:18 -07:00
tedyu f1d453599a HBASE-15433 SnapshotManager#restoreSnapshot not update table and region count quota correctly when encountering exception (Jianwei Cui) 2016-03-21 03:46:23 -07:00
Sean Busbey f48c92d143 HBASE-15478 add comments to syncRunnerIndex handling explaining constraints on possible values.
Signed-off-by: zhangduo <zhangduo@apache.org>
2016-03-21 00:28:15 -05:00
stack 64204b96c1 HBASE-15349 Update surefire version to 2.19.1; Trying a REVERT to see if this responsible for plethora of test timeouts on hadoopqa 2016-03-20 11:22:22 -07:00
tedyu dc6cd0bb74 HBASE-15479 No more garbage or beware of autoboxing (Vladimir Rodionov) 2016-03-19 17:22:53 -07:00
Yu Li d2d8058ab3 HBASE-15451 Remove unnecessary wait in MVCC 2016-03-19 21:24:07 +08:00
tedyu 3a6d683d63 HBASE-15456 CreateTableProcedure/ModifyTableProcedure needs to fail when there is no family in table descriptor (huaxiang sun) 2016-03-18 10:21:43 -07:00
tedyu 8c031e9d6b HBASE-15456 Revert due to unit test failure in thrift module 2016-03-17 17:36:44 -07:00
tedyu fbf58f330b HBASE-15325 ResultScanner allowing partial result will miss the rest of the row if the region is moved between two rpc requests (Phil Yang) 2016-03-17 11:03:29 -07:00
tedyu 2b8a7f8d7b HBASE-15334 Add avro support for spark hbase connector (Zhan Zhang) 2016-03-17 09:11:38 -07:00
Mikhail Antonov a2c99b133f HBASE-15390 Unnecessary MetaCache evictions cause elevated number of requests to meta (ADDENDUM) 2016-03-17 01:08:36 -07:00
Yu Li abc27c6061 Add liyu to pom.xml 2016-03-17 13:05:16 +08:00
tedyu 448ac5b37c HBASE-12940 Expose listPeerConfigs and getPeerConfig to the HBase shell (Geoffrey Jacoby) 2016-03-16 18:42:52 -07:00
Elliott Clark ac1a7a4a78 HBASE-15470 Add a setting for Priority queue length
Summary:
Move the config keys to one place
Make Two different config keys. One for default, one for priority

Test Plan: unit tests

Differential Revision: https://reviews.facebook.net/D55575
2016-03-16 16:52:04 -07:00
Elliott Clark ecec35ae4e HBASE-15441 Fix WAL splitting when region has moved multiple times
Summary:
Currently WAL splitting is broken when a region has been opened multiple times in recent minutes.

Region open and region close write event markers to the wal. These markers should have the sequence id in them. However it is currently getting 1. That means that if a region has moved multiple times in the last few mins then multiple split log workers will try and create the recovered edits file for sequence id 1. One of the workers will fail and on failing they will delete the recovered edits. Causing all split wal attempts to fail.

We need to:

It appears that the close event with a sequence id of one is coming from region warm up.

This patch fixes that by making sure the close on warm up doesn't happen. Also splitting will ignore any of the events that are already in the logs.

Test Plan: Unit tests pass

Differential Revision: https://reviews.facebook.net/D55557
2016-03-16 16:26:09 -07:00
Mikhail Antonov 3adcc750e3 HBASE-15390 Unnecessary MetaCache evictions cause elevated number of requests to meta 2016-03-16 16:22:36 -07:00
Matteo Bertozzi 3bf0945a11 HBASE-15430 Failed taking snapshot - Manifest proto-message too large (JunHo Cho) 2016-03-16 08:52:02 -07:00
stack 9eb25bd5c9 HBASE-15360 Fix flaky TestSimpleRpcScheduler (Duo Zhang) 2016-03-16 08:04:10 -07:00
tedyu f120602f1e HBASE-15456 CreateTableProcedure/ModifyTableProcedure needs to fail when there is no family in table descriptor (huaxiang sun) 2016-03-15 16:15:03 -07:00
Enis Soztutar 51259fe4a5 HBASE-15377 Per-RS Get metric is time based, per-region metric is size-based (Heng Chen) 2016-03-15 11:22:18 -07:00
tedyu 8fcc1e8e95 HBASE-15463 Region normalizer should check whether split/merge is enabled 2016-03-15 09:03:27 -07:00
Enis Soztutar ca816f0780 HBASE-6721 RegionServer Group based Assignment (Francis Liu) 2016-03-14 18:28:50 -07:00
tedyu 122e6f5793 HBASE-15439 getMaximumAllowedTimeBetweenRuns in ScheduledChore ignores the TimeUnit (Jingcheng Du) 2016-03-14 07:09:58 -07:00