Apache
/
hbase
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,675 Commits 95 Branches 866 Tags 550 MiB
Commit Graph

4675 Commits

Author SHA1 Message Date
larsh 7ce77fc5d3 HBASE-5526 ^Cnfigurable file and directory based umask (Jesse Yates) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1298657 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-08 23:58:17 +00:00
mbautin 961455cd9d [jira] [HBASE-5074] Support checksums in HBase block cache 
Author: Dhruba

Summary:
HFile is enhanced to store a checksum for each block. HDFS checksum verification
is avoided while reading data into the block cache. On a checksum verification
failure, we retry the file system read request with hdfs checksums switched on
(thanks Todd).

I have a benchmark that shows that it reduces iops on the disk by about 40%. In
this experiment, the entire memory on the regionserver is allocated to the
regionserver's jvm and the OS buffer cache size is negligible. I also measured
negligible (<5%) additional cpu usage while using hbase-level checksums.

The salient points of this patch:

1. Each hfile's trailer used to have a 4 byte version number. I enhanced this so
that these 4 bytes can be interpreted as a (major version number, minor
version). Pre-existing hfiles have a minor version of 0. The new hfile format
has a minor version of 1 (thanks Mikhail). The hfile major version remains
unchanged at 2. The reason I did not introduce a new major version number is
because the code changes needed to store/read checksums do not differ much from
existing V2 writers/readers.

2. Introduced a HFileSystem object which is a encapsulates the FileSystem
objects needed to access data from hfiles and hlogs.  HDFS FileSystem objects
already had the ability to switch off checksum verifications for reads.

3. The majority of the code changes are located in hbase.io.hfie package. The
retry of a read on an initial checksum failure occurs inside the hbase.io.hfile
package itself.  The code changes to hbase.regionserver package are minor.

4. The format of a hfileblock is the header followed by the data followed by the
checksum(s). Each 16 K (configurable) size of data has a 4 byte checksum.  The
hfileblock header has two additional fields: a 4 byte value to store the
bytesPerChecksum and a 4 byte value to store the size of the user data
(excluding the checksum data). This is well explained in the associated
javadocs.

5. I added a test to test backward compatibility. I will be writing more unit
tests that triggers checksum verification failures aggressively. I have left a
few redundant log messages in the code (just for easier debugging) and will
remove them in later stage of this patch. I will also be adding metrics on
number of checksum verification failures/success in a later version of this
diff.

6. By default, hbase-level checksums are switched on and hdfs level checksums
are switched off for hfile-reads. No changes to Hlog code path here.

Test Plan: The default setting is to switch on hbase checksums for hfile-reads,
thus all existing tests actually validate the new code pieces. I will be writing
more unit tests for triggering checksum verification failures.

Reviewers: mbautin

Reviewed By: mbautin

CC: JIRA, tedyu, mbautin, dhruba, todd, stack

Differential Revision: https://reviews.facebook.net/D1521

git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1298641 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-08 22:55:49 +00:00
larsh 17a4a3fd52 HBASE-5529 MR test failures becuase MALLOC_ARENA_MAX is not set (Gregory Chanan) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1298574 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-08 20:55:57 +00:00
Zhihong Yu c4426b5b79 HBASE-5515 Add a processRow API that supports atomic multiple reads and writes on a row (Scott Chen) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1298533 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-08 19:40:27 +00:00
Zhihong Yu 990de61c9b HBASE-5514 Compile against hadoop 0.24-SNAPSHOT (Mingjie) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1298488 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-08 17:58:08 +00:00
Michael Stack 0f7285a81a HBASE-4890 fix possible NPE in HConnectionManager 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1298272 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-08 05:00:01 +00:00
Andrew Kyle Purtell edc6696bc5 HBASE-5480. Fixups to MultithreadedTableMapper for Hadoop 0.23 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1298259 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-08 02:28:39 +00:00
Michael Stack e4f4022493 HBASE-5519 Incorrect warning in splitlogmanager 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1297707 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-06 20:46:02 +00:00
Michael Stack 9bc5b685bb HBASE-5522 hbase 0.92 test artifacts are missing from Maven central 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1297681 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-06 20:25:01 +00:00
ramkrishna 6daf62029c HBASE-5531 Maven hadoop profile (version 23) needs to be updated with latest 23 snapshot (Laxman) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1297591 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-06 17:38:22 +00:00
larsh 723e8559e3 HBASE-5523 Fix Delete Timerange logic for KEEP_DELETED_CELLS 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1297343 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-06 06:17:11 +00:00
Zhihong Yu 88115f3d8b HBASE-5507 ThriftServerRunner.HbaseHandler.getRegionInfo() and getTableRegions() do not use ByteBuffer correctly (Scott Chen) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1297300 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-06 00:00:36 +00:00
Michael Stack debdfb7f22 HBASE-5524 Add a couple of more filters to our rat exclusion set 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1297277 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-05 23:10:04 +00:00
Zhihong Yu 44e348ea1b HBASE-5010 Pass region info in LoadBalancer.randomAssignment(List<ServerName> servers) (Anoop Sam John) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1297155 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-05 18:41:03 +00:00
Zhihong Yu ee6f0fb231 HBASE-5371 Introduce AccessControllerProtocol.checkPermissions(Permission[] permissons) API, addendum (Enis) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1296709 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-03 21:48:53 +00:00
Michael Stack 0132176084 HBASE-5286 bin/hbase's logic of adding Hadoop jar files to the classpath is fragile when presented with split packaged Hadoop 0.23 installation 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1296663 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-03 16:46:57 +00:00
Michael Stack 180b41f518 HBASE-5286 bin/hbase's logic of adding Hadoop jar files to the classpath is fragile when presented with split packaged Hadoop 0.23 installation 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1296661 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-03 16:45:25 +00:00
Michael Stack 4343a0a00d HBASE-5486 Warn message in HTable: Stringify the byte[] 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1296478 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-02 21:16:03 +00:00
Zhihong Yu 9a857c8e87 HBASE-5508 Add an option to allow test output to show on the terminal (Scott Chen) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1296370 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-02 18:32:36 +00:00
Zhihong Yu a2392c913e HBASE-5506 Add unit test for ThriftServerRunner.HbaseHandler.getRegionInfo() (Scott Chen) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1296365 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-02 18:30:27 +00:00
Michael Stack 514d7cc8b6 HBASE-5430 Fix licenses in 0.92.1 -- RAT plugin won't pass 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1296358 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-02 18:05:43 +00:00
Michael Stack 2c37d21a52 HBASE-5511 More doc on maven release process 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1296316 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-02 17:26:34 +00:00
larsh 9ffc748589 HBASE-5489 Addendum 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1296011 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-02 00:26:25 +00:00
Doug Meil 0c4e1dfa75 hbase-5503. performance.xml, troubleshooting.xml - adding Troubleshooting case study 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1295915 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-01 21:40:34 +00:00
Michael Stack 7147bc5424 HBASE-5489 Add HTable accessor to get regions for a key range 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1295729 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-01 18:43:29 +00:00
Michael Stack dace419238 HBASE-4403 Adopt interface stability/audience classifications from Hadoop AND HBASE-5502 region_mover.rb fails to load regions back to original server for regions only containing empty tables 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1295710 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-01 17:53:03 +00:00
Michael Stack 2b5b8bb4f8 HBASE-4403 Adopt interface stability/audience classifications from Hadoop 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1295668 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-01 16:48:44 +00:00
Michael Stack 324a2cfeb4 HBASE-5454 Refuse operations from Admin before master is initialized 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1295433 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-01 06:59:51 +00:00
Michael Stack 8eab15acf5 HBASE-5491 Remove HBaseConfiguration.create() call from coprocessor.Exec class 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1295429 13f79535-47bb-0310-9956-ffa450edef68
 2012-03-01 06:49:59 +00:00
larsh 5bc6e3e4ec Set version to 0.95-SNAPSHOT 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1295331 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-29 22:45:54 +00:00
larsh 6e612849dc HBASE-5497 Add protobuf as M/R dependency jar (mapred) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1295329 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-29 22:42:37 +00:00
Doug Meil 4f3da55d25 hbase-5496. ops_mgt.xml - fleshing out HBase Monitoring section. 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1295321 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-29 22:21:20 +00:00
Doug Meil 1ac8c6c0b8 hbase-5495 book.xml - adding an FAQ entry to General section 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1295296 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-29 21:39:10 +00:00
Doug Meil 45e46edb34 hbase-5493. external_apis.xml. adding link to thrift javadoc 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1295285 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-29 21:24:48 +00:00
Zhihong Yu ba56d88d60 HBASE-5441 HRegionThriftServer may not start because of a race-condition (Scott Chen) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1295192 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-29 17:58:35 +00:00
Michael Stack 09eaaba6b4 HBASE-5485 LogCleaner refers to non-existant SnapshotLogCleaner 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1294799 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-28 19:39:33 +00:00
Michael Stack 09a8719814 HBASE-5484 Spelling mistake in error message in HMasterCommandLine 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1294790 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-28 19:01:46 +00:00
Michael Stack 91dfa48fd4 HBASE-5477 Cannot build RPM for hbase-0.92.0 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1294316 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-27 20:40:39 +00:00
Michael Stack 4ef6cb8bc9 HBASE-5483 Allow configurable host to bind to for starting REST server from commandline 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1294229 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-27 17:01:04 +00:00
Michael Stack c176d41582 HBASE-5481 Uncaught UnknownHostException prevents HBase from starting 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1294217 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-27 16:51:26 +00:00
Michael Stack 5b61755b0f HBASE-5473 Metrics does not push pread time 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1293532 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-25 04:22:55 +00:00
Michael Stack e540751766 HBASE-5350 Fix jamon generated package names 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1293485 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-24 23:59:37 +00:00
Michael Stack 3544e1c5f7 HBASE-5351 hbase completebulkload to a new table fails in a race 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1293479 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-24 23:40:27 +00:00
larsh d229b63a28 HBASE-5455 Add test to avoid unintentional reordering of items in HbaseObjectWritable (Michael Drzal) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1293455 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-24 22:29:40 +00:00
Michael Stack d7050d3541 HBASE-5325 Expose basic information about the master-status through jmx beans 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1293416 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-24 21:06:12 +00:00
larsh a54ec1b7cc HBASE-5440 Allow Import to optionally use HFileOutputFormat 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1293101 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-24 06:47:29 +00:00
Michael Stack e531e2595c HBASE-4365 Add a decent heuristic for region size 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1293099 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-24 06:41:37 +00:00
Michael Stack ce71b8a1bc HBASE-5166 MultiThreaded Table Mapper analogous to MultiThreaded Mapper in hadoop 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1293098 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-24 06:38:18 +00:00
mbautin a01166ec92 [jira] [HBASE-5442] Use builder pattern in StoreFile and HFile 
Summary: Cleaning up the factory method explosion in HFile writer and StoreFile.
Now, adding a new parameter to HFile/StoreFile writer initialization will not
require modifying factory method invocations all over the codebase.

Test Plan:
Run unit tests
Deploy to dev cluster and run a load test

Reviewers: JIRA, stack, tedyu, Kannan, Karthik, Liyin

Reviewed By: stack

Differential Revision: https://reviews.facebook.net/D1893

git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1293095 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-24 06:06:53 +00:00
Michael Stack bbf0a0074e HBASE-5464 Log warning message when thrift calls throw exceptions 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1293093 13f79535-47bb-0310-9956-ffa450edef68
 2012-02-24 05:48:13 +00:00