Apache
/
hbase
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
4,774 Commits 95 Branches 866 Tags 550 MiB
Commit Graph

4774 Commits

Author SHA1 Message Date
mbautin ec1c804fc1 [jira] [HBASE-4218] HFile data block encoding framework and delta encoding 
implementation (Jacek Midgal, Mikhail Bautin)

Summary:

Adding a framework that allows to "encode" keys in an HFile data block. We
support two modes of encoding: (1) both on disk and in cache, and (2) in cache
only. This is distinct from compression that is already being done in HBase,
e.g. GZ or LZO. When data block encoding is enabled, we store blocks in cache
in an uncompressed but encoded form. This allows to fit more blocks in cache
and reduce the number of disk reads.

The most common example of data block encoding is delta encoding, where we take
advantage of the fact that HFile keys are sorted and share a lot of common
prefixes, and only store the delta between each pair of consecutive keys.
Initial encoding algorithms implemented are DIFF, FAST_DIFF, and PREFIX.

This is based on the delta encoding patch developed by Jacek Midgal during his
2011 summer internship at Facebook. The original patch is available here:
https://reviews.apache.org/r/2308/diff/.

Test Plan: Unit tests. Distributed load test on a five-node cluster.

Reviewers: JIRA, tedyu, stack, nspiegelberg, Kannan

Reviewed By: Kannan

CC: tedyu, todd, mbautin, stack, Kannan, mcorgan, gqchen

Differential Revision: https://reviews.facebook.net/D447



git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1236031 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-26 02:58:57 +00:00
mbautin 0d0b5b4dcc adding mbautin as a developer to pom.xml 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1235922 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-25 20:35:02 +00:00
Michael Stack 4a8b0e9059 HBASE-5278 HBase shell script refers to removed 'migrate' functionality 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1235884 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-25 19:47:13 +00:00
mbautin 07dbe903de HBASE-5230 : ensure that compactions do not cache-on-write data blocks 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1235882 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-25 19:41:07 +00:00
Michael Stack 6bcf607630 More home page edits and ref to favicon.ico 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1235539 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-24 22:54:05 +00:00
Zhihong Yu f20083b193 HBASE-4720 revert until agreement is reached on solution 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1235388 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-24 18:17:23 +00:00
Zhihong Yu f4872ddd1f HBASE-4720 Implement atomic update operations (checkAndPut, checkAndDelete) for REST client/server (Mubarak) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1235362 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-24 17:19:59 +00:00
Michael Stack 5095b451e4 HBASE-5264 Add 0.92.0 upgrade guide 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1235062 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-23 23:27:05 +00:00
Michael Stack a56613fd71 Add Riley's slow query doc from hbase-4117 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1235023 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-23 22:06:03 +00:00
Zhihong Yu 59b1b68d12 HBASE-5255 Use singletons for OperationStatus to save memory (Benoit) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1235016 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-23 21:56:45 +00:00
Doug Meil de42b015a4 hbase-5260. troubleshooting.xml - fixed incorrect XML tag 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1234936 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-23 19:04:10 +00:00
Zhihong Yu 35a60f187d HBASE-5139 Addendum handles startRow being null for the case where median is in the first region 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1234903 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-23 17:07:45 +00:00
Michael Stack e453c3c46d Add hbasecon2012 to news on front page 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1234681 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-23 05:00:28 +00:00
Doug Meil 5e0bc5edb5 hbase-5254 book.xml. Arch/Regions, added link to troubleshooting section on hbase objects on HDFS 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1234679 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-23 03:12:52 +00:00
Doug Meil 927e1eec6d hbase-5253 book.xml - adding chart of object heirarchy in Arch/Regions 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1234677 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-23 03:03:39 +00:00
Doug Meil 8f90bbef96 hbase-5252. book.xml, added section in Data Model about joins 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1234674 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-23 02:27:54 +00:00
Michael Stack d8610080d2 Edit of the mailing list/search archive section 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1234636 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-22 22:25:50 +00:00
Michael Stack 0d61857881 HBASE-5245 HBase shell should use alternate jruby if JRUBY_HOME is set, should pass along JRUBY_OPTS 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1234626 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-22 21:57:48 +00:00
Michael Stack 3724f4859b Formatting fix 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1234620 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-22 21:34:56 +00:00
ramkrishna 3331341429 HBASE-5243 LogSyncerThread not getting shutdown waiting for the interrupted flag (Ram) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1234559 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-22 17:43:16 +00:00
ramkrishna fbf165d347 HBASE-5235 HLogSplitter writer thread's streams not getting closed when any of the writer threads has exceptions.(Ram) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1234509 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-22 13:20:15 +00:00
Zhihong Yu da0f65de98 HBASE-5246 Regenerate code with thrift 0.8.0 (Scott Chen) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1234445 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-22 01:52:49 +00:00
ramkrishna fc8d2fe645 HBASE-5237 Addendum for HBASE-5160 and HBASE-4397 (Ram) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1234377 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-21 17:47:08 +00:00
Andrew Kyle Purtell 7436e93ee9 HBASE-5228. [REST] Rip out transform feature 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1234045 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-20 18:06:34 +00:00
Doug Meil c06edb6808 performance.xml - added another paragraph in "One Hot Region" section. 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1233610 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-19 22:31:16 +00:00
Doug Meil 7e363179c3 hbase-5233. [book] configuration.xml - adding example of /etc/hosts for loopback ip config 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1233602 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-19 22:15:55 +00:00
Jonathan Hsieh b77699c4ab Adding jmhsieh as a developer 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1233197 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-19 06:29:01 +00:00
Zhihong Yu 8ebcadd679 HBASE-5223 TestMetaReaderEditor is missing call to CatalogTracker.stop() 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1233101 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-18 22:43:55 +00:00
Zhihong Yu 1c8829e0ad HBASE-5208 Allow setting Scan start/stop row individually in TableInputFormat 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1233071 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-18 21:22:49 +00:00
Doug Meil c5cea96fbd hbase-5227. [book] book.xml, troubleshooting.xml - added caution about rowlocks. 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1232953 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-18 16:55:38 +00:00
Doug Meil 22bc7c8231 hbase-5226. troubleshooting.xml, client section: adding reference to client slowdown due to admin API calls. 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1232947 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-18 16:22:02 +00:00
Zhihong Yu 9b1e9d3c29 HBASE-5176 AssignmentManager#getRegion: logging nit adds a redundant '+' (Karthik K) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1232567 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-17 20:40:26 +00:00
Doug Meil ef4eb46d9f hbase-5220 troubleshooting.xml - zkcli info 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1232559 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-17 20:07:35 +00:00
larsh 01246cdc99 HBASE-5203 Group atomic put/delete operation into a single WALEdit to handle region server failures. (Lars H) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1232551 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-17 19:38:13 +00:00
Doug Meil 25dc6b2530 hbase-5218 book.xml - Arch/HFile, added link to HFile v2 info in appendix from this section. 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1232547 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-17 19:29:53 +00:00
Doug Meil 51b31b05da hbase-5216 book.xml - added detail on Arch "when to use hbase" section 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1232524 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-17 18:29:00 +00:00
Zhihong Yu c0d127fcb2 HBASE-5201 Add new files 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1232506 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-17 17:49:34 +00:00
Zhihong Yu 916b3a166c HBASE-5201 Utilize TThreadedSelectorServer and remove redundant code in ThriftServer and HRegionThriftServer (Scott Chen) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1232504 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-17 17:38:58 +00:00
Zhihong Yu 174248a26f HBASE-5153 revert due to failed Jenkins builds 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1232490 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-17 17:15:23 +00:00
Doug Meil 27bf216d9c [book] book.xml, ops_mgt.xml additional clarification that compaction does not do region merging 
also, added link to script in HBASE-1621 in Ops Mgt chapter.

git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1232408 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-17 13:32:33 +00:00
Michael Stack e04919de09 Update version on hadoop versions to include note on hadoop 1.0.0 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1232308 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-17 05:15:44 +00:00
Zhihong Yu 5ce4d352de HBASE-5153 Add retry logic in HConnectionImplementation#resetZooKeeperTrackers (Jieshan) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1232292 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-17 04:06:30 +00:00
Michael Stack e20aa2b3a6 HBASE-5204 Backward compatibility fixes for 0.92 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1231985 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-16 13:08:54 +00:00
larsh 323d17d3ae HBASE-3584 Allow atomic put/delete in one call (Lars H) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1231441 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-14 04:59:44 +00:00
Zhihong Yu a9c183ff98 HBASE-5150 Failure in a thread may not fail a test, clean up log splitting test (Jimmy Xiang) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1231364 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-13 22:44:03 +00:00
Michael Stack 0876497eb9 HBASE-5193 Use TBoundedThreadPoolServer in HRegionThriftServer 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1231347 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-13 22:23:52 +00:00
Zhihong Yu ee03d1626d HBASE-5196 Failure in region split after PONR could cause region hole (Jimmy Xiang) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1231302 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-13 21:17:30 +00:00
Zhihong Yu 8d07130c67 HBASE-5143 Fix config typo in pluggable load balancer factory (Harsh J) 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1231281 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-13 20:07:40 +00:00
Michael Stack 240e30c4ab Meetup at ebay 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1231193 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-13 17:16:09 +00:00
Michael Stack 1962888156 HBASE-5115 Change HBase color from purple to 'International Orange (Engineering)' 
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1230854 13f79535-47bb-0310-9956-ffa450edef68
 2012-01-13 01:21:52 +00:00