2007-05-18 17:19:08 -04:00
|
|
|
HBase Change Log
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
Trunk (unreleased changes)
|
2007-10-15 14:42:28 -04:00
|
|
|
INCOMPATIBLE CHANGES
|
|
|
|
|
|
|
|
|
|
NEW FEATURES
|
2007-10-16 15:11:07 -04:00
|
|
|
HADOOP-2061 Add new Base64 dialects
|
2007-10-25 12:12:51 -04:00
|
|
|
HADOOP-2084 Add a LocalHBaseCluster
|
2007-10-15 14:42:28 -04:00
|
|
|
|
|
|
|
|
OPTIMIZATIONS
|
|
|
|
|
|
|
|
|
|
BUG FIXES
|
2007-10-15 17:05:23 -04:00
|
|
|
HADOOP-2059 In tests, exceptions in min dfs shutdown should not fail test
|
|
|
|
|
(e.g. nightly #272)
|
2007-10-16 20:09:55 -04:00
|
|
|
HADOOP-2064 TestSplit assertion and NPE failures (Patch build #952 and #953)
|
2007-10-18 18:21:09 -04:00
|
|
|
HADOOP-2056 A table with row keys containing colon fails to split regions
|
2007-10-20 04:06:52 -04:00
|
|
|
HADOOP-2079 HADOOP-2056 Fix generated HLog, HRegion names
|
2007-10-30 15:08:23 -04:00
|
|
|
HADOOP-2124 Use of `hostname` does not work on Cygwin in some cases
|
2007-11-01 11:01:43 -04:00
|
|
|
HADOOP-2083 TestTableIndex failed in #970 and #956
|
2007-11-01 18:49:08 -04:00
|
|
|
HADOOP-2109 Fixed race condition in processing server lease timeout.
|
2007-11-01 20:43:07 -04:00
|
|
|
HADOOP-2137 hql.jsp : The character 0x19 is not valid
|
2007-11-03 21:47:24 -04:00
|
|
|
HADOOP-2109 Fix another race condition in processing dead servers,
|
|
|
|
|
Fix error online meta regions: was using region name and not
|
2007-11-05 00:06:35 -05:00
|
|
|
startKey as key for map.put. Change TestRegionServerExit to
|
|
|
|
|
always kill the region server for the META region. This makes
|
|
|
|
|
the test more deterministic and getting META reassigned was
|
|
|
|
|
problematic.
|
2007-11-06 15:47:35 -05:00
|
|
|
HADOOP-2155 Method expecting HBaseConfiguration throws NPE when given Configuration
|
2007-11-06 15:48:27 -05:00
|
|
|
HADOOP-2156 BufferUnderflowException for un-named HTableDescriptors
|
2007-11-09 16:57:24 -05:00
|
|
|
HADOOP-2161 getRow() is orders of magnitudes slower than get(), even on rows
|
|
|
|
|
with one column (Clint Morgan and Stack)
|
2007-11-13 18:38:21 -05:00
|
|
|
HADOOP-2040 Hudson hangs AFTER test has finished
|
2007-11-25 22:05:37 -05:00
|
|
|
HADOOP-2274 Excess synchronization introduced by HADOOP-2139 negatively
|
|
|
|
|
impacts performance
|
2007-11-26 19:23:37 -05:00
|
|
|
HADOOP-2196 Fix how hbase sits in hadoop 'package' product
|
2007-11-27 02:11:00 -05:00
|
|
|
HADOOP-2276 Address regression caused by HADOOP-2274, fix HADOOP-2173 (When
|
|
|
|
|
the master times out a region servers lease, the region server
|
|
|
|
|
may not restart)
|
2007-10-15 14:42:28 -04:00
|
|
|
|
|
|
|
|
IMPROVEMENTS
|
|
|
|
|
HADOOP-2401 Add convenience put method that takes writable
|
|
|
|
|
(Johan Oskarsson via Stack)
|
2007-11-06 22:48:13 -05:00
|
|
|
HADOOP-2074 Simple switch to enable DEBUG level-logging in hbase
|
2007-10-26 12:16:53 -04:00
|
|
|
HADOOP-2088 Make hbase runnable in $HADOOP_HOME/build(/contrib/hbase)
|
2007-10-31 21:48:46 -04:00
|
|
|
HADOOP-2126 Use Bob Jenkins' hash for bloom filters
|
2007-11-06 15:45:28 -05:00
|
|
|
HADOOP-2157 Make Scanners implement Iterable
|
2007-11-09 20:09:33 -05:00
|
|
|
HADOOP-2176 Htable.deleteAll documentation is ambiguous
|
HADOOP-2139 (phase 1) Increase parallelism in region servers
There are a lot of changes in this patch. The memcache has been changed from a per/region object to a per/column object, and HLocking has been removed since we do not have to maintain any locks across RPC calls.
This necessitated major changes to HRegion and HStore
Additionally there were many changes required to the unit tests since they tend to exploit some private interfaces that weren't designed to be public. Some of those interfaces changed so the test cases did as well.
This patch is the result of extensive analysis of the multiple threads in HBase that contend for shared resources: updates, reads, scanners, cache flushing, compaction and region splitting.
Many of the tests are timing sensitive, and since we tend to make "dormant" intervals as short as possible to speed up the Hudson build, we may go through several iterations of getting them right before Hudson is happy. This is especially true since two test cases failed on my dual cpu windows machine while running the tests under Ant, but ran fine under Eclipse.
However, now that the tests are passing locally, I believe the changes are doing the right thing, but may require some parameter tweaks.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@596835 13f79535-47bb-0310-9956-ffa450edef68
2007-11-20 16:53:30 -05:00
|
|
|
HADOOP-2139 (phase 1) Increase parallelism in region servers.
|
2007-11-23 14:16:45 -05:00
|
|
|
HADOOP-2267 [Hbase Shell] Change the prompt's title from 'hbase' to 'hql'.
|
2007-11-25 02:17:38 -05:00
|
|
|
HADOOP-2139 (phase 2) Make region server more event driven
|
2007-11-27 01:12:46 -05:00
|
|
|
HADOOP-2289 Useless efforts of looking for the non-existant table in select
|
|
|
|
|
command.
|
2007-11-27 03:35:08 -05:00
|
|
|
HADOOP-2262 HADOOP-2261 fail fast on non-existing table, change abort to
|
|
|
|
|
function after commit even if commit was successful
|
2007-10-15 14:42:28 -04:00
|
|
|
|
2007-11-06 15:45:28 -05:00
|
|
|
Release 0.15.1
|
|
|
|
|
Branch 0.15
|
2007-05-18 17:19:08 -04:00
|
|
|
|
2007-08-18 12:29:09 -04:00
|
|
|
INCOMPATIBLE CHANGES
|
2007-09-21 05:27:46 -04:00
|
|
|
HADOOP-1931 Hbase scripts take --ARG=ARG_VALUE when should be like hadoop
|
|
|
|
|
and do ---ARG ARG_VALUE
|
2007-08-18 12:29:09 -04:00
|
|
|
|
|
|
|
|
NEW FEATURES
|
2007-08-24 15:32:29 -04:00
|
|
|
HADOOP-1768 FS command using Hadoop FsShell operations
|
2007-08-29 19:39:52 -04:00
|
|
|
(Edward Yoon via Stack)
|
2007-09-10 11:56:16 -04:00
|
|
|
HADOOP-1784 Delete: Fix scanners and gets so they work properly in presence
|
|
|
|
|
of deletes. Added a deleteAll to remove all cells equal to or
|
|
|
|
|
older than passed timestamp. Fixed compaction so deleted cells
|
|
|
|
|
do not make it out into compacted output. Ensure also that
|
|
|
|
|
versions > column max are dropped compacting.
|
2007-09-13 00:01:05 -04:00
|
|
|
HADOOP-1720 Addition of HQL (Hbase Query Language) support in Hbase Shell.
|
|
|
|
|
The old shell syntax has been replaced by HQL, a small SQL-like
|
|
|
|
|
set of operators, for creating, altering, dropping, inserting,
|
|
|
|
|
deleting, and selecting, etc., data in hbase.
|
|
|
|
|
(Inchul Song and Edward Yoon via Stack)
|
2007-09-24 17:38:10 -04:00
|
|
|
HADOOP-1913 Build a Lucene index on an HBase table
|
|
|
|
|
(Ning Li via Stack)
|
2007-10-05 23:09:50 -04:00
|
|
|
HADOOP-1957 Web UI with report on cluster state and basic browsing of tables
|
2007-08-18 12:29:09 -04:00
|
|
|
|
|
|
|
|
OPTIMIZATIONS
|
|
|
|
|
|
|
|
|
|
BUG FIXES
|
2007-08-22 12:59:43 -04:00
|
|
|
HADOOP-1527 Region server won't start because logdir exists
|
2007-08-20 18:35:27 -04:00
|
|
|
HADOOP-1723 If master asks region server to shut down, by-pass return of
|
2007-08-29 19:39:52 -04:00
|
|
|
shutdown message
|
2007-08-18 12:29:09 -04:00
|
|
|
HADOOP-1729 Recent renaming or META tables breaks hbase shell
|
2007-08-18 14:04:53 -04:00
|
|
|
HADOOP-1730 unexpected null value causes META scanner to exit (silently)
|
2007-08-22 00:05:08 -04:00
|
|
|
HADOOP-1747 On a cluster, on restart, regions multiply assigned
|
2007-08-24 12:24:40 -04:00
|
|
|
HADOOP-1776 Fix for sporadic compaction failures closing and moving
|
2007-08-29 19:39:52 -04:00
|
|
|
compaction result
|
2007-08-24 20:42:45 -04:00
|
|
|
HADOOP-1780 Regions are still being doubly assigned
|
2007-08-28 18:08:56 -04:00
|
|
|
HADOOP-1797 Fix NPEs in MetaScanner constructor
|
2007-08-29 00:30:58 -04:00
|
|
|
HADOOP-1799 Incorrect classpath in binary version of Hadoop
|
2007-08-29 16:06:41 -04:00
|
|
|
HADOOP-1805 Region server hang on exit
|
2007-08-29 19:39:52 -04:00
|
|
|
HADOOP-1785 TableInputFormat.TableRecordReader.next has a bug
|
|
|
|
|
(Ning Li via Stack)
|
2007-08-30 17:28:25 -04:00
|
|
|
HADOOP-1800 output should default utf8 encoding
|
2007-09-07 23:09:42 -04:00
|
|
|
HADOOP-1801 When hdfs is yanked out from under hbase, hbase should go down gracefully
|
2007-09-15 11:14:53 -04:00
|
|
|
HADOOP-1813 OOME makes zombie of region server
|
2007-08-30 18:09:13 -04:00
|
|
|
HADOOP-1814 TestCleanRegionServerExit fails too often on Hudson
|
2007-09-24 18:52:16 -04:00
|
|
|
HADOOP-1820 Regionserver creates hlogs without bound
|
2007-09-30 19:46:04 -04:00
|
|
|
(reverted 2007/09/25) (Fixed 2007/09/30)
|
2007-09-01 02:22:01 -04:00
|
|
|
HADOOP-1821 Replace all String.getBytes() with String.getBytes("UTF-8")
|
2007-09-04 18:26:02 -04:00
|
|
|
HADOOP-1832 listTables() returns duplicate tables
|
2007-09-05 12:00:01 -04:00
|
|
|
HADOOP-1834 Scanners ignore timestamp passed on creation
|
2007-09-07 03:28:42 -04:00
|
|
|
HADOOP-1847 Many HBase tests do not fail well.
|
2007-09-07 23:04:54 -04:00
|
|
|
HADOOP-1847 Many HBase tests do not fail well. (phase 2)
|
2007-09-11 19:39:17 -04:00
|
|
|
HADOOP-1870 Once file system failure has been detected, don't check it again
|
|
|
|
|
and get on with shutting down the hbase cluster.
|
2007-09-15 17:27:10 -04:00
|
|
|
HADOOP-1888 NullPointerException in HMemcacheScanner (reprise)
|
2007-09-19 12:45:01 -04:00
|
|
|
HADOOP-1903 Possible data loss if Exception happens between snapshot and
|
|
|
|
|
flush to disk.
|
|
|
|
|
HADOOP-1920 Wrapper scripts broken when hadoop in one location and hbase in
|
|
|
|
|
another
|
2007-09-20 03:37:34 -04:00
|
|
|
HADOOP-1923, HADOOP-1924 a) tests fail sporadically because set up and tear
|
|
|
|
|
down is inconsistent b) TestDFSAbort failed in nightly #242
|
2007-09-21 16:03:54 -04:00
|
|
|
HADOOP-1929 Add hbase-default.xml to hbase jar
|
2007-09-25 19:04:08 -04:00
|
|
|
HADOOP-1941 StopRowFilter throws NPE when passed null row
|
2007-09-30 12:09:38 -04:00
|
|
|
HADOOP-1966 Make HBase unit tests more reliable in the Hudson environment.
|
2007-10-01 15:45:02 -04:00
|
|
|
HADOOP-1975 HBase tests failing with java.lang.NumberFormatException
|
2007-10-04 15:58:56 -04:00
|
|
|
HADOOP-1990 Regression test instability affects nightly and patch builds
|
2007-10-04 23:26:32 -04:00
|
|
|
HADOOP-1996 TestHStoreFile fails on windows if run multiple times
|
2007-10-05 05:39:32 -04:00
|
|
|
HADOOP-1937 When the master times out a region server's lease, it is too
|
|
|
|
|
aggressive in reclaiming the server's log.
|
2007-10-08 12:25:33 -04:00
|
|
|
HADOOP-2004 webapp hql formatting bugs
|
2007-10-08 23:25:00 -04:00
|
|
|
HADOOP_2011 Make hbase daemon scripts take args in same order as hadoop
|
|
|
|
|
daemon scripts
|
2007-10-09 17:48:11 -04:00
|
|
|
HADOOP-2017 TestRegionServerAbort failure in patch build #903 and
|
|
|
|
|
nightly #266
|
2007-10-11 10:45:36 -04:00
|
|
|
HADOOP-2029 TestLogRolling fails too often in patch and nightlies
|
2007-10-12 13:46:37 -04:00
|
|
|
HADOOP-2038 TestCleanRegionExit failed in patch build #927
|
2007-08-18 12:29:09 -04:00
|
|
|
|
|
|
|
|
IMPROVEMENTS
|
2007-08-21 16:53:24 -04:00
|
|
|
HADOOP-1737 Make HColumnDescriptor data publically members settable
|
2007-08-22 19:59:30 -04:00
|
|
|
HADOOP-1746 Clean up findbugs warnings
|
2007-08-27 19:35:15 -04:00
|
|
|
HADOOP-1757 Bloomfilters: single argument constructor, use enum for bloom
|
|
|
|
|
filter types
|
2007-08-30 20:37:46 -04:00
|
|
|
HADOOP-1760 Use new MapWritable and SortedMapWritable classes from
|
|
|
|
|
org.apache.hadoop.io
|
2007-09-10 13:42:50 -04:00
|
|
|
HADOOP-1793 (Phase 1) Remove TestHClient (Phase2) remove HClient.
|
2007-09-05 18:35:13 -04:00
|
|
|
HADOOP-1794 Remove deprecated APIs
|
2007-08-30 11:39:11 -04:00
|
|
|
HADOOP-1802 Startup scripts should wait until hdfs as cleared 'safe mode'
|
2007-09-07 16:32:45 -04:00
|
|
|
HADOOP-1833 bin/stop_hbase.sh returns before it completes
|
|
|
|
|
(Izaak Rubin via Stack)
|
2007-09-05 12:12:39 -04:00
|
|
|
HADOOP-1835 Updated Documentation for HBase setup/installation
|
2007-09-07 16:32:45 -04:00
|
|
|
(Izaak Rubin via Stack)
|
2007-09-10 19:27:34 -04:00
|
|
|
HADOOP-1868 Make default configuration more responsive
|
2007-09-12 12:48:54 -04:00
|
|
|
HADOOP-1884 Remove useless debugging log messages from hbase.mapred
|
2007-09-16 22:57:31 -04:00
|
|
|
HADOOP-1856 Add Jar command to hbase shell using Hadoop RunJar util
|
|
|
|
|
(Edward Yoon via Stack)
|
2007-09-28 17:10:49 -04:00
|
|
|
HADOOP-1928 Have master pass the regionserver the filesystem to use
|
|
|
|
|
HADOOP-1789 Output formatting
|
2007-10-02 17:42:07 -04:00
|
|
|
HADOOP-1960 If a region server cannot talk to the master before its lease
|
|
|
|
|
times out, it should shut itself down
|
2007-10-11 16:33:53 -04:00
|
|
|
HADOOP-2035 Add logo to webapps
|
2007-08-18 12:29:09 -04:00
|
|
|
|
|
|
|
|
|
|
|
|
|
Below are the list of changes before 2007-08-18
|
|
|
|
|
|
2007-05-18 17:19:08 -04:00
|
|
|
1. HADOOP-1384. HBase omnibus patch. (jimk, Vuk Ercegovac, and Michael Stack)
|
2007-05-21 16:35:19 -04:00
|
|
|
2. HADOOP-1402. Fix javadoc warnings in hbase contrib. (Michael Stack)
|
2007-05-21 18:23:03 -04:00
|
|
|
3. HADOOP-1404. HBase command-line shutdown failing (Michael Stack)
|
2007-05-22 01:30:07 -04:00
|
|
|
4. HADOOP-1397. Replace custom hbase locking with
|
|
|
|
|
java.util.concurrent.locks.ReentrantLock (Michael Stack)
|
2007-05-22 09:51:10 -04:00
|
|
|
5. HADOOP-1403. HBase reliability - make master and region server more fault
|
|
|
|
|
tolerant.
|
2007-05-22 16:24:22 -04:00
|
|
|
6. HADOOP-1418. HBase miscellaneous: unit test for HClient, client to do
|
|
|
|
|
'Performance Evaluation', etc.
|
2007-05-23 11:14:00 -04:00
|
|
|
7. HADOOP-1420, HADOOP-1423. Findbugs changes, remove reference to removed
|
|
|
|
|
class HLocking.
|
2007-05-23 17:30:25 -04:00
|
|
|
8. HADOOP-1424. TestHBaseCluster fails with IllegalMonitorStateException. Fix
|
|
|
|
|
regression introduced by HADOOP-1397.
|
2007-05-24 13:12:00 -04:00
|
|
|
9. HADOOP-1426. Make hbase scripts executable + add test classes to CLASSPATH.
|
2007-05-24 18:37:26 -04:00
|
|
|
10. HADOOP-1430. HBase shutdown leaves regionservers up.
|
2007-05-29 13:23:15 -04:00
|
|
|
11. HADOOP-1392. Part1: includes create/delete table; enable/disable table;
|
|
|
|
|
add/remove column.
|
2007-06-03 01:09:21 -04:00
|
|
|
12. HADOOP-1392. Part2: includes table compaction by merging adjacent regions
|
|
|
|
|
that have shrunk in size.
|
2007-06-04 13:14:10 -04:00
|
|
|
13. HADOOP-1445 Support updates across region splits and compactions
|
2007-06-05 11:11:57 -04:00
|
|
|
14. HADOOP-1460 On shutdown IOException with complaint 'Cannot cancel lease
|
|
|
|
|
that is not held'
|
2007-06-11 12:46:27 -04:00
|
|
|
15. HADOOP-1421 Failover detection, split log files.
|
|
|
|
|
For the files modified, also clean up javadoc, class, field and method
|
|
|
|
|
visibility (HADOOP-1466)
|
2007-06-11 16:52:29 -04:00
|
|
|
16. HADOOP-1479 Fix NPE in HStore#get if store file only has keys < passed key.
|
2007-06-11 19:41:48 -04:00
|
|
|
17. HADOOP-1476 Distributed version of 'Performance Evaluation' script
|
2007-06-12 17:08:27 -04:00
|
|
|
18. HADOOP-1469 Asychronous table creation
|
2007-06-14 03:40:49 -04:00
|
|
|
19. HADOOP-1415 Integrate BSD licensed bloom filter implementation.
|
2007-06-14 18:08:56 -04:00
|
|
|
20. HADOOP-1465 Add cluster stop/start scripts for hbase
|
2007-06-14 20:39:34 -04:00
|
|
|
21. HADOOP-1415 Provide configurable per-column bloom filters - part 2.
|
2007-06-18 18:59:14 -04:00
|
|
|
22. HADOOP-1498. Replace boxed types with primitives in many places.
|
2007-06-25 18:54:56 -04:00
|
|
|
23. HADOOP-1509. Made methods/inner classes in HRegionServer and HClient protected
|
|
|
|
|
instead of private for easier extension. Also made HRegion and HRegionInfo public too.
|
|
|
|
|
Added an hbase-default.xml property for specifying what HRegionInterface extension to use
|
2007-07-05 15:50:04 -04:00
|
|
|
for proxy server connection. (James Kennedy via Jim Kellerman)
|
2007-06-27 00:50:10 -04:00
|
|
|
24. HADOOP-1534. [hbase] Memcache scanner fails if start key not present
|
2007-06-27 18:37:07 -04:00
|
|
|
25. HADOOP-1537. Catch exceptions in testCleanRegionServerExit so we can see
|
|
|
|
|
what is failing.
|
2007-06-28 18:29:07 -04:00
|
|
|
26. HADOOP-1543 [hbase] Add HClient.tableExists
|
2007-07-05 15:50:04 -04:00
|
|
|
27. HADOOP-1519 [hbase] map/reduce interface for HBase. (Vuk Ercegovac and
|
|
|
|
|
Jim Kellerman)
|
2007-07-01 20:47:13 -04:00
|
|
|
28. HADOOP-1523 Hung region server waiting on write locks
|
2007-07-04 00:48:18 -04:00
|
|
|
29. HADOOP-1560 NPE in MiniHBaseCluster on Windows
|
2007-07-05 15:50:04 -04:00
|
|
|
30. HADOOP-1531 Add RowFilter to HRegion.HScanner
|
|
|
|
|
Adds a row filtering interface and two implemenentations: A page scanner,
|
|
|
|
|
and a regex row/column-data matcher. (James Kennedy via Stack)
|
2007-07-05 16:12:37 -04:00
|
|
|
31. HADOOP-1566 Key-making utility
|
2007-07-07 02:09:25 -04:00
|
|
|
32. HADOOP-1415 Provide configurable per-column bloom filters.
|
|
|
|
|
HADOOP-1466 Clean up visibility and javadoc issues in HBase.
|
2007-07-09 23:06:32 -04:00
|
|
|
33. HADOOP-1538 Provide capability for client specified time stamps in HBase
|
|
|
|
|
HADOOP-1466 Clean up visibility and javadoc issues in HBase.
|
2007-07-11 10:23:00 -04:00
|
|
|
34. HADOOP-1589 Exception handling in HBase is broken over client server connections
|
2007-07-11 17:54:15 -04:00
|
|
|
35. HADOOP-1375 a simple parser for hbase (Edward Yoon via Stack)
|
2007-07-12 18:08:25 -04:00
|
|
|
36. HADOOP-1600 Update license in HBase code
|
2007-07-12 19:22:20 -04:00
|
|
|
37. HADOOP-1589 Exception handling in HBase is broken over client server
|
2007-07-14 16:08:01 -04:00
|
|
|
38. HADOOP-1574 Concurrent creates of a table named 'X' all succeed
|
2007-07-14 18:25:32 -04:00
|
|
|
39. HADOOP-1581 Un-openable tablename bug
|
2007-07-14 19:53:39 -04:00
|
|
|
40. HADOOP-1607 [shell] Clear screen command (Edward Yoon via Stack)
|
2007-07-14 20:26:15 -04:00
|
|
|
41. HADOOP-1614 [hbase] HClient does not protect itself from simultaneous updates
|
2007-07-16 18:19:59 -04:00
|
|
|
42. HADOOP-1468 Add HBase batch update to reduce RPC overhead
|
2007-07-17 20:38:58 -04:00
|
|
|
43. HADOOP-1616 Sporadic TestTable failures
|
2007-07-17 22:26:03 -04:00
|
|
|
44. HADOOP-1615 Replacing thread notification-based queue with
|
|
|
|
|
java.util.concurrent.BlockingQueue in HMaster, HRegionServer
|
2007-07-21 01:06:13 -04:00
|
|
|
45. HADOOP-1606 Updated implementation of RowFilterSet, RowFilterInterface
|
|
|
|
|
(Izaak Rubin via Stack)
|
2007-07-23 17:33:47 -04:00
|
|
|
46. HADOOP-1579 Add new WhileMatchRowFilter and StopRowFilter filters
|
|
|
|
|
(Izaak Rubin via Stack)
|
2007-07-23 19:33:05 -04:00
|
|
|
47. HADOOP-1637 Fix to HScanner to Support Filters, Add Filter Tests to
|
|
|
|
|
TestScanner2 (Izaak Rubin via Stack)
|
2007-07-26 10:15:17 -04:00
|
|
|
48. HADOOP-1516 HClient fails to readjust when ROOT or META redeployed on new
|
|
|
|
|
region server
|
2007-07-26 17:30:55 -04:00
|
|
|
49. HADOOP-1646 RegionServer OOME's under sustained, substantial loading by
|
|
|
|
|
10 concurrent clients
|
2007-07-26 17:58:22 -04:00
|
|
|
50. HADOOP-1468 Add HBase batch update to reduce RPC overhead (restrict batches
|
|
|
|
|
to a single row at a time)
|
HADOOP-1709 Make HRegionInterface more like that of HTable
HADOOP-1725 Client find of table regions should not include offlined, split parents
Changes:
New class MapWritable replaces KeyedData and KeyedDataArrayWritable
HBaseAdmin, HConnectionManager, HMaster, HRegionInterface,
HRegionServer, HTable, TestScanner2:
- getRow returns MapWritable instead of array of KeyedData
- next returns MapWritable instead of array of KeyedData
GroupingTableMap, IdentityTableMap, IdentityTableReduce,
TableInputFormat, TableMap, TableOutputCollector, TableOutputFormat,
TestTableMapReduce:
- use MapWritable instead of KeyedData and KeyedDataArrayWritable
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@566878 13f79535-47bb-0310-9956-ffa450edef68
2007-08-16 18:51:03 -04:00
|
|
|
51. HADOOP-1528 HClient for multiple tables (phase 1) (James Kennedy & JimK)
|
2007-08-02 04:15:26 -04:00
|
|
|
52. HADOOP-1528 HClient for multiple tables (phase 2) all HBase client side code
|
|
|
|
|
(except TestHClient and HBaseShell) have been converted to use the new client
|
|
|
|
|
side objects (HTable/HBaseAdmin/HConnection) instead of HClient.
|
2007-08-02 20:02:19 -04:00
|
|
|
53. HADOOP-1528 HClient for multiple tables - expose close table function
|
2007-08-03 18:39:43 -04:00
|
|
|
54. HADOOP-1466 Clean up warnings, visibility and javadoc issues in HBase.
|
HADOOP-1662 Make region splits faster
Splits are now near-instantaneous. On split, daughter splits create
'references' to store files up in the parent region using new 'HalfMapFile'
class to proxy accesses against the top-half or bottom-half of
backing MapFile. Parent region is deleted after all references in daughter
regions have been let go.
Below includes other cleanups and at least one bug fix for fails adding
>32k records and improvements to make it more likely TestRegionServerAbort
will complete..
A src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestHStoreFile.java
Added. Tests new Reference HStoreFiles. Test new HalfMapFileReader inner
class of HStoreFile. Test that we do the right thing when HStoreFiles
are smaller than a MapFile index range (i.e. there is not 'MidKey').
Test we do right thing when key is outside of a HalfMapFile.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/HBaseTestCase.java
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestGet.java
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestScanner.java
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestTimestamp.java
getHRegionDir moved from HStoreFile to HRegion.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestBatchUpdate.java
Let out exception rather than catch and call 'fail'.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/MiniHBaseCluster.java
Refactored so can start and stop a minihbasecluster w/o having to
subclass this TestCase. Refactored methods in this class to use the
newly added methods listed below.
(MasterThread, RegionServerThread, startMaster, startRegionServers
shutdown): Added.
Added logging of abort, close and wait. Also on abort/close
was doing a remove that made it so subsequent wait had nothing to
wait on.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestSplit.java
Added tests that assert all works properly at region level on
multiple levels of splits and then do same on a cluster.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestHRegion.java
Removed catch and 'fail()'.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStoreFile.java
Javadoc to explain how split now works. Have constructors flow
into each other rather than replicate setup per instance. Moved
in here operations such as delete, rename, and length of store files
(No need of clients to remember to delete map and info files).
(REF_NAME_PARSER, Reference, HalfMapFile, isReference,
writeReferenceFiles, writeSplitInfo, readSplitInfo,
createOrFail, getReader, getWriter, toString): Added.
(getMapDir, getMapFilePath, getInfoDir, getInfoFilePath): Added
a bunch of overrides for reference handling.
(loadHStoreFiles): Amended to load references off disk.
(splitStoreFiles): Redone to instead write references into
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStore.java
Rename maps as readers and mapFiles as storefiles.
Moved BloomFilterReader and Writer into HStoreFile. Removed
getMapFileReader and getMapFileWriter (They are in HStoreFile now).
(getReaders): Added.
(HStoreSize): Added. Data Structure to hold aggregated size
of all HStoreFiles in HStore, the largest, its midkey, and
if the HStore is splitable (May not be if references).
Previous we only did largest file; less accurate.
(getLargestFileSize): Renamed size and redone to aggregate
sizes, etc.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HColumnDescriptor.java
Have constructors waterfall down through each other rather than
repeat initializations.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMerge.java
Use new HStoreSize structure.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegionServer.java
Added delayed remove of HRegion (Now done in HMaster as part of
meta scan). Change LOG.error and LOG.warn so they throw stack trace
instead of just the Exception.toString as message.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HConstants.java
(COLUMN_FAMILY_STR): Added.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HLog.java
Added why to log of splitting.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HLogEdit.java
Short is not big enough to hold edits tha could contain a sizable
web page.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HTable.java
(getTableName): Added.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java
Added constructor to BaseScanner that takes name of table we're
scanning (ROOT or META usually). Added to scanOneRegion handling
of split regions. Collect splits to check while scanning and
then outside of the scanning, so we can modify the META table
is needed, do the checks of daughter regions and update on
change of state. Made LOG.warn and LOG.error print stack trace.
(isSplitParent, cleanupSplits, hasReferences): Added.
Added toString to each of the PendingOperation implementations.
In the ShutdownPendingOperation scan of meta data, removed
check of startcode (if the server name is that of the dead
server, it needs reassigning even if start code is good).
Also, if server name is null -- possible if we are missing
edits off end of log -- then the region should be reassigned
just in case its from the dead server. Also, if reassigning,
clear from pendingRegions. Server may have died after sending
region is up but before the server confirms receipt in the
meta scan. Added mare detail to each log. In OpenPendingOperation
we were trying to clear pendingRegion in wrong place -- it was
never executed (regions were always pending).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegionInfo.java
Add split boolean. Output offline and split status in toString.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMemcache.java
Comments.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegion.java
Moved getRegionDir here from HStoreFile.
(COL_SPLITA, COL_SPLITB): Added.
(closeAndSplit): Refactored to use new fast split method.
StringUtils.formatTimeDiff(System.currentTimeMillis(), startTime));
(splitStoreFile): Moved into HStoreFile.
(getSplitRegionDir, getSplitsDir, toString): Added.
(needsSplit): Refactored to exploit new HStoreSize structure.
Also manages notion of 'unsplitable' region.
(largestHStore): Refactored.
(removeSplitFromMETA, writeSplitToMETA, getSplit, hasReference): Added.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/util/Keying.java
(intToBytes, getBytes): Added.
A src/contrib/hbase/src/java/org/apache/hadoop/hbase/util/Writables.java
Utility reading and writing Writables.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@564012 13f79535-47bb-0310-9956-ffa450edef68
2007-08-08 16:30:13 -04:00
|
|
|
55. HADOOP-1662 Make region splits faster
|
2007-08-10 18:11:05 -04:00
|
|
|
56. HADOOP-1678 On region split, master should designate which host should
|
|
|
|
|
serve daughter splits. Phase 1: Master balances load for new regions and
|
|
|
|
|
when a region server fails.
|
2007-08-13 23:37:01 -04:00
|
|
|
57. HADOOP-1678 On region split, master should designate which host should
|
|
|
|
|
serve daughter splits. Phase 2: Master assigns children of split region
|
|
|
|
|
instead of HRegionServer serving both children.
|
2007-08-14 21:18:47 -04:00
|
|
|
58. HADOOP-1710 All updates should be batch updates
|
2007-08-15 14:20:53 -04:00
|
|
|
59. HADOOP-1711 HTable API should use interfaces instead of concrete classes as
|
|
|
|
|
method parameters and return values
|
HADOOP-1644 [hbase] Compactions should not block updates
Disentangles flushes and compactions; flushes can proceed while a
compaction is happening. Also, don't compact unless we hit
compaction threshold: i.e. don't automatically compact on HRegion
startup so regions can come online the faster.
M src/contrib/hbase/conf/hbase-default.xml
(hbase.hregion.compactionThreashold): Moved to be a hstore property
as part of encapsulating compaction decision inside hstore.
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/HBaseTestCase.java
Refactored. Moved here generalized content loading code that can
be shared by tests. Add to setup and teardown the setup and removal
of local test dir (if it exists).
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestCompare.java
Added test of HStoreKey compare (It works other than one would at
first expect).
M src/contrib/hbase/src/test/org/apache/hadoop/hbase/TestSplit.java
Bulk of content loading code has been moved up into the parent class.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HConnectionManager.java
(tableExists): Restore to a check of if the asked-for table is in list of
tables. As it was, a check for tableExists would just wait on all timeouts
and retries to expire and then report table does not exist.. Fixed up
debug message listing regions of a table. Added protection against meta
table not having a COL_REGINFO (Seen in cluster testing -- probably a bug
in row removal).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStoreFile.java
Loading store files, even if it was noticed that there was no corresponding
map file, was still counting file as valid. Also fix merger -- was
constructing MapFile.Reader directly rather than asking HStoreFile for
the reader (HStoreFile knows how to do MapFile references)
(rename): Added check that move succeeded and logging. In cluster-testing,
the hdfs move of compacted file into place has failed on occasion (Need
more info).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStore.java
Encapsulate ruling on whether a compaction should take place inside HStore.
Added reading of the compactionThreshold her. Compaction threshold is
currently just number of store files. Later may include other factors such
as count of reference files. Cleaned up debug messages around
reconstruction log. Removed compaction if size > 1 from constructor. Let
compaction happen after we've been deployed (Compactions that happen while
we are online can continue to take updates. Compaction in the constructor
puts off our being able to take in updates).
(close): Changed so it now returns set of store files. This used to be done
by calls to flush. Since flush and compaction have been disentangled, a
compaction can come in after flush and the list of files could be off.
Having it done by close, can be sure list of files is complete.
(flushCache): No longer returns set of store files. Added 'merging compaction'
where we pick an arbitrary store file from disk and merge into it the content
of memcache (Needs work).
(getAllMapFiles): Renamed getAllStoreFiles.
(needsCompaction): Added.
(compactHelper): Added passing of maximum sequence number if already
calculated. If compacting one file only, we used skip without rewriting
the info file. Fixed.
Refactored. Moved guts to new compact(outFile, listOfStores) method.
(compact, CompactionReader): Added overrides and interface to support
'merging compaction' that takes files and memcache. In compaction,
if we failed the move of the compacted file, all data had already been
deleted. Changing, so deletion happens after confirmed move of
compacted file.
(getFull): Fixed bug where NPE when read of maps came back null.
Revealed by our NOT compacting stores on startup. Meant could be two
backing stores one of which had no data regards queried key.
(getNMaps): Renamed countOfStoreFiles.
(toString): Added.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HStoreKey.java
Added comment on 'odd'-looking comparison.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegionServer.java
Javadoc edit.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HLogEdit.java
Only return first 128 bytes of value when toStringing (On cluster,
was returning complete web pages in log).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMaster.java
Removed confusing debug message (made sense once -- but not now).
Test rootRegionLocation for null before using it (can be null).
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HMemcache.java
Added comment that delete behavior needs study.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/HRegion.java
Fixed merge so it doesn't do the incremental based off files
returned by flush. Instead all is done in the one go after
region closes (using files returned by close).
Moved duplicated code to new filesByFamily method.
(WriteState): Removed writesOngoing in favor of compacting and
flushing flags.
(flushCache): No longer returns list of files.
M src/contrib/hbase/src/java/org/apache/hadoop/hbase/util/Writables.java
Fix javadoc.
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@566459 13f79535-47bb-0310-9956-ffa450edef68
2007-08-15 21:07:51 -04:00
|
|
|
60. HADOOP-1644 Compactions should not block updates
|
2007-08-15 21:56:29 -04:00
|
|
|
60. HADOOP-1672 HBase Shell should use new client classes
|
|
|
|
|
(Edward Yoon via Stack).
|
HADOOP-1709 Make HRegionInterface more like that of HTable
HADOOP-1725 Client find of table regions should not include offlined, split parents
Changes:
New class MapWritable replaces KeyedData and KeyedDataArrayWritable
HBaseAdmin, HConnectionManager, HMaster, HRegionInterface,
HRegionServer, HTable, TestScanner2:
- getRow returns MapWritable instead of array of KeyedData
- next returns MapWritable instead of array of KeyedData
GroupingTableMap, IdentityTableMap, IdentityTableReduce,
TableInputFormat, TableMap, TableOutputCollector, TableOutputFormat,
TestTableMapReduce:
- use MapWritable instead of KeyedData and KeyedDataArrayWritable
git-svn-id: https://svn.apache.org/repos/asf/lucene/hadoop/trunk/src/contrib/hbase@566878 13f79535-47bb-0310-9956-ffa450edef68
2007-08-16 18:51:03 -04:00
|
|
|
61. HADOOP-1709 Make HRegionInterface more like that of HTable
|
|
|
|
|
HADOOP-1725 Client find of table regions should not include offlined, split parents
|
|
|
|
|
|
