Apache
/
hbase
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity
12,816 Commits 95 Branches 866 Tags 550 MiB
Commit Graph

5248 Commits

Author SHA1 Message Date
zhangduo b35cf8f93a HBASE-16225 Refactor ScanQueryMatcher 2016-08-02 14:01:28 +08:00
Andrew Purtell 78361243c8 HBASE-16296 Reverse scan performance degrades when using filter lists (Ted Yu) 2016-08-01 17:41:10 -07:00
Enis Soztutar aa0235f98c HBASE-16288 HFile intermediate block level indexes might recurse forever creating multi TB files 2016-08-01 11:08:56 -07:00
chenheng 7e6f562754 HBASE-16234 Expect and handle nulls when assigning replicas(Yi Liang) 2016-08-01 11:21:13 +08:00
Joseph Hwang 3729320099 HBASE-16209 Removed unnecessary method and method calls from AssignmentManager and AssignmentManager UI (addendum) 
Signed-off-by: Matteo Bertozzi <matteo.bertozzi@cloudera.com>
 2016-07-29 16:05:00 -07:00
Joseph Hwang fe44383619 HBASE-16209 provide an ExponentialBackOffPolicy sleep between failed region open requests 
Signed-off-by: Elliott Clark <eclark@apache.org>
 2016-07-29 09:47:00 -07:00
Yu Sun 2b1cbeb28d HBASE-16300 LruBlockCache.CACHE_FIXED_OVERHEAD should calculate LruBlockCache size correctly 
Signed-off-by: Yu Li <liyu@apache.org>
 2016-07-29 14:29:01 +08:00
Matteo Bertozzi 66fb697f81 HBASE-16275 Change ServerManager#onlineServers from ConcurrentHashMap to ConcurrentSkipListMap (Huaxiang Sun) 2016-07-28 11:13:58 -07:00
Matteo Bertozzi 64804a0497 HBASE-16293 TestSnapshotFromMaster#testSnapshotHFileArchiving flakey (Huaxiang Sun) 2016-07-28 10:42:30 -07:00
anoopsjohn 00c2a5da7b HBASE-16286 When TagRewriteCell are not copied to MSLAB, deep clone it while adding to Memstore. 2016-07-27 22:29:26 +05:30
zhangduo bcf409e11f HBASE-16280 Use hash based map in SequenceIdAccounting 2016-07-26 11:31:32 +08:00
Phil Yang 0da8119fbc HBASE-16281 TestMasterReplication is flaky 
Signed-off-by: zhangduo <zhangduo@apache.org>
 2016-07-26 10:54:46 +08:00
Reid Chan abfd584fe6 HBASE-14743 Add metrics around HeapMemoryManager. (Reid Chan) 
Change-Id: I7305f7b7034b216930b5fb5c57de9ba5eabf96d8

Signed-off-by: Apekshit Sharma <appy@apache.org>
 2016-07-25 14:32:38 -07:00
Apekshit Sharma eff38ccf8c Revert HBASE-14743 because of wrong attribution. Since I added commit message to the raw patch, it's making me as author instead of Reid. I should have used --author flag to set Reid as author. 
This reverts commit 064271da16.
 2016-07-25 14:32:38 -07:00
Apekshit Sharma 064271da16 HBASE-14743 Add metrics around HeapMemoryManager. (Reid Chan) 
Change-Id: I60b2435355b3e605e7d91cbf5aca5d2988f26f33
 2016-07-25 13:45:50 -07:00
anoopsjohn 2df0ef549a HBASE-16205 When Cells are not copied to MSLAB, deep clone it while adding to Memstore. 2016-07-25 22:58:32 +05:30
Phil Yang 6dbce2a8cb HBASE-16266 Do not throw ScannerTimeoutException when catch UnknownScannerException 
Signed-off-by: zhangduo <zhangduo@apache.org>
 2016-07-25 22:26:19 +08:00
Stephen Yuan Jiang bdd7782f05 HBASE-16008 A robust way deal with early termination of HBCK (Stephen Yuan Jiang) 2016-07-23 14:47:11 -07:00
Stephen Yuan Jiang cd0b85e0a8 Revert HBASE-15406: Split / merge switch left disabled after early termination of hbck (HBASE-16008 will address this issue) 2016-07-22 20:09:03 -07:00
Joseph Hwang 744248c131 HBASE-16096 Properly remove the replication queue and peer znodes after calling ReplicationSourceManager.removePeer(). 
Signed-off-by: Elliott Clark <eclark@apache.org>
 2016-07-22 18:11:41 -07:00
Joseph Hwang 03fe257a64 HBASE-16221 Have ThriftScanner update the ConnectionCache's last used time overtime getScannerRow() to keep the connection alive for long lived scanners 
Signed-off-by: Elliott Clark <eclark@apache.org>
 2016-07-22 17:58:39 -07:00
Matteo Bertozzi cc766df28b HBASE-16272 Overflow in ServerName's compareTo method (Huaxiang Sun) 2016-07-22 15:28:04 -07:00
Jerry He 519f87fc2b HBASE-15291 Revert due to race condition resulting in closed FileSystem error 2016-07-21 18:51:47 -07:00
Gary Helmling 65834a1ced HBASE-16217 Pass through the calling user in ObserverContext 2016-07-21 16:45:09 -07:00
stack 9d740f7b8b HBASE-16263 Move all to do w/ protobuf -- *.proto files and generated classes -- under hbase-protocol 
Signed-off-by: stack <stack@apache.org>
 2016-07-21 10:02:05 -07:00
Jerry He ce657d5911 HBASE-13701 Consolidate SecureBulkLoadEndpoint into HBase core as default for bulk load -- addendum 2016-07-20 13:28:01 -07:00
Jerry He 70f330dc84 Consolidate SecureBulkLoadEndpoint into HBase core as default for bulk load 2016-07-19 19:53:10 -07:00
Enis Soztutar 5051ab4e70 HBASE-16244 LocalHBaseCluster start timeout should be configurable (Siddharth Wagle) 2016-07-19 14:12:47 -07:00
Stephen Yuan Jiang 638ca24f01 HBASE-14552 (addendum) Procedure V2: Reimplement DispatchMergingRegionHandler (add nonce support) (Stephen Yuan Jiang) 2016-07-19 08:49:23 -07:00
zhangduo 515c499f95 HBASE-16110 AsyncFS WAL doesn't work with Hadoop 2.8+ 
Signed-off-by: Sean Busbey <busbey@apache.org>
 2016-07-18 06:54:20 -05:00
Phil Yang a1cc2c4bfe HBASE-16144 Replication queue's lock will live forever if RS acquiring the lock has died prematurely 
(cherry picked from commit a8dfd8c436)
 2016-07-18 06:48:17 -05:00
Sean Busbey a0aa017ae9 Revert "Replication queue's lock will live forever if RS acquiring the lock has died prematurely" 
This reverts commit a8dfd8c436.
 2016-07-18 06:47:56 -05:00
chenheng cf0f4f72d9 HBASE-16076 Cannot configure split policy in HBase shell 2016-07-18 10:28:05 +08:00
Phil Yang a8dfd8c436 Replication queue's lock will live forever if RS acquiring the lock has died prematurely 2016-07-18 09:32:36 +08:00
tedyu 9bc7ecfb9d HBASE-16235 Addendum uses hfile count of 20 (ChiaPing) 2016-07-16 16:07:11 -07:00
Jerry He 60847a2d76 HBASE-3727 MultiHFileOutputFormat (yi liang) 2016-07-15 21:07:12 -07:00
tedyu 66c68f8ea3 HBASE-16235 TestSnapshotFromMaster#testSnapshotHFileArchiving will fail if there are too many hfiles (ChiaPing Tsai) 2016-07-15 15:07:14 -07:00
Stephen Yuan Jiang 3978aa3af0 HBASE-13823 Procedure V2: remove unnecessaery operaions on AssignmentManager#recoverTableInDisablingState() and recoverTableInEnablingState() (Stephen Yuan Jiang) 2016-07-15 12:26:31 -07:00
Stephen Yuan Jiang f04eeecffc HBASE-14552 Procedure V2: Reimplement DispatchMergingRegionHandler (Stephen Yuan Jiang) 2016-07-15 07:06:53 -07:00
Stephen Yuan Jiang 28d8609e5d HBASE-16233 Procedure V2: Support acquire/release shared table lock concurrently (Stephen Yuan Jiang) 2016-07-14 21:32:59 -07:00
zhangduo 243fe287b9 HBASE-16219 addendum move MasterMetaBootstrap to the right place 2016-07-15 09:58:17 +08:00
Matteo Bertozzi a55af38689 HBASE-16219 Move meta bootstrap out of HMaster 2016-07-14 06:14:53 -07:00
Enis Soztutar a07892558b HBASE-16095 Add priority to TableDescriptor and priority region open thread pool 2016-07-13 18:19:59 -07:00
Yu Li 3b3c3dc02d HBASE-16195 Should not add chunk into chunkQueue if not using chunk pool in HeapMemStoreLAB 2016-07-13 09:33:24 +08:00
Andrew Purtell 911706a873 HBASE-16220 Demote log level for "HRegionFileSystem - No StoreFiles for" messages to TRACE 2016-07-12 15:44:06 -07:00
Joseph Hwang f292048ffd HBASE-16208 Check that the row exists before attempting to remove a queue from TableBasedReplicationQueuesImpl 
Signed-off-by: Elliott Clark <eclark@apache.org>
 2016-07-12 14:50:07 -07:00
Joseph Hwang 90ba723dc6 HBASE-16214 Add in a UI description for the Replication Table 
Signed-off-by: Elliott Clark <eclark@apache.org>
 2016-07-12 12:21:55 -07:00
Matteo Bertozzi 2650711e94 HBASE-16207 can't restore snapshot without "Admin" permission 2016-07-12 05:50:06 -07:00
Joseph Hwang ccf293d7fb HBASE-16081 Wait for Replication Tasks to complete before killing the ThreadPoolExecutor inside of HBaseInterClusterReplicationEndpoint 
Signed-off-by: Mikhail Antonov <antonov@apache.org>
 2016-07-11 17:34:12 -07:00
Gary Helmling 3b5fbf8d73 HBASE-16160 Support RPC encryption with direct ByteBuffers (Colin Ma via garyh) 2016-07-11 10:33:22 -07:00
Yu Li 356c130702 HBASE-16194 Should count in MSLAB chunk allocation into heap size change when adding duplicate cells, addendum 2016-07-11 22:44:32 +08:00
tedyu 0875c35781 HBASE-16185 TestReplicationSmallTests fails in master branch (Phil Yang) 2016-07-10 23:28:32 -07:00
Yu Li 9cf012cd00 HBASE-16194 Should count in MSLAB chunk allocation into heap size change when adding duplicate cells 2016-07-10 13:45:17 +08:00
stack 496fd9837a HBASE-16176 Bug fixes/improvements on HBASE-15650 Remove 
TimeRangeTracker as point of contention when many threads reading a StoreFile
 Fixes HBASE-16074 ITBLL fails, reports lost big or tiny families broken
 scanning because of a side effect of a clean up in  HBASE-15650 to make
 TimeRange construction consistent exposed a latent issue in
 TimeRange#compare. See HBASE-16074 for more detail.

Also change HFile Writer constructor so we pass in the TimeRangeTracker, if one,
on construction rather than set later (the flag and reference were not volatile
so could have made for issues in concurrent case). And make sure the construction
of a TimeRange from a TimeRangeTracer on open of an HFile Reader never makes a
bad minimum value, one that would preclude us reading any values from a file
(set min to 0)

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/TimeRange.java
 Call through to next constructor (if minStamp was 0, we'd skip setting
 allTime=true). Add asserts that timestamps are not < 0 cos it messes
 us up if they are (we already were checking for < 0 on construction but
 assert passed in timestamps are not < 0).

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
 Add constructor override that takes a TimeRangeTracker (set when flushing
 but not when compacting)

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Store.java
 Add override creating an HFile in tmp that takes a TimeRangeTracker

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
 Add override for HFile Writer that takes a TimeRangeTracker Take it on
 construction instead of having it passed by a setter later (flags and
 reference set by the setter were not volatile... could have been prob
 in concurrent case)

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java
 Log WARN if bad initial TimeRange value (and then 'fix' it)

M hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestTimeRangeTracker.java
 A few tests to prove serialization works as expected and that we'll get a bad min if not constructed properly.

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
 Handle OLDEST_TIMESTAMP explictly. Don't expect TimeRange to do it.

M hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestQueryMatcher.java
 Refactor from junit3 to junit4 and add test for this weird case.
 2016-07-08 18:20:31 -07:00
Yu Li 3c39cbd92c HBASE-16201 NPE in RpcServer causing intermittent UT failure of TestMasterReplication#testHFileCyclicReplication 2016-07-09 02:02:00 +08:00
Yu Li 1578a045b6 Revert "NPE in RpcServer causing intermittent UT failure of TestMasterReplication#testHFileCyclicReplication" 
This reverts commit a33097e067.
 2016-07-09 02:01:43 +08:00
stack 5e0d97e0e9 HBASE-16171 Fix the potential problems in TestHCM.testConnectionCloseAllowsInterrupt (Colin Ma) 2016-07-08 10:18:36 -07:00
Yu Li a33097e067 NPE in RpcServer causing intermittent UT failure of TestMasterReplication#testHFileCyclicReplication 2016-07-09 01:05:10 +08:00
anoopsjohn 581d2b7de5 HBASE-16162 Compacting Memstore : unnecessary push of active segments to pipeline. 2016-07-06 18:54:35 +05:30
Elliott Clark ae92668dd6 HBASE-16087 Replication shouldn't start on a master if if only hosts system tables 2016-07-05 13:50:37 -07:00
tedyu 20a99b4c06 HBASE-16144 Revert - test failure in TestMultiSlaveReplication#testMultiSlaveReplication seems to be related 2016-07-05 13:19:33 -07:00
Andrew Purtell 318751cfd6 HBASE-16091 Canary takes lot more time when there are delete markers in the table (Vishal Khandelwal) 2016-07-05 11:13:48 -07:00
tedyu d016338e45 HBASE-16157 The incorrect block cache count and size are caused by removing duplicate block key in the LruBlockCache (ChiaPing Tsai) 2016-07-05 10:21:27 -07:00
tedyu b5cdb38dc7 HBASE-16144 Replication queue's lock will live forever if RS acquiring the lock has died prematurely (Phil Yang) 2016-07-05 07:16:12 -07:00
Yu Li 7f44dfd85f HBASE-16132 Scan does not return all the result when regionserver is busy (binlijin) 2016-07-04 21:45:04 +08:00
zhangduo 6944a17ad4 HBASE-16135 PeerClusterZnode under rs of removed peer may never be deleted 2016-07-04 19:26:27 +08:00
anoopsjohn d22c23c396 HBASE-16163 Compacting Memstore : CompactionPipeline#addFirst can use LinkedList#addFirst. 2016-07-03 21:35:29 +05:30
anoopsjohn 97f2294bfd HBASE-16164 Missing close of new compacted segments in few occasions which might leak MSLAB chunks from pool. 2016-07-02 11:42:08 +05:30
chenheng 561eb82968 HBASE-15844 We should respect hfile.block.index.cacheonwrite when write intermediate index Block 2016-07-02 09:04:15 +08:00
tedyu fe75edb556 HBASE-16157 Revert - ChiaPing has new fix 2016-07-01 13:00:05 -07:00
tedyu d92a99da0e HBASE-16125 RegionMover uses hardcoded, Unix-style tmp folder - breaks Windows (Vladimir Rodionov) 2016-07-01 10:02:25 -07:00
tedyu 5a7c9939cb HBASE-16157 The incorrect block cache count and size are caused by removing duplicate block key in the LruBlockCache (ChiaPing Tsai) 2016-07-01 09:43:39 -07:00
tedyu cc73c03118 HBASE-16108 RowCounter should support multiple key ranges (Konstantin Ryakhovskiy) 2016-07-01 03:41:17 -07:00
anoopsjohn 139f0ed53c HBASE-16155 Compacting Memstore : Few log improvements. 2016-07-01 10:14:07 +05:30
stack d1d8cc71c9 HBASE-16150 Remove ConcurrentIndex (Hiroshi Ikeda) 2016-06-30 15:03:32 -07:00
Ramkrishna e2566eb0ee HBASE-16153 Correct the config name 
'hbase.memestore.inmemoryflush.threshold.factor' (Ram)
 2016-06-30 17:44:08 +05:30
Jonathan M Hsieh e1d130946b HBASE-16117 Fix Connection leak in mapred.TableOutputFormat 2016-06-29 17:18:43 -07:00
tedyu 42106b89b1 HBASE-16114 Get regionLocation of required regions only for MR jobs (Thiruvel Thirumoolan) 2016-06-29 16:34:44 -07:00
Ramkrishna 9b1ecb31f0 HBASE-16143 Change MemstoreScanner constructor to accept 
List<KeyValueScanner> (Ram)
 2016-06-29 23:10:55 +05:30
Jonathan M Hsieh 727394c2e9 HBASE-16140 bump owasp.esapi from 2.1.0 to 2.1.0.1 2016-06-29 08:51:00 -07:00
tedyu 6fb0dd07a8 HBASE-16139 Use CellUtil instead of KeyValueUtil in Import (NIDHI GAMBHIR) 2016-06-29 02:45:00 -07:00
Jingcheng Du 432868b038 HBASE-15976 RegionServerMetricsWrapperRunnable will be failure when disable blockcache. - Addendum 2016-06-29 16:31:23 +08:00
tedyu 394e7224a9 HBASE-16122 PerformanceEvaluation should provide user friendly hint when client threads argument is missing (Konstantin Ryakhovskiy) 2016-06-28 17:42:51 -07:00
Matteo Bertozzi 8bc4d4131c HBASE-16121 Require only MasterServices to the ServerManager constructor 2016-06-28 07:42:40 -07:00
Jingcheng Du e768c4afac HBASE-15976 RegionServerMetricsWrapperRunnable will be failure when disable blockcache. 2016-06-28 13:45:41 +08:00
tedyu 13cf4fb3fc HBASE-16052 Improve HBaseFsck Scalability (Ben Lau) 2016-06-27 13:24:35 -07:00
tedyu 424b789e03 HBASE-14007 Writing to table through MR should fail upfront if table does not exist/is disabled (Thiruvel Thirumoolan) 2016-06-27 06:41:00 -07:00
tedyu ba9286911f HBASE-16109 Add Interface audience annotation to a few classes 2016-06-27 06:29:04 -07:00
Elliott Clark fc4b8aa89d HBASE-16089 Add on FastPath for CoDel 2016-06-25 12:57:15 -07:00
Matteo Bertozzi e8599a2921 HBASE-16103 Procedure v2 - TestCloneSnapshotProcedure relies on execution order 2016-06-24 09:09:59 -07:00
Ramkrishna 99dc300d37 HBASE-16055 PutSortReducer loses any Visibility/acl attribute set on the 
Puts (Ram)
 2016-06-24 15:09:53 +05:30
Jingcheng Du 518faa735b HBASE-15353 Add metric for number of CallQueueTooBigException's 2016-06-24 13:00:06 +08:00
chenheng 68c1b34dbc HBASE-16040 Remove configuration "hbase.replication" 2016-06-24 10:41:35 +08:00
Gary Helmling f4cec2e202 HBASE-16085 Add a metric for failed compactions 2016-06-23 15:38:58 -07:00
tedyu 9cd097cfac HBASE-16090 ResultScanner is not closed in SyncTable#finishRemainingHashRanges() 2016-06-23 07:28:35 -07:00
Matteo Bertozzi d66316fd80 HBASE-16068 Procedure v2 - use consts for conf properties in tests 2016-06-22 22:48:07 -07:00
Ramkrishna c1a559a666 HBASE-16070 Mapreduce Serialization classes do not have Interface audience 
(Ram)
 2016-06-23 10:22:24 +05:30
tedyu d297ecdf3c HBASE-16083 Addendum fixes config property names in tests (Joseph Hwang) 2016-06-22 17:31:57 -07:00
tedyu 0ea4d2f461 HBASE-16083 revert Addendum - Joseph discovered the mistake first 2016-06-22 17:31:24 -07:00
tedyu da09fd0781 HBASE-16083 Addendum fixes config property names in tests 2016-06-22 16:35:37 -07:00
Joseph Hwang f3b3a7c9b5 HBASE-16080 Fix flakey test TestTableBasedReplicationSourceManager.cleanupFailoverQueues(). 
Instead of running the primary test in a separate thread and hoping it finishes in time, just run the test in the primary thread.

Signed-off-by: Elliott Clark <eclark@apache.org>
 2016-06-22 12:26:35 -07:00
tedyu 6fc71c5d31 HBASE-16049 TestRowProcessorEndpoint is failing on Apache Builds (Guanghao Zhang) 2016-06-22 12:24:07 -07:00
tedyu 2846113b59 HBASE-16012 Major compaction can't work due to obsolete scanner read point in RegionServer (Guanghao Zhang) 2016-06-22 02:08:45 -07:00
Joseph Hwang 152594560e HBASE-16036 Made Replication Table creation non-blocking. 
All ReplicationTableBase method's that need to access the Replication Table will block until it is created though.
Also refactored ReplicationSourceManager so that abandoned queue adoption is run in the background too so that it does not block HRegionServer initialization.

Signed-off-by: Elliott Clark <eclark@apache.org>
 2016-06-21 13:05:50 -07:00
tedyu b006e41a37 HBASE-16051 TestScannerHeartbeatMessages fails on some machines (Phil Yang) 2016-06-21 07:32:30 -07:00
Yu Li 471f942ec8 HBASE-16032 Possible memory leak in StoreScanner 2016-06-21 20:06:33 +08:00
tedyu 3de0c63105 HBASE-16035 Nested AutoCloseables might not all get closed (Sean Mackrory) 2016-06-20 16:25:50 -07:00
Matteo Bertozzi 313a3d23a8 HBASE-14644 Region in transition metric is broken -- addendum (Huaxiang Sun) 2016-06-20 10:46:25 -07:00
tedyu be56b5ae71 HBASE-16059 Region normalizer fails to trigger merge action where one of the regions is empty 2016-06-20 10:40:50 -07:00
tedyu 095a82584e HBASE-16062 Improper error handling in WAL Reader/Writer creation (Vladimir Rodionov) 2016-06-20 07:34:26 -07:00
Ramkrishna 71c8cd5b1f HBASE-16042 Add support in PE tool for InMemory Compaction (Ram) 2016-06-20 16:29:24 +05:30
stack 01684c3417 HBASE-16023 Fastpath for the FIFO rpcscheduler AMENDMENT 2016-06-20 08:34:32 +01:00
Yu Li 4421fdf981 HBASE-16066 Resolve RpC_REPEATED_CONDITIONAL_TEST findbugs warnings in HMaster, addendum to HBASE-15467 2016-06-19 17:25:26 +08:00
Enis Soztutar 6717f0e388 HBASE-16058 TestHRegion fails on 1.4 builds 2016-06-17 18:41:51 -07:00
Enis Soztutar bdb0cc8808 HBASE-15467 Remove 1.x/2.0 TableDescriptor incompatibility 2016-06-17 17:25:04 -07:00
Enis Soztutar 81a9c1ac31 HBASE-16053 Master code is not setting the table in ENABLING state in create table 2016-06-17 13:29:33 -07:00
Joseph Hwang 61ff6ced5b HBASE-16018 Refactored the ReplicationPeers interface to clear up what some methods do and move away from a ZooKeeper-specific implementation. 
Also added some documentation for undocumented methods.

Signed-off-by: Elliott Clark <eclark@apache.org>
 2016-06-17 13:04:21 -07:00
tedyu 5147fb12a5 HBASE-16048 Tag InternalScanner with LimitedPrivate(HBaseInterfaceAudience.COPROC) 2016-06-17 07:27:52 -07:00
Ashu Pachauri 3abd52bdc6 HBASE-15429 Add split policy for busy regions 
Signed-off-by: Elliott Clark <eclark@apache.org>
 2016-06-16 17:36:54 -07:00
tedyu 6d0e0e3721 HBASE-5291 Addendum 2 passes correct path to deleteRecursively 2016-06-16 17:00:57 -07:00
tedyu d8902ba0e6 HBASE-16045 endtime argument for VerifyReplication was incorrectly specified in usage 2016-06-16 15:50:29 -07:00
Mikhail Antonov 62a4a2c029 HBASE-16047 TestFastFail is broken again 2016-06-16 14:09:48 -07:00
Yu Li 6d02f36ac7 HBASE-16033 Add more details in logging of responseTooSlow/TooLarge 2016-06-16 16:42:10 +08:00
Ramkrishna f19f1d9e99 HBASE-15999 NPE in MemstoreCompactor (Ram) 2016-06-16 10:06:40 +05:30
tedyu 1bad166f67 HBASE-5291 Addendum removes duplicate spnego (Josh Elser) 2016-06-15 14:59:17 -07:00
Enis Soztutar bff35d6387 HBASE-15950 Fix memstore size estimates to be more tighter 2016-06-15 13:34:39 -07:00
Mikhail Antonov 5f9d1a71c8 HBASE-16026 Master UI should display status of additional ZK switches 2016-06-15 12:59:24 -07:00
Joseph Hwang 2093aadec1 HBASE-15974 Create a ReplicationQueuesClientHBaseImpl 
Building on HBase-15958.
Provided a ReplicationQueuesClientHBaseImpl that relies on the HBase Replication Table to track WAL queues.
Refactored out a large section of ReplicationQueuesHBaseImpl into a ReplicationTableClient class that handles Replication Table operations.

Signed-off-by: Elliott Clark <eclark@apache.org>
 2016-06-15 10:43:14 -07:00
tedyu ae5fe1e616 HBASE-5291 Add Kerberos HTTP SPNEGO authentication support to HBase web consoles (Josh Elser) 2016-06-15 09:26:44 -07:00
tedyu 6c60bc9f6c HBASE-16017 HBase TableOutputFormat has connection leak in getRecordWriter (Zhan Zhang) 2016-06-15 06:32:23 -07:00
chenheng db234bf15d HBASE-16023 Fastpath for the FIFO rpcscheduler. addendum 2016-06-15 16:20:55 +08:00
stack ee86e91e7e HBASE-16023 Fastpath for the FIFO rpcscheduler Adds an executor that does balanced queue and fast path handing off requests directly to waiting handlers if any present. Idea taken from Apace Kudu (incubating). See https://gerrit.cloudera.org/#/c/2938/7/src/kudu/rpc/service_queue.h 
M hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/RpcExecutor.java
 Refactor which makes a Handler type. Put all 'handler' stuff inside this
 new type. Also make it so subclass can provide its own Handler type.

M hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/SimpleRpcScheduler.java
 Name the handler threads for their type so can tell if configs are
 having an effect.

Signed-off-by: stack <stack@apache.org>
 2016-06-14 17:42:01 -07:00
stack fa50d456a8 HBASE-15971 Regression: Random Read/WorkloadC slower in 1.x than 0.98 Restore FIFO as the default in place of 'deadline' that sorted on request priority. The sort costs us throughput. Enable it if you want it rather than have it as default. 
Name threads to include the scheduler type.
 2016-06-14 16:51:24 -07:00
Stephen Yuan Jiang df24004353 HBASE-15746 Remove extra RegionCoprocessor preClose() in RSRpcServices#closeRegion (Stephen Yuan Jiang) 2016-06-14 13:30:41 -07:00
anoopsjohn 17bcf14fea HBASE-15525 OutOfMemory could occur when using BoundedByteBufferPool during RPC bursts. 2016-06-14 19:03:54 +05:30
Stephen Yuan Jiang e486d274c5 HBASE-16016 Fix overflow deadline value in AssignmentManager#waitForAssignment (Stephen Yuan Jiang) 2016-06-13 18:21:41 -07:00
Ramkrishna 56c209c8aa HBASE-15991 CompactingMemstore#InMemoryFlushRunnable should implement 
Comparable/Comparator (Ram)
 2016-06-13 17:42:35 +05:30
Ramkrishna 2d0448fa84 Revert HBASE-15991 commit 2016-06-13 15:05:50 +05:30
Ramkrishna 2738518d43 HBASE-15991 CompactingMemstore#InMemoryFlushRunnable should implement 
Comparable/Comparator (Ram)
 2016-06-13 12:43:52 +05:30
tedyu 6360cc0307 HBASE-16007 Job's Configuration should be passed to TableMapReduceUtil#addDependencyJars() in WALPlayer 2016-06-11 07:05:27 -07:00
Matteo Bertozzi 4c885f4fdb HBASE-14644 Region in transition metric is broken (Huaxiang Sun) 2016-06-10 21:52:24 -07:00
Sean Mackrory 6da6babe4f HBASE-15946. Eliminate possible security concerns in Store File metrics. 
Invoking 'hbase hfile' inside a servlet raises several concerns. This
patch avoids invoking a separate process, and also adds validation that
the file being read is at least inside the HBase root directory.

Signed-off-by: Mikhail Antonov <antonov@apache.org>
 2016-06-09 16:08:19 -07:00
Joseph Hwang babdedc1b0 HBASE-15958 Implement ClaimQueues on top of HBase 
Building on HBase-15883.
Now implementing the claim queues procedure within an HBase table.
Also added UnitTests to test claimQueue.
Peer tracking will still be performed by ZooKeeper though.
Also modified the queueId tracking procedure so we no longer have to perform scans over the Replication Table.
This does make our queue naming schema slightly different from ReplicationQueuesZKImpl though.

Signed-off-by: Elliott Clark <eclark@apache.org>
 2016-06-09 15:05:54 -07:00
tedyu 55a04b7810 HBASE-15990 The priority value of subsequent coprocessors in the Coprocessor.Priority.SYSTEM list are not incremented by one (ChiaPing Tsai) 2016-06-09 10:26:10 -07:00
Ashish Singhi 9012a0b123 HBASE-15952 Bulk load data replication is not working when RS user does not have permission on hfile-refs node 2016-06-09 18:44:29 +05:30
stack 031b745001 HBASE-15994 Allow selection of RpcSchedulers 
Adds logging by the RpcExecutors of their run configs
Adds a FifoRpcSchedulerFactory so you can try Fifo scheduler.
 2016-06-08 20:23:11 -07:00
stack 407aa4d496 Revert "HBASE-15994 Allow selection of RpcSchedulers Adds logging by the RpcExecutors of their run configs Changes the default RpcSchedulerFactory from SimpleRpcSchedulerFactory.class to RpcSchedulerFactoryImpl.class. RpcSchedulerFactoryImpl.class. defaults to using SimpleRpcSchedulerFactory.class and the SimpleRpcScheduler, as has been default up to this, unless you set "hbase.region.server.rpc.scheduler.class" to org.apache.hadoop.hbase.ipc.FifoRpcScheduler" 
This reverts commit 3ac4a57fd2.
 2016-06-08 20:22:50 -07:00
stack 3ac4a57fd2 HBASE-15994 Allow selection of RpcSchedulers Adds logging by the RpcExecutors of their run configs Changes the default RpcSchedulerFactory from SimpleRpcSchedulerFactory.class to RpcSchedulerFactoryImpl.class. RpcSchedulerFactoryImpl.class. defaults to using SimpleRpcSchedulerFactory.class and the SimpleRpcScheduler, as has been default up to this, unless you set "hbase.region.server.rpc.scheduler.class" to org.apache.hadoop.hbase.ipc.FifoRpcScheduler 
Signed-off-by: stack <stack@apache.org>
 2016-06-08 20:20:50 -07:00
Jonathan M Hsieh a7172d5611 HBASE-15959 Fix flaky test TestRegionServerMetrics.testMobMetrics (Huaxiang Sun and Jingcheng Du) 2016-06-08 19:07:13 -07:00
Matteo Bertozzi d9463bcce0 HBASE-15989 Remove hbase.online.schema.update.enable 2016-06-08 13:09:31 -07:00
Matteo Bertozzi d5d9b7d500 HBASE-15107 Procedure v2 - Procedure Queue with Region locks 2016-06-08 12:52:58 -07:00
Enis Soztutar d05a3722c8 HBASE-15600 Add provision for adding mutations to memstore or able to write to same region in batchMutate coprocessor hooks (Rajeshbabu and Enis) 2016-06-07 20:12:51 -07:00
stack 3a95552cfe HBASE-15948 Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers" Adds HADOOP-9955 RPC idle connection closing is extremely inefficient 
Changes how we do accounting of Connections to match how it is done in Hadoop.
Adds a ConnectionManager class. Adds new configurations for this new class.

"hbase.ipc.client.idlethreshold" 4000
"hbase.ipc.client.connection.idle-scan-interval.ms" 10000
"hbase.ipc.client.connection.maxidletime" 10000
"hbase.ipc.client.kill.max", 10
"hbase.ipc.server.handler.queue.size", 100

The new scheme does away with synchronization that purportedly would freeze out
reads while we were cleaning up stale connections (according to HADOOP-9955)

Also adds in new mechanism for accepting Connections by pulling in as many
as we can at a time adding them to a Queue instead of doing one at a time.
Can help when bursty traffic according to HADOOP-9956. Removes a blocking
while Reader is busy parsing a request. Adds configuration
"hbase.ipc.server.read.connection-queue.size" with default of 100 for
queue size.

Signed-off-by: stack <stack@apache.org>
 2016-06-07 16:42:21 -07:00
stack e66ecd7db6 Revert "HBASE-15948 Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers"" 
Revert mistaken commit...
This reverts commit e0b70c00e7.
 2016-06-07 16:41:30 -07:00
stack 6d5a25935e Revert "HBASE-15967 Metric for active ipc Readers and make default fraction of cpu count" 
Revert mistaken commit
This reverts commit 1125215aad.
 2016-06-07 16:41:01 -07:00
stack 1125215aad HBASE-15967 Metric for active ipc Readers and make default fraction of cpu count 
Add new metric hbase.regionserver.ipc.runningReaders
Also make it so Reader count is a factor of processor count
 2016-06-07 13:10:14 -07:00
stack e0b70c00e7 HBASE-15948 Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers" 
Adds HADOOP-9955 RPC idle connection closing is extremely inefficient
Then removes queue added by HADOOP-9956 at Enis suggestion

    Changes how we do accounting of Connections to match how it is done in Hadoop.
    Adds a ConnectionManager class. Adds new configurations for this new class.

    "hbase.ipc.client.idlethreshold" 4000
    "hbase.ipc.client.connection.idle-scan-interval.ms" 10000
    "hbase.ipc.client.connection.maxidletime" 10000
    "hbase.ipc.client.kill.max", 10
    "hbase.ipc.server.handler.queue.size", 100

    The new scheme does away with synchronization that purportedly would freeze out
    reads while we were cleaning up stale connections (according to HADOOP-9955)

    Also adds in new mechanism for accepting Connections by pulling in as many
    as we can at a time adding them to a Queue instead of doing one at a time.
    Can help when bursty traffic according to HADOOP-9956. Removes a blocking
    while Reader is busy parsing a request. Adds configuration
    "hbase.ipc.server.read.connection-queue.size" with default of 100 for
    queue size.
 2016-06-07 13:10:14 -07:00
Andrew Purtell 376ad0d986 HBASE-15698 Increment TimeRange not serialized to server (Ted Yu) 2016-06-06 22:17:00 -07:00
Enis Soztutar 3d7840a173 HBASE-15954 REST server should log requests with TRACE instead of DEBUG 2016-06-06 10:58:37 -07:00
Stephen Yuan Jiang de1b5ff776 HBASE-15955 Disable action in CatalogJanitor#setEnabled should wait for active cleanup scan to finish (Stephen Yuan Jiang) 2016-06-03 15:49:23 -07:00
Joseph Hwang 21e98271c3 HBASE-15883 Adding WAL files and tracking offsets in HBase. 
Implemented ReplicationQueuesHBaseImpl that tracks WAL offsets and replication queues in an HBase table.
Only wrote the basic tracking methods, have not implemented claimQueue() or HFileRef methods yet.
Wrote a basic unit test for ReplicationQueueHBaseImpl that tests the implemented functions on a single Region Server

Signed-off-by: Elliott Clark <elliott@fb.com>
Signed-off-by: Elliott Clark <eclark@apache.org>
 2016-06-03 15:23:10 -07:00
Apekshit 9a53d8b385 HBASE-15929 There are two tests named TestRegionServerMetrics. This causes slight issues when parsing console output for hanging tests and flaky tests analysis. Moving .../hbase/TestRegionServerMetrics.java to .../hbase/regionserver/TestRegionServerReadRequestMetrics.java. (Apekshit) 
Change-Id: I379c15fe2c2c01bed53bddf7619d5f2a07c5640e
 2016-06-03 15:13:49 -07:00
Apekshit b557f0bec6 HBASE-15949 Cleanup TestRegionServerMetrics. 
@Before and @After to setup/teardown tables using @Rule to set table name based on testname.
Refactor out copy-pasted code fragments to single function.
(Apekshit)

Change-Id: Ic22e5027cc3952bab5ec30070ed20e98017db65a
 2016-06-03 14:27:24 -07:00
Matteo Bertozzi f0c159b5fe HBASE-15927 Remove HMaster.assignRegion() 2016-06-03 12:36:25 -07:00
Ashish Singhi 0cbce07626 HBASE-15888 Extend HBASE-12769 for bulk load data replication 2016-06-03 18:42:00 +05:30
Lars Hofhansl fc890a2ecb HBASE-15881 Allow BZIP2 compression. 2016-06-02 12:19:02 -07:00
tedyu 7e5d530870 HBASE-15727 Canary Tool for Zookeeper (churro morales) 2016-06-02 10:15:08 -07:00
tedyu cd2588001c HBASE-15933 Addendum - make merge decision when sizes of both regions are known 2016-06-02 09:30:36 -07:00
tedyu cfe868d56e HBASE-15933 NullPointerException may be thrown from SimpleRegionNormalizer#getRegionSize() 2016-06-02 01:55:28 -07:00
Apekshit 1a27278624 HBASE-15915 Set timeouts on hanging tests. Tests run: 8, Time elapsed: 94.191 sec - in org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures Tests run: 103, Time elapsed: 69.057 sec - in org.apache.hadoop.hbase.regionserver.TestHRegion Tests run: 103, Time elapsed: 67.957 sec - in org.apache.hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush Tests run: 5, Time elapsed: 34.629 sec - in org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster Tests run: 9, Time elapsed: 82.913 sec - in org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient Tests run: 9, Time elapsed: 80.458 sec - in org.apache.hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient (Apekshit) 
Change-Id: Ia7b986ca6276ff5498f588c0b1b9c570e2a9d798

Signed-off-by: stack <stack@apache.org>
 2016-05-30 21:13:55 -07:00
Daniel Vimont e6d613de70 HBASE-15835 - Prevent HBaseTestingUtility#startMiniCluster from throwing _HMasterAddress already in use_ RuntimeException when another HBase thread is running 
Signed-off-by: stack <stack@apache.org>
 2016-05-30 20:05:19 -07:00
Jurriaan Mous a1f0c1cbb3 HBASE-15875 Remove HTable references and HTableInterface 
Signed-off-by: stack <stack@apache.org>
 2016-05-30 19:56:54 -07:00
Jurriaan Mous cdd532da8a HBASE-15610 Remove deprecated HConnection for 2.0 thus removing all PB references for 2.0 
Signed-off-by: stack <stack@apache.org>
 2016-05-29 07:50:55 -07:00
Sean Mackrory f4470af95d HBASE-15891. Closeable resources potentially not getting closed if exception is thrown. 
Signed-off-by: stack <stack@apache.org>
 2016-05-28 21:32:22 -07:00
Sean Mackrory 3b6e6e6c25 HBASE-15889. String case conversions are locale-sensitive, used without locale 
Signed-off-by: Sean Busbey <busbey@apache.org>
 2016-05-28 10:41:31 -07:00
Mikhail Antonov 60c8f76a9d HBASE-15908 Checksum verification is broken due to incorrect passing of ByteBuffers in DataChecksum (Mikhail Antonov and Appy) 2016-05-28 03:14:52 -07:00
Gary Helmling da0d74cd27 HBASE-15830 SASL encryption doesn't work with AsyncRpcChannelImpl (Colin Ma) 2016-05-26 21:58:27 -07:00
Enis Soztutar 43d1a999fe HBASE-15837 Memstore size accounting is wrong if postBatchMutate() throws exception 2016-05-26 11:12:17 -07:00
Apekshit 94696d4067 HBASE-14635 Fix flaky test TestSnapshotCloneIndependence 
Signed-off-by: Matteo Bertozzi <matteo.bertozzi@cloudera.com>
 2016-05-25 21:14:06 -07:00
Joseph Hwang b3362ccb0c HBASE-15471 Added in Priority, General, and Replication queue sizes for RegionServers to the Web UI 
Changed UI labels so that queue "size" refers to size in bytes and queue "length" refers to number of items in queue.

Signed-off-by: Elliott Clark <elliott@fb.com>
 2016-05-25 15:10:48 -07:00
tedyu fa74baeb40 HBASE-15884 NPE in StoreFileScanner#skipKVsNewerThanReadpoint during reverse scan (Sergey Soldatov) 2016-05-25 08:58:28 -07:00
tedyu 39dc19236e HBASE-15806 revert due to discussion on security 2016-05-24 02:08:11 -07:00
tedyu c03ea895c4 HBASE-15806 An endpoint-based export tool (ChiaPing Tsai) 2016-05-23 10:51:25 -07:00
Jurriaan Mous 7130a222ce HBASE-15876 Remove doBulkLoad(Path hfofDir, final HTable table) though it has not been through a full deprecation cycle 
Signed-off-by: stack <stack@apache.org>
 2016-05-23 10:25:22 -07:00
tedyu ae42c65cfd HBASE-15802 ConnectionUtils should use ThreadLocalRandom instead of Random (Matt Warhaftig) 2016-05-21 21:15:10 -07:00
Jerry He 56e4b85d06 HBASE-14818 user_permission does not list namespace permissions (li xiang) 2016-05-21 20:06:05 -07:00
tedyu 0671cba65f HBASE-15529 Override needBalance in StochasticLoadBalancer (Guanghao Zhang) 2016-05-21 04:03:00 -07:00
anoopsjohn 0e52ac2464 HBASE-15228 Add the methods to RegionObserver to trigger start/complete restoring WALs. (Toshihiro Suzuki) 2016-05-21 09:56:28 +05:30
tedyu 2bb07e209e HBASE-15529 Revert due to TestRegionRebalancing failing 2016-05-20 12:56:17 -07:00
Gary Helmling 4793988aef HBASE-15856 Don't cache unresolved addresses for connections 2016-05-20 09:36:11 -07:00
tedyu 58798fc091 HBASE-15529 Override needBalance in StochasticLoadBalancer (Guanghao Zhang) 2016-05-20 07:44:05 -07:00
Matteo Bertozzi d34f5d4b96 HBASE-15865 Move TestTableDeleteFamilyHandler and TestTableDescriptorModification handler tests to procedure 2016-05-20 07:03:11 -07:00
Matteo Bertozzi 7a84ee262f HBASE-15864 Reuse the testing helper to wait regions in transition 2016-05-20 06:48:38 -07:00
eshcar a27504c701 HBASE-14920: Compacting memstore 
Signed-off-by: stack <stack@apache.org>
 2016-05-20 03:41:43 -07:00
Jerry He af5146ee04 HBASE-15465 userPermission returned by getUserPermission() for the selected namespace does not have namespace set (li xiang) 2016-05-19 20:35:58 -07:00
tedyu a050e1d9f8 HBASE-15847 VerifyReplication prefix filtering (Geoffrey Jacoby) 2016-05-19 06:38:27 -07:00
Vincent 460b41c800 HBASE-15617 Canary in regionserver mode might not enumerate all regionservers 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2016-05-18 16:25:29 -07:00
Stephen Yuan Jiang 8aa8abfcb6 HBASE-15850 Localize the configuration change in testCheckTableLocks to reduce flakiness of TestHBaseFsck test suite (Stephen Yuan Jiang) 2016-05-18 13:22:41 -07:00
Matteo Bertozzi 5ac54e6abb HBASE-15843 Replace RegionState.getRegionInTransition() Map with a Set 2016-05-18 12:48:26 -07:00
Jerry He 393bcd69f9 HBASE-15841 Performance Evaluation tool total rows may not be set correctly 2016-05-18 12:09:13 -07:00
Enis Soztutar c1ada0a373 HBASE-15824 LocalHBaseCluster gets bind exception in master info port - ADDENDUM 2016-05-18 11:36:10 -07:00
Phil Yang 3bd9220f0c HBASE-15593 Time limit of scanning should be offered by client 
Signed-off-by: zhangduo <zhangduo@apache.org>
 2016-05-18 19:13:15 +08:00
Ramkrishna 7b5d5394c0 HBASE-15784 Misuse core/maxPoolSize of LinkedBlockingQueue in 
ThreadPoolExecutor (Jingcheng Du)
 2016-05-18 12:40:43 +05:30
zhangduo b2b3b1fa4d Revert "HBASE-15593 Time limit of scanning should be offered by client (Phil Yang)" 
This reverts commit e47bfb9078.
 2016-05-18 13:08:50 +08:00
stack e47bfb9078 HBASE-15593 Time limit of scanning should be offered by client (Phil Yang) 2016-05-17 12:49:31 -07:00
Elliott Clark 9269b8199e HBASE-15840 WAL.proto compilation broken for cpp 2016-05-17 12:04:14 -07:00
tedyu 838c306c0a HBASE-15769 Perform validation on cluster key for add_peer (Matt Warhaftig) 2016-05-17 11:01:26 -07:00
tedyu 00e61d2cfd HBASE-15842 SnapshotInfo should display ownership information 2016-05-16 20:34:04 -07:00
Enis Soztutar c3223a59ff HBASE-15824 LocalHBaseCluster gets bind exception in master info port 2016-05-16 10:32:43 -07:00
Sean Busbey e0aff10901 HBASE-15780 Make AuthUtil public and rely on it for talking to secure HBase. 
Signed-off-by: Gary Helmling <garyh@apache.org>
Signed-off-by: Mikhail Antonov <antonov@apache.org>
Signed-off-by: stack <stack@apache.org>
 2016-05-16 00:50:14 -05:00
Mikhail Antonov 2482062d34 HBASE-15615 Wrong sleep time when RegionServerCallable need retry (Guanghao Zhang) 2016-05-15 20:37:26 -07:00
Apekshit 60e19f60a9 HBASE-13532 Make UnknownScannerException less scary by giving more information in the exception string. (Apekshit) 
Signed-off-by: Matteo Bertozzi <matteo.bertozzi@cloudera.com>
 2016-05-13 15:00:23 -07:00
Jerry He acca95fb50 HBASE-15808 Reduce potential bulk load intermediate space usage and waste 2016-05-12 15:43:48 -07:00
zhangduo 1267f76e9a HBASE-15813 Rename DefaultWALProvider to a more specific name and clean up unnecessary reference to it 2016-05-12 11:27:00 +08:00
tedyu c867858c44 HBASE-15812 Revert pending investigation on whether metrics dump can be re-implemented 2016-05-11 09:39:20 -07:00
tedyu f3fee82ac4 HBASE-15812 HttpServer fails to come up against hadoop trunk 2016-05-11 09:00:27 -07:00
Jurriaan Mous a11091c49c HBASE-15798 Add Async RpcChannels to all RpcClients 
Signed-off-by: stack <stack@apache.org>
 2016-05-10 07:44:51 -07:00
Matteo Bertozzi 8604f9eebb HBASE-15415 Improve Master WebUI snapshot information (huaxiang sun) 2016-05-09 20:20:49 -07:00
Apekshit 11740570c1 HBASE-15236 Inconsistent cell reads over multiple bulk-loaded HFiles. In KeyValueHeap, if two cells are same i.e. have same key and timestamp, then instead of directly using seq id to determine newer one, we should use StoreFile.Comparater.SEQ_ID because that's what is used to determine order of hfiles. In this patch, we assign each scanner an order based on it's index in storefiles list, which is then used in KeyValueHeap to disambiguate between same cells. Changes the getSequenceId() in KeyValueScanner class to getScannerOrder(). Testing: Adds unit test to TestKeyValueHeap. Manual testing: Three cases (Tables t, t2, t3 in the jira description), single region, 2 hfiles with same seq id, timestamps and duplicate KVs. Made sure that returned kv was same for get and scan. (Apekshit) 
Change-Id: I22600c91c0a51fb63eb17db73472839d2f13957c

Signed-off-by: stack <stack@apache.org>
 2016-05-09 16:57:06 -07:00
Esteban Gutierrez a6e29676db HBASE-15612: Minor improvements to CellCounter and RowCounter documentation 
Signed-off-by: stack <stack@apache.org>
 2016-05-09 14:12:43 -07:00
Enis Soztutar b75b226804 HBASE-15740 Replication source.shippedKBs metric is undercounting because it is in KB 2016-05-09 10:25:49 -07:00
zhangduo 05378cbf69 HBASE-15609 Addendum fix compilation error 2016-05-09 20:45:04 +08:00
Ramkrishna 97ad33c691 HBASE-15609 Remove PB references from Result, DoubleColumnInterpreter and 
any such public facing class for 2.0 (Ram)
 2016-05-09 14:56:00 +05:30
Apekshit 48b77dbcbc HBASE-11625 - Verifies data before building HFileBlock. - Adds HFileBlock.Header class which contains information about location of fields. Testing: Adds CorruptedFSReaderImpl to TestChecksum. 
Change-Id: I107e9636b28abb6b15ec329e885f1e31b1b1b988

Signed-off-by: stack <stack@apache.org>
 2016-05-08 17:33:04 -07:00
Matteo Bertozzi 99e2deb86f HBASE-15778 replace master.am and master.sm direct access with getter calls 2016-05-07 05:19:15 -07:00
Matteo Bertozzi 513ca3483f HBASE-15776 Replace master.am.getTableStateManager() with the direct master.getTableStateManager() 2016-05-06 19:30:45 -07:00
Gary Helmling 86ca09e0e5 HBASE-15773 Improvements to CellCounter job 2016-05-06 11:08:18 -07:00
Matteo Bertozzi d90f0571e6 HBASE-15781 Remove unused TableEventHandler and TotesHRegionInfo 2016-05-06 10:05:42 -07:00
Ashish Singhi 34e9a6ff30 HBASE-15669 HFile size is not considered correctly in a replication request 2016-05-06 17:26:17 +05:30
Ramkrishna bec81b1977 HBASE-15608 Remove PB references from SnapShot related Exceptions (ram) 2016-05-06 17:06:25 +05:30
zhangduo 0d252918f6 HBASE-15743 Add Transparent Data Encryption support for FanOutOneBlockAsyncDFSOutput 2016-05-06 08:55:58 +08:00
tedyu 532b914f08 HBASE-15759 RegionObserver.preStoreScannerOpen() doesn't have acces to current readpoint 2016-05-05 15:20:37 -07:00
tedyu c716a75416 HBASE-15767 upgrade httpclient to 4.3.6 
Signed-off-by: Sean Busbey <busbey@apache.org>
 2016-05-05 15:41:26 -05:00
Alex Moundalexis 0bf065a5d5 HBASE-15768 fix capitalization of ZooKeeper usage 
Signed-off-by: Sean Busbey <busbey@apache.org>
 2016-05-05 15:35:44 -05:00
Stephen Yuan Jiang b7ce55f001 HBASE-15575 Rename table DDL *Handler methods in MasterObserver to more meaningful names (Stephen Yuan Jiang) 2016-05-05 11:44:22 -07:00
tedyu efcf94def7 HBASE-15563 'counter' may overflow in BoundedGroupingStrategy (Matt Warhaftig) 2016-05-05 10:28:33 -07:00
Matteo Bertozzi c0ce47d1cb HBASE-15755 SnapshotDescriptionUtils and SnapshotTestingUtils do not have any Interface audience marked 2016-05-05 07:24:29 -07:00
Gary Helmling d23d600d76 HBASE-15741 Provide backward compatibility for HBase coprocessor service names 2016-05-04 12:09:57 -07:00
Enis Soztutar 29eb3e8bcc HBASE-15613 TestNamespaceCommand times out 2016-05-04 11:37:03 -07:00
Ramkrishna 809ed326f0 HBASE-15760 TestBlockEvictionFromClient#testParallelGetsAndScanWithWrappedRegionScanner 
fails in master branch (Ram)
 2016-05-04 22:07:11 +05:30
Matteo Bertozzi a8a2c516a0 HBASE-15763 Isolate Wal related stuff from MasterFileSystem 2016-05-04 07:59:44 -07:00
zhangduo 6abe1879dd HBASE-15754 Add testcase for AES encryption 2016-05-04 09:40:57 +08:00
Yu Li 082d193455 HBASE-15675 Add more details about region on table.jsp 2016-05-03 20:18:48 +08:00
tedyu c77a153001 HBASE-15752 ClassNotFoundException is encountered when custom WAL codec is not found in WALPlayer job 2016-05-03 01:57:25 -07:00
Ramkrishna c06a976a98 HBASE-15607 Remove PB references from Admin for 2.0 (Ram) 2016-05-03 10:51:46 +05:30
chenheng d77972ff16 HBASE-15714 We are calling checkRow() twice in doMiniBatchMutation() 2016-05-03 09:42:41 +10:00
Mikhail Antonov 58c4c3d174 HBASE-15703 Deadline scheduler needs to return to the client info about skipped calls, not just drop them 2016-05-02 15:23:07 -07:00
Mikhail Antonov bbc7b90335 HBASE-15281 Allow the FileSystem inside HFileSystem to be wrapped (Rajesh Nishtala) 2016-05-02 13:24:51 -07:00
anoopsjohn d1130582d5 HBASE-15735 Tightening of the CP contract. 2016-05-02 09:48:51 +05:30
Jurriaan Mous 15631a76f5 HBASE-15744 Port over small format/text improvements from HBASE-13784 
(Jurriaan Mous)

Signed-off-by: stack <stack@apache.org>
 2016-05-01 19:48:43 -07:00
chenheng ccd8888b4b Revert "HBASE-15278 AsyncRPCClient hangs if Connection closes before RPC call response" 
This reverts commit 01c0448ccd.
 2016-04-30 11:37:05 +08:00
chenheng 01c0448ccd HBASE-15278 AsyncRPCClient hangs if Connection closes before RPC call response 2016-04-30 09:27:32 +08:00
tedyu c236409c3b HBASE-15357 TableInputFormatBase getSplitKey does not handle signed bytes correctly (Nathan Schile) 2016-04-29 17:08:25 -07:00
Mikhail Antonov 730b077666 HBASE-15551 Make call queue too big exception use servername 2016-04-29 10:25:29 -07:00
anoopsjohn 4706303186 HBASE-15706 HFilePrettyPrinter should print out nicely formatted tags. (huaxiang sun) 2016-04-29 09:23:57 +05:30
tedyu 8a28c23431 HBASE-15686 Add override mechanism for the exempt classes when dynamically loading table coprocessor 2016-04-28 09:19:12 -07:00
Ramkrishna 889d89a74c HBASE-15697 Excessive TestHRegion running time on branch-1 (Ram) 2016-04-28 14:24:39 +05:30
tedyu 9343ef438f HBASE-15676 FuzzyRowFilter fails and matches all the rows in the table if the mask consists of all 0s (Matt Warhaftig) 2016-04-27 21:00:11 -07:00
chenheng e8b37422c3 HBASE-15720 Print row locks at the debug dump page 2016-04-28 11:40:29 +08:00
Gary Helmling ce318a2906 HBASE-15658 Avoid excess meta cache clearing for RegionServerCallable 2016-04-26 22:54:13 -07:00
tedyu ebb5d421f9 HBASE-15707 ImportTSV bulk output does not support tags with hfile.format.version=3 (huaxiang sun) 2016-04-26 11:21:29 -07:00
Phil Yang 53d7316075 HBASE-15645 hbase.rpc.timeout is not used in operations of HTable 
Signed-off-by: stack <stack@apache.org>
 2016-04-26 10:51:29 -07:00
tedyu dce59294ff HBASE-15634 TestDateTieredCompactionPolicy#negativeForMajor is flaky (Duo Zhang and Ted Yu) 2016-04-26 09:23:54 -07:00
tedyu 793d976d87 HBASE-14252 RegionServers fail to start when setting hbase.ipc.server.callqueue.scan.ratio to 0 (Yubao Liu and Ted Yu) 2016-04-25 13:26:47 -07:00
zhangduo 1eac103ead HBASE-15628 Implement an AsyncOutputStream which can work with any FileSystem implementation 2016-04-25 22:20:27 +08:00
zhangduo 3cfe363f13 HBASE-15699 Can not sync AsyncFSWAL if no edit is appended 2016-04-25 13:17:26 +08:00
Enis Soztutar 106fe99a9a HBASE-15670 Add missing Snapshot.proto to the maven profile for compiling protobuf 2016-04-22 11:34:49 -07:00
Matteo Bertozzi 1ecb10ce07 HBASE-15688 Use MasterServices directly instead of casting to HMaster when possible 2016-04-22 10:15:58 -07:00
Enis Soztutar 4c0587134a HBASE-15671 Add per-table metrics on memstore, storefile and regionsize (Alicia Ying Shu) 2016-04-21 13:33:26 -07:00
chenheng 03f3c392a3 HBASE-15674 HRegionLocator#getAllRegionLocations should put the results in cache 2016-04-21 14:19:39 +08:00
zhangduo 58f175f0ea HBASE-15368 Add pluggable window support 2016-04-21 12:02:16 +08:00
Andrew Purtell 2c26fe37ac HBASE-15614 Report metrics from JvmPauseMonitor 2016-04-20 17:37:34 -07:00
Enis Soztutar 18d70bc680 HBASE-15518 Add Per-Table metrics back (Alicia Ying Shu) 2016-04-20 14:35:45 -07:00
stack 6dd938c20b HBASE-15640 L1 cache doesn't give fair warning that it is showing partial stats only when it hits limit 2016-04-20 14:14:37 -07:00
stack fa215a67e2 HBASE-15385 PREFETCH_BLOCKS_ON_OPEN in HColumnDescriptor is ignored 2016-04-20 09:38:30 -07:00
tedyu 3e1bdccc53 HBASE-15679 Assertion on wrong variable in TestReplicationThrottler#testThrottling 2016-04-20 07:11:17 -07:00
stack 954a41790c HBASE-15672 hadoop.hbase.security.visibility.TestVisibilityLabelsWithDeletes fails (Anoop Sam John) 2016-04-19 17:10:48 -07:00
Apekshit 501030d204 HBASE-15673 Fix latency metrics for multiGet. - Also fixes some stuff in help text. (Apekshit) 
Change-Id: I49d76896d5cc070b9f60dadf6746865e98fbbbd0

Signed-off-by: stack <stack@apache.org>
 2016-04-19 16:12:43 -07:00
zhangduo bd3b9753a9 HBASE-15665 Support using different StoreFileComparators for different CompactionPolicies 2016-04-19 19:35:17 +08:00
Gábor Lipták 6d7a7fa3aa HBASE-13372 Add unit tests for SplitTransaction and RegionMergeTransaction listeners 
Signed-off-by: Andrew Purtell <apurtell@apache.org>
 2016-04-18 17:04:56 -07:00
Ashish Singhi 70687c18bb HBASE-15668 HFileReplicator fails to replicate other hfiles in the request when a hfile in not found in FS anywhere 2016-04-18 22:17:02 +05:30
zhangduo f2e0aca2b6 HBASE-15664 Use Long.MAX_VALUE instead of HConstants.FOREVER in CompactionPolicy 2016-04-18 20:10:31 +08:00
tedyu e9211e415a HBASE-15287 mapreduce.RowCounter returns incorrect result with binary row key inputs (Matt Warhaftig) 2016-04-16 12:46:21 -07:00
stack d815211f5a HBASE-15650 Remove TimeRangeTracker as point of contention when many threads reading a StoreFile 
Refactor so we use the immutable, unsynchronized TimeRange when doing
    time-based checks at read time rather than use heavily synchronized
    TimeRangeTracker; let TimeRangeTracker be for write-time only.

    While in here, changed the Segment stuff so that when an immutable
    segment, it uses TimeRange rather than TimeRangeTracker too.

    M hbase-common/src/main/java/org/apache/hadoop/hbase/io/TimeRange.java
      Make allTime final.
      Add a includesTimeRange method copied from TimeRangeTracker.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/TimeRangeTracker.java
      Change name of a few methods so they match TimeRange methods that do
      same thing.
      (getTimeRangeTracker, getTimeRange, toTimeRange) add utility methods

    M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ImmutableSegment.java
      Change ImmutableSegment so it uses a TimeRange rather than
      TimeRangeTracker.. it is read-only. Redo shouldSeek, getMinTimestamp,
      updateMetaInfo, and getTimeRangeTracker so we use TimeRange-based
      implementations instead of TimeRangeTracker implementations.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MutableSegment.java
      Implement shouldSeek, getMinTimestamp, updateMetaInfo, and
      getTimeRangeTracker using TimeRangeTracker.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java
      Make methods that were using TimeRangeTracker abstract and instead
      have the implementations do these methods how they want either using
      TimeRangeTracker when a mutable segment or TimeRange when an immutable
      segment.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java
      Change Reader to use TimeRange-based checks instead of
      TimeRangeTracker.

    Signed-off-by: stack <stack@apache.org>
 2016-04-15 10:13:42 -07:00
Matteo Bertozzi 3e42f82600 HBASE-15622 Superusers does not consider the keytab credentials 2016-04-15 08:28:39 -07:00
chenheng a9f1c4a871 HBASE-15406 Split / merge switch left disabled after early termination of hbck 2016-04-15 14:09:52 +08:00
Apekshit 7efb9edecb HBASE-15296 Break out writer and reader from StoreFile. Done using Intellij15 Refactor > Move. (Apekshit) 
Change-Id: Ie719569cc3393e0b5361e9d462c3cf125ad5144e

Signed-off-by: stack <stack@apache.org>
 2016-04-13 22:43:03 -07:00
Elliott Clark 5a7c8dcb68 HBASE-15504 Fix Balancer in 1.3 not moving regions off overloaded regionserver 2016-04-13 18:07:00 -07:00
Apekshit 8f9e46a640 HBASE-15405 Fix PE logging and wrong defaults in help message. (Apekshit) 
Change-Id: Ib3007fae8d05e651b62bf4c15352ed43ea79e739
 2016-04-13 12:10:33 -07:00
Matteo Bertozzi ae7e5e29f9 HBASE-15481 Add pre/post roll to WALObserver 
Signed-off-by: Sean Busbey <busbey@apache.org>
 2016-04-13 00:43:51 -05:00
Stephen Yuan Jiang 543e7081f5 HBASE-15636 hard coded wait time out value in HBaseTestingUtility#waitUntilAllRegionsAssigned might cause test failure (Stephen Yuan Jiang) 2016-04-12 16:46:39 -07:00
Matteo Bertozzi d03cdc795c HBASE-15621 Suppress Hbase SnapshotHFile cleaner error messages when a snaphot is going on (Huaxiang Sun) 2016-04-12 12:17:55 -07:00
Matteo Bertozzi 0f23e61ec5 HBASE-15632 Undo the checking of lastStoreFlushTimeMap.isEmpty() introduced in HBASE-13145 (Huaxiang Sun) 2016-04-12 11:35:18 -07:00
tedyu ff9c92e168 HBASE-14985 TimeRange constructors should set allTime when appropriate (Geoffrey Jacoby) 2016-04-11 16:59:45 -07:00
Matteo Bertozzi 8964573394 HBASE-15627 Miss space and closing quote in AccessController#checkSystemOrSuperUser (Huaxiang Sun) 2016-04-11 08:28:26 -07:00
tedyu 8541fe4ad1 HBASE-15093 Replication can report incorrect size of log queue for the global source when multiwal is enabled (Ashu Pachauri) 2016-04-11 08:17:20 -07:00
Vasudevan a395922ad5 HBASE-15605 Remove PB references from HCD and HTD for 2.0 (Ram) 2016-04-11 14:12:07 +05:30
Jerry He 80df1cb7b6 HBASE-15591 ServerCrashProcedure not yielding 2016-04-10 17:02:39 -07:00
zhangduo f7d44e929f HBASE-15527 Refactor Compactor related classes 2016-04-10 09:26:28 +08:00
tedyu e0f31ba6e6 HBASE-15507 Online modification of enabled ReplicationPeerConfig (Geoffrey Jacoby) 2016-04-08 21:26:31 -07:00
zhangduo 6ea4994569 HBASE-15407 Add SASL support for fan out OutputStream 2016-04-08 21:46:47 +08:00
stack e450d94a2c HBASE-15380 Purge rollback support in Store etc. 2016-04-07 22:35:23 -07:00
zhangduo 394b89d153 HBASE-15537 Make multi WAL work with WALs other than FSHLog 2016-04-08 10:36:16 +08:00
Enis Soztutar 2dcd08bc3d HBASE-15586 Unify human readable numbers in the web UI 2016-04-07 17:07:33 -07:00
tedyu a146a71a33 HBASE-15400 Use DateTieredCompactor for Date Tiered Compaction - drop TestDateTieredCompaction.java 2016-04-07 15:01:00 -07:00
tedyu f60fc9d1a0 HBASE-15400 Use DateTieredCompactor for Date Tiered Compaction (Clara Xiong) 2016-04-07 14:58:59 -07:00
tedyu 3826894f89 HBASE-15369 Handle NPE in region.jsp (Samir Ahmic) 2016-04-06 06:52:51 -07:00
Enis Soztutar c93cffb95c HBASE-15587 FSTableDescriptors.getDescriptor() logs stack trace erronously 2016-04-05 18:13:40 -07:00
tedyu a93a8878fe HBASE-15485 Filter.reset() should not be called between batches (Phil Yang) 2016-04-04 12:52:24 -07:00
Ashish Singhi 33396c3629 HBASE-15578 Handle HBASE-15234 for ReplicationHFileCleaner 2016-04-04 15:02:19 +05:30
Matteo Bertozzi 79868bd394 HBASE-15582 SnapshotManifestV1 too verbose when there are no regions 2016-04-01 20:55:21 -07:00
Ashish Singhi 25419d8b18 HBASE-15424 Add bulk load hfile-refs for replication in ZK after the event is appended in the WAL 2016-04-01 15:40:36 +05:30
tedyu 5d79790c55 HBASE-15293 Handle TableNotFound and IllegalArgument exceptions in table.jsp (Samir Ahmic) 2016-04-01 02:13:18 -07:00
Stephen Yuan Jiang 89d750166d HBASE-15568 Procedure V2 - Remove CreateTableHandler in HBase Apache 2.0 release (Stephen Yuan Jiang) 2016-04-01 01:31:09 -07:00
Gary Helmling 2d8e0a0477 HBASE-15234 Don't abort ReplicationLogCleaner on ZooKeeper errors 2016-03-31 22:28:18 -07:00
Stephen Yuan Jiang e1d5c3d269 HBASE-15521 Procedure V2 - RestoreSnapshot and CloneSnapshot (Stephen Yuan Jiang) 2016-03-31 21:49:13 -07:00
Harsh J 7d3a89ce8e HBASE-15396 Enhance mapreduce.TableSplit to add encoded region name 
Signed-off-by: Sean Busbey <busbey@apache.org>
 2016-03-31 22:59:43 -05:00
tedyu 8c7f044efb HBASE-15571 Make MasterProcedureManagerHost accessible through MasterServices 2016-03-31 15:45:46 -07:00
zhangduo d6fd859451 HBASE-15538 Implement secure async protobuf wal writer 2016-03-31 10:33:35 +08:00
stack 9d56105eec HBASE-15324 Jitter may cause desiredMaxFileSize overflow in ConstantSizeRegionSplitPolicy and trigger unexpected split (Yu Li) 2016-03-30 13:31:09 -07:00
stack 21301a8a95 HBASE-15566 Add timeouts on TestMobFlushSnapshotFromClient and TestRegionMergeTransactionOnCluster 2016-03-30 13:23:20 -07:00
Elliott Clark a71ce6e738 HBASE-14983 Create metrics for per block type hit/miss ratios 
Summary: Missing a root index block is worse than missing a data block. We should know the difference

Test Plan: Tested on a local instance. All numbers looked reasonable.

Differential Revision: https://reviews.facebook.net/D55563
 2016-03-30 11:41:11 -07:00
Elliott Clark b18de5ef45 HBASE-15559 Fix BaseMasterAndRegionObserver doesn't implement all the methods 2016-03-30 11:17:08 -07:00
tedyu 31aee19f28 HBASE-15327 Canary will always invoke admin.balancer() in each sniffing period when writeSniffing is enabled (Jianwei Cui) 2016-03-30 08:58:42 -07:00
Enis Soztutar afdfd1bd9c HBASE-15295 MutateTableAccess.multiMutate() does not get high priority causing a deadlock - ADDENDUM for failing test 2016-03-29 15:02:18 -07:00
chenheng 7f39baf0f4 HBASE-11393 Replication TableCfs should be a PB object rather than a string 2016-03-29 10:25:29 +08:00
Enis Soztutar 0520097611 HBASE-15295 MutateTableAccess.multiMutate() does not get high priority causing a deadlock 2016-03-28 17:56:32 -07:00
tedyu cbf9c1e116 HBASE-15191 CopyTable and VerifyReplication - Option to specify batch size, versions (Parth Shah) 2016-03-28 17:25:00 -07:00
tedyu 7793bc54c2 HBASE-15548 SyncTable: sourceHashDir is supposed to be optional but won't work without (Dave Latham) 2016-03-28 16:52:44 -07:00
zhangduo c96b642f15 HBASE-15265 Implement an asynchronous FSHLog 2016-03-26 15:41:27 +08:00
tedyu b865fb9434 HBASE-15496 Throw RowTooBigException only for user scan/get (Guanghao Zhang) 2016-03-25 18:18:35 -07:00
zhangduo 11d11d3fcc HBASE-15389 Write out multiple files when compaction 2016-03-25 15:07:54 +08:00
chenheng e9c4f12856 HBASE-15384 Avoid using '/tmp' directory in TestBulkLoad 2016-03-25 12:06:55 +08:00
tedyu 07c597c8a2 HBASE-15526 Make SnapshotManager accessible through MasterServices 2016-03-24 15:59:46 -07:00
tedyu ad3feaa448 HBASE-15515 Improve LocalityBasedCandidateGenerator in Balancer (Guanghao Zhang) 2016-03-24 11:10:58 -07:00
Jerry He 7c93098210 HBASE-14256 Flush task message may be confusing when region is recovered (Gabor Liptak) 2016-03-24 10:06:38 -07:00
Elliott Clark e71bc71c84 HBASE-15475 Addendum 2016-03-24 09:11:19 -07:00
zhangduo fd5c0934b6 HBASE-15520 Fix broken TestAsyncIPC 2016-03-24 10:46:44 +08:00
Elliott Clark 925c185969 HBASE-15475 Allow TimestampsFilter to provide a seek hint 
Summary:
Allow TestTimestampFilterSeekHint to provide a seek next hint.
This can be incorrect as it might skip deletes. However it can
make things much much faster.

Test Plan: Added a unit test.

Differential Revision: https://reviews.facebook.net/D55617
 2016-03-23 18:56:52 -07:00
stack d7a4499dfc HBASE-15392 Single Cell Get reads two HFileBlocks 
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java
moreRowsMayExistAfterCell Exploit the fact a Scan is a Get Scan. Also save compares
if no non-default stopRow.

M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
optimize Add doc on what is being optimized. Also, if a Get Scan, do not
optimize else we'll keep going after our row is DONE.
Another place to make use of the Get Scan fact is when we are DONE.. if
Get Scan, we can close out the scan.

M hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
Add tests for Get Scans and optimize around block loading.
 2016-03-23 14:37:14 -07:00
Matteo Bertozzi 7177e4be64 HBASE-15508 Add command for exporting snapshot in hbase command script (Yufeng Jiang) 2016-03-23 10:26:32 -07:00
zhangduo 5fcadb86ab HBASE-15495 Connection leak in FanOutOneBlockAsyncDFSOutputHelper 2016-03-23 09:56:40 +08:00
stack 12f66e3060 HBASE-15477 Purge 'next block header' from cached blocks 
When we read from HDFS, we overread to pick up the next blocks header.
Doing this saves a seek as we move through the hfile; we save having to
do an explicit seek just to read the block header every time we need to
read the body.  We used to read in the next header as part of the
current blocks buffer. This buffer was then what got persisted to
blockcache; so we were over-persisting: our block plus the next blocks'
header (33 bytes).

This patch undoes this over-persisting.

Removes support for version 1 blocks (0.2 was added in hbase-0.92.0).
Not needed any more.

There is an open question on whether checksums should be persisted
when caching. The code seems to say no but if cache is SSD backed or
backed by anything that does not do error correction, we'll want
checksums.

Adds loads of documentation.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java
  (write) Add writing from a ByteBuff.

M hbase-common/src/main/java/org/apache/hadoop/hbase/nio/ByteBuff.java
  (toString) Add one so ByteBuff looks like ByteBuffer when you click on
  it in IDE

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
  Remove support for version 1 blocks.

  Cleaned up handling of metadata added when we serialize a block to
  caches. Metadata is smaller now.

  When we serialize (used when caching), do not persist the next blocks
  header if present.

  Removed a bunch of methods, a few of which had overlapping
  functionality and others that exposed too much of our internals.
  Also removed a bunch of constructors and unified the constructors we
  had left over making them share a common init method.
  Shutdown access to defines that should only be used internally here.

  Renamed all to do w/ 'EXTRA' and 'extraSerialization' to instead talk
  about metadata saved to caches; was unclear previously what EXTRA was
  about.

  Renamed static final declarations as all uppercase.

  (readBlockDataInternal): Redid. Couldn't make sense of it previously.
  Undid heavy-duty parse of header by constructing HFileBlock. Other
  cleanups. Its 1/3rd the length it used to be. More to do in here.
 2016-03-22 18:45:17 -07:00
stack a13d6e000d Revert "HBASE-15392 Single Cell Get reads two HFileBlocks M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java moreRowsMayExistAfterCell Exploit the fact a Scan is a Get Scan. Also save compares if no non-default stopRow." 
Revert mistaken commit

This reverts commit 7073f69993.
 2016-03-22 18:39:22 -07:00
stack c49d0caf53 Revert "HBASE-15477 Purge 'next block header' from cached blocks" 
Overcommit. Revert.

This reverts commit ef94b55239.
 2016-03-22 18:38:29 -07:00
stack 54a543de22 Revert "HBASE-15477 Purge 'next block header' from cached blocks" 
Overcommit. Revert to fix.

This reverts commit 000117ad9f.
 2016-03-22 18:37:25 -07:00
Enis Soztutar 3f3613a234 HBASE-15212 RRCServer should enforce max request size 2016-03-22 16:23:15 -07:00
stack 8af9ed7433 HBASE-15416 TestHFileBackedByBucketCache is flakey since it went in 2016-03-22 15:44:25 -07:00
Enis Soztutar b3fe4ed16c HBASE-15412 Add average region size metric (Alicia Ying Shu) 2016-03-22 14:46:27 -07:00
zhangduo 2348478506 HBASE-15360 addendum fix testCoDelScheduling 
Signed-off-by: stack <stack@apache.org>
 2016-03-22 13:52:26 -07:00
stack 000117ad9f HBASE-15477 Purge 'next block header' from cached blocks 
When we read from HDFS, we overread to pick up the next blocks header.
Doing this saves a seek as we move through the hfile; we save having to
do an explicit seek just to read the block header every time we need to
read the body.  We used to read in the next header as part of the
current blocks buffer. This buffer was then what got persisted to
blockcache; so we were over-persisting: our block plus the next blocks'
header (33 bytes).

This patch undoes this over-persisting.

Removes support for version 1 blocks (0.2 was added in hbase-0.92.0).
Not needed any more.

There is an open question on whether checksums should be persisted
when caching. The code seems to say no but if cache is SSD backed or
backed by anything that does not do error correction, we'll want
checksums.

Adds loads of documentation.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java
  (write) Add writing from a ByteBuff.

M hbase-common/src/main/java/org/apache/hadoop/hbase/nio/ByteBuff.java
  (toString) Add one so ByteBuff looks like ByteBuffer when you click on
  it in IDE

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
  Remove support for version 1 blocks.

  Cleaned up handling of metadata added when we serialize a block to
  caches. Metadata is smaller now.

  When we serialize (used when caching), do not persist the next blocks
  header if present.

  Removed a bunch of methods, a few of which had overlapping
  functionality and others that exposed too much of our internals.
  Also removed a bunch of constructors and unified the constructors we
  had left over making them share a common init method.
  Shutdown access to defines that should only be used internally here.

  Renamed all to do w/ 'EXTRA' and 'extraSerialization' to instead talk
  about metadata saved to caches; was unclear previously what EXTRA was
  about.

  Renamed static final declarations as all uppercase.

  (readBlockDataInternal): Redid. Couldn't make sense of it previously.
  Undid heavy-duty parse of header by constructing HFileBlock. Other
  cleanups. Its 1/3rd the length it used to be. More to do in here.
 2016-03-22 13:42:38 -07:00
stack ef94b55239 HBASE-15477 Purge 'next block header' from cached blocks 
When we read from HDFS, we overread to pick up the next blocks header.
Doing this saves a seek as we move through the hfile; we save having to
do an explicit seek just to read the block header every time we need to
read the body.  We used to read in the next header as part of the
current blocks buffer. This buffer was then what got persisted to
blockcache; so we were over-persisting: our block plus the next blocks'
header (33 bytes).

This patch undoes this over-persisting.

Removes support for version 1 blocks (0.2 was added in hbase-0.92.0).
Not needed any more.

There is an open question on whether checksums should be persisted
when caching. The code seems to say no but if cache is SSD backed or
backed by anything that does not do error correction, we'll want
checksums.

Adds loads of documentation.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java
  (write) Add writing from a ByteBuff.

M hbase-common/src/main/java/org/apache/hadoop/hbase/nio/ByteBuff.java
  (toString) Add one so ByteBuff looks like ByteBuffer when you click on
  it in IDE

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
  Remove support for version 1 blocks.

  Cleaned up handling of metadata added when we serialize a block to
  caches. Metadata is smaller now.

  When we serialize (used when caching), do not persist the next blocks
  header if present.

  Removed a bunch of methods, a few of which had overlapping
  functionality and others that exposed too much of our internals.
  Also removed a bunch of constructors and unified the constructors we
  had left over making them share a common init method.
  Shutdown access to defines that should only be used internally here.

  Renamed all to do w/ 'EXTRA' and 'extraSerialization' to instead talk
  about metadata saved to caches; was unclear previously what EXTRA was
  about.

  Renamed static final declarations as all uppercase.

  (readBlockDataInternal): Redid. Couldn't make sense of it previously.
  Undid heavy-duty parse of header by constructing HFileBlock. Other
  cleanups. Its 1/3rd the length it used to be. More to do in here.
 2016-03-22 13:34:51 -07:00
stack 891569bbd9 Add javadoc on how BBIOEngine works (by Anoop Sam John) 2016-03-22 13:01:13 -07:00
stack 7073f69993 HBASE-15392 Single Cell Get reads two HFileBlocks M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/ScanQueryMatcher.java moreRowsMayExistAfterCell Exploit the fact a Scan is a Get Scan. Also save compares if no non-default stopRow. 
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java
optimize Add doc on what is being optimized. Also, if a Get Scan, do not
optimize else we'll keep going after our row is DONE.
Another place to make use of the Get Scan fact is when we are DONE.. if
Get Scan, we can close out the scan.

M hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStoreScanner.java
Add tests for Get Scans and optimize around block loading.
 2016-03-22 13:00:41 -07:00
tedyu 47471c35e3 HBASE-15488 Add ACL for setting split merge switch 2016-03-22 10:38:13 -07:00
Enis Soztutar 797562e6c3 HBASE-15464 Flush / Compaction metrics revisited 2016-03-21 17:50:02 -07:00
tedyu f1d453599a HBASE-15433 SnapshotManager#restoreSnapshot not update table and region count quota correctly when encountering exception (Jianwei Cui) 2016-03-21 03:46:23 -07:00
Sean Busbey f48c92d143 HBASE-15478 add comments to syncRunnerIndex handling explaining constraints on possible values. 
Signed-off-by: zhangduo <zhangduo@apache.org>
 2016-03-21 00:28:15 -05:00
Yu Li d2d8058ab3 HBASE-15451 Remove unnecessary wait in MVCC 2016-03-19 21:24:07 +08:00
tedyu 3a6d683d63 HBASE-15456 CreateTableProcedure/ModifyTableProcedure needs to fail when there is no family in table descriptor (huaxiang sun) 2016-03-18 10:21:43 -07:00
tedyu 8c031e9d6b HBASE-15456 Revert due to unit test failure in thrift module 2016-03-17 17:36:44 -07:00
tedyu fbf58f330b HBASE-15325 ResultScanner allowing partial result will miss the rest of the row if the region is moved between two rpc requests (Phil Yang) 2016-03-17 11:03:29 -07:00
Elliott Clark ac1a7a4a78 HBASE-15470 Add a setting for Priority queue length 
Summary:
Move the config keys to one place
Make Two different config keys. One for default, one for priority

Test Plan: unit tests

Differential Revision: https://reviews.facebook.net/D55575
 2016-03-16 16:52:04 -07:00
Elliott Clark ecec35ae4e HBASE-15441 Fix WAL splitting when region has moved multiple times 
Summary:
Currently WAL splitting is broken when a region has been opened multiple times in recent minutes.

Region open and region close write event markers to the wal. These markers should have the sequence id in them. However it is currently getting 1. That means that if a region has moved multiple times in the last few mins then multiple split log workers will try and create the recovered edits file for sequence id 1. One of the workers will fail and on failing they will delete the recovered edits. Causing all split wal attempts to fail.

We need to:

It appears that the close event with a sequence id of one is coming from region warm up.

This patch fixes that by making sure the close on warm up doesn't happen. Also splitting will ignore any of the events that are already in the logs.

Test Plan: Unit tests pass

Differential Revision: https://reviews.facebook.net/D55557
 2016-03-16 16:26:09 -07:00
Matteo Bertozzi 3bf0945a11 HBASE-15430 Failed taking snapshot - Manifest proto-message too large (JunHo Cho) 2016-03-16 08:52:02 -07:00
stack 9eb25bd5c9 HBASE-15360 Fix flaky TestSimpleRpcScheduler (Duo Zhang) 2016-03-16 08:04:10 -07:00
tedyu f120602f1e HBASE-15456 CreateTableProcedure/ModifyTableProcedure needs to fail when there is no family in table descriptor (huaxiang sun) 2016-03-15 16:15:03 -07:00
Enis Soztutar 51259fe4a5 HBASE-15377 Per-RS Get metric is time based, per-region metric is size-based (Heng Chen) 2016-03-15 11:22:18 -07:00
tedyu 8fcc1e8e95 HBASE-15463 Region normalizer should check whether split/merge is enabled 2016-03-15 09:03:27 -07:00
Enis Soztutar ca816f0780 HBASE-6721 RegionServer Group based Assignment (Francis Liu) 2016-03-14 18:28:50 -07:00
anoopsjohn eea8b38dfa HBASE-15180 Reduce garbage created while reading Cells from Codec Decoder. 2016-03-11 13:58:41 +05:30
Enis Soztutar a979d85582 HBASE-15435 Add WAL (in bytes) written metric (Alicia Ying Shu) 2016-03-10 20:16:30 -08:00
stack bb5bfe82f2 Revert "HBASE-15261 Make Throwable t in DaughterOpener volatile (Huaxiang Sun)" 
Not needed afterall

This reverts commit d15ae0e6ab.
 2016-03-10 15:41:04 -08:00
tedyu d14b6c3810 HBASE-15425 Failing to write bulk load event marker in the WAL is ignored (Ashish Singhi) 2016-03-09 18:49:41 -08:00
tedyu ad9b91a904 HBASE-15378 Scanner cannot handle heartbeat message with no results (Phil Yang) 2016-03-09 09:28:54 -08:00
Matteo Bertozzi 9e967e5c1d HBASE-15113 Procedure v2 - Speedup eviction of sys operation results 2016-03-08 19:47:15 -08:00
Liu Shaohui 7e4e8dc3e9 HBASE-15420: TestCacheConfig failed after HBASE-15338 2016-03-09 10:15:35 +08:00
Gábor Lipták 14217cef24 HBASE-15364 Fix unescaped < and > characters in Javadoc 2016-03-08 16:44:04 -08:00
Mikhail Antonov 46cc3d4972 HBASE-15137 CallTimeoutException and CallQueueTooBigException should trigger PFFE 2016-03-07 20:54:54 -08:00
stack 4f044cf6be HBASE-15421 Convert TestStoreScanner to junit4 from junit3 and clean up close of scanners 2016-03-07 20:46:46 -08:00
Mikhail Antonov e477c143bc HBASE-15354 Use same criteria for clearing meta cache for all operations (addendum) (Ashu Pachauri) 2016-03-07 18:11:15 -08:00
stack d15ae0e6ab HBASE-15261 Make Throwable t in DaughterOpener volatile (Huaxiang Sun) 2016-03-07 16:34:11 -08:00
stack 0f14856b01 HBASE-15416 TestHFileBackedByBucketCache is flakey since it went in 2016-03-07 16:31:09 -08:00
tedyu ed977fd125 HBASE-15243 Utilize the lowest seek value when all Filters in MUST_PASS_ONE FilterList return SEEK_NEXT_USING_HINT 2016-03-07 16:15:59 -08:00
tedyu dfc650a953 HBASE-15243 Revert due to compilation error - master branch change makes the patch stale 2016-03-07 15:59:29 -08:00
tedyu 05c1309b3c HBASE-15243 Utilize the lowest seek value when all Filters in MUST_PASS_ONE FilterList return SEEK_NEXT_USING_HINT 2016-03-07 15:41:06 -08:00
tedyu 05161fcbfd HBASE-15393 Addendum adds TestReplicationAdminWithTwoDifferentZKClusters (Ashish) 2016-03-07 04:56:00 -08:00
chenheng f30afa05d9 HBASE-15376 ScanNext metric is size-based while every other per-operation metric is time based 2016-03-07 17:36:40 +08:00
Liu Shaohui 4483b3cbed HBASE-15338: Add a option to disable the data block cache for testing the performance of underlying file system 2016-03-07 16:37:07 +08:00
Jesse Yates ef712df944 HBASE-14703 HTable.mutateRow does not collect stats (Heng Chen) 2016-03-05 11:01:45 -08:00
tedyu d083e4f29f HBASE-15393 Enable table replication command will fail when parent znode is not default in peer cluster (Ashish Singhi) 2016-03-05 06:28:09 -08:00
Ashu Pachauri 27cf0c8c36 HBASE-15354 Same criteria to clear metacache across operations (Ashu Pachauri) 
Signed-off-by: Mikhail Antonov <antonov@apache.org>
 2016-03-04 17:55:35 -08:00
stack 7dabcf23e8 HBASE-15373 DEPRECATED_NAME_OF_NO_LIMIT_THROUGHPUT_CONTROLLER_CLASS value is wrong in CompactionThroughputControllerFactory 2016-03-03 19:42:42 -08:00
tedyu 4fba1c3627 HBASE-15291 FileSystem not closed in secure bulkLoad (Yong Zhang) 2016-03-03 15:59:37 -08:00
Jonathan M Hsieh f658f3ef83 HBASE-15356 Remove unused imports (Youngjoon Kim) 2016-03-03 11:42:38 -08:00
stack 8ace5bbfce HBASE-15366 Add doc, trace-level logging, and test around hfileblock 
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java
 Make it emit its toString in format that matches the way we log
 elsewhere

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
 Capitalize statics.

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java
 Verify and cleanup documentation of hfileblock format at head of class.
 Explain what 'EXTRA_SERIALIZATION_SPACE' is all about.
 Connect how we serialize and deserialize... done in different places
 and one way when pulling from HDFS and another when pulling from cache
 (TO BE FIXED). Shut down a load of public access.

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java
 Add trace-level logging

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileScanner.java
 Make it Closeable
 2016-03-03 09:16:30 -08:00
chenheng 4b3e38705c HBASE-15329 Cross-Site Scripting: Reflected in table.jsp (Samir Ahmic) 2016-03-03 12:20:00 +08:00
zhangduo 5e395c4294 HBASE-15375 Do not write to '/tmp' in TestRegionMover 2016-03-02 17:04:19 +08:00
tedyu edc5ab5af9 HBASE-15291 Revert due to the bug Haitao discovered 2016-03-02 00:49:17 -08:00
eshcar dc44943666 HBASE-15359 Simplifying segment hierarchy 
Signed-off-by: stack <stack@apache.org>
 2016-03-01 23:25:11 -08:00