80a1f8fa2a
HBASE-17428 Implement informational RPCs for space quotas
...
Create some RPCs that can expose the in-memory state that the
RegionServers and Master hold to drive the space quota "state machine".
Then, create some hbase shell commands to interact with those.
2017-05-22 13:41:35 -04:00
4ad49bc3ac
HBASE-17478 Avoid reporting FS use when quotas are disabled
...
Also, gracefully produce responses when quotas are disabled.
2017-05-22 13:41:35 -04:00
6c9082fe16
HBASE-17259 API to remove space quotas on a table/namespace
2017-05-22 13:41:35 -04:00
34ba143fc8
HBASE-17001 Enforce quota violation policies in the RegionServer
...
The nuts-and-bolts of filesystem quotas. The Master must inform
RegionServers of the violation of a quota by a table. The RegionServer
must apply the violation policy as configured. Need to ensure
that the proper interfaces exist to satisfy all necessary policies.
This required a massive rewrite of the internal tracking by
the general space quota feature. Instead of tracking "violations",
we need to start tracking "usage". This allows us to make the decision
at the RegionServer level as to when the files in a bulk load request
should be accept or rejected which ultimately lets us avoid bulk loads
dramatically exceeding a configured space quota.
2017-05-22 13:41:35 -04:00
98b4181f43
HBASE-16999 Implement master and regionserver synchronization of quota state
...
* Implement the RegionServer reading violation from the quota table
* Implement the Master reporting violations to the quota table
* RegionServers need to track its enforced policies
2017-05-22 13:41:35 -04:00
533470f8c8
HBASE-16998 Implement Master-side analysis of region space reports
...
Adds a new Chore to the Master that analyzes the reports that are
sent by RegionServers. The Master must then, for all tables with
quotas, determine the tables that are violating quotas and move
those tables into violation. Similarly, tables no longer violating
the quota can be moved out of violation.
The Chore is the "stateful" bit, managing which tables are and
are not in violation. Everything else is just performing
computation and informing the Chore on the updated state.
Added InterfaceAudience annotations and clean up the QuotaObserverChore
constructor. Cleaned up some javadoc and QuotaObserverChore. Reuse
the QuotaViolationStore impl objects.
2017-05-22 13:41:35 -04:00
7fb0ac26e3
HBASE-17557 HRegionServer#reportRegionSizesForQuotas() should respond to UnsupportedOperationException
2017-05-22 13:41:35 -04:00
6b334cd817
HBASE-17000 Implement computation of online region sizes and report to the Master
...
Includes a trivial implementation of the Master-side collection to
avoid. Only enough to write a test to verify RS collection.
2017-05-22 13:41:35 -04:00
f74e051bce
HBASE-16996 Implement storage/retrieval of filesystem-use quotas into quota table (Josh Elser)
2017-05-22 13:41:35 -04:00
23ea2c36f5
HBASE-18068 Fix flaky test TestAsyncSnapshotAdminApi
...
- internalRestoreSnapshot() returns future which completes by just getting proc_id from master. Changed it to wait for the procedure to complete.
- Refactor TestAsyncSnapshotAdminApi: Add cleanup() which deletes all tables and snapshots after every test run. Simplifies individual tests.
Change-Id: Idc30fb699db32d58fd0f60da220953a430f1d3cc
2017-05-22 09:20:37 -07:00
3aac047a4f
HBASE-18069 Fix flaky test TestReplicationAdminWithClusters#testDisableAndEnableReplication
2017-05-22 17:17:25 +08:00
709f5a1980
HBASE-18075 Support non-latin table names and namespaces
2017-05-21 22:24:12 -04:00
1ceb25cf09
HBASE-18081 The way we process connection preamble in SimpleRpcServer is broken
2017-05-21 20:36:33 +08:00
1520c8fd4d
HBASE-18056 Make the default behavior of CompactionPipeline to merge it segments into one, due to better read performance in this case
2017-05-21 12:27:57 +03:00
8b70d043e4
HBASE-18071 Fix flaky test TestStochasticLoadBalancer#testBalanceCluster
...
Test was failing on clusters with large number of servers or regions. Using commonly using config settings like some other tests seems to work.
Signed-off-by: Michael Stack <stack@apache.org>
2017-05-19 11:09:28 -07:00
3fe4b28bb0
HBASE-15616 Allow null qualifier for all table operations
2017-05-19 17:47:08 +08:00
958cd2d1b7
HBASE-18035 Meta replica does not give any primaryOperationTimeout to primary meta region (huaxiang sun)
2017-05-18 15:56:41 -07:00
6dc4190c07
HBASE-18049 It is not necessary to re-open the region when MOB files cannot be found
2017-05-18 18:54:58 +08:00
37dd8ff722
HBASE-11013: Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions
...
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2017-05-18 17:39:50 +08:00
32d2062b5c
HBASE-18019 Close redundant memstore scanners
2017-05-18 16:07:21 +08:00
62d7323023
HBASE-18053 AsyncTableResultScanner will hang when scan wrong column family
2017-05-17 12:16:51 +08:00
c1b45a2c45
HBASE-18016 Changes to inherit default behavior of abort from StateMachineProcedure making TruncateTableProcedure abortable
...
This will allow abort and manual cleanup of stuck instances of TruncateTableProcedure.
Signed-off-by: Michael Stack <stack@apache.org>
2017-05-16 18:57:54 -07:00
5eb1b7b96c
HBASE-18018 Changes to support abort for all procedures by default
...
The default behavior for abort() method of StateMachineProcedure is changed to support aborting all procedures irrespective of if rollback is supported or not. Currently its observed that sometimes procedures may fail on a step which will be considered as retryable error as abort is not supported. As a result procedure may stuck in a endless loop repeating same step again.User should have an option to abort any stuck procedure and do clean up manually. Please refer to HBASE-18016 and discussion there.
Signed-off-by: Michael Stack <stack@apache.org>
2017-05-16 18:56:32 -07:00
67d1358311
HBASE-18043 Institute a hard limit for individual cell size that cannot be overridden by clients. - Addendum to fix test issue in TestMobStoreScanner.
2017-05-16 18:53:27 +05:30
ad9ffaaafd
HBASE-18055 Releasing L2 cache HFileBlocks before shipped() when switching from pread to stream causes result corruption
2017-05-16 21:16:36 +08:00
6b60ba8ade
HBASE-18043 Institute a hard limit for individual cell size that cannot be overridden by clients
2017-05-15 18:03:33 -07:00
341223d86c
HBASE-18012 Move RpcServer.Connection to a separated file
2017-05-15 18:07:38 +08:00
5cdaca5c00
HBASE-16436 Adding CellChunkMap code, its tests and fixes to all code review comments
2017-05-14 16:04:36 +03:00
305ffcb040
HBASE-17938 General fault - tolerance framework for backup/restore operations (Vladimir Rodionov)
2017-05-12 09:27:58 -07:00
da68537ae6
HBASE-17786 Create LoadBalancer perf-test tool to benchmark Load Balancer algorithm performance
...
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
2017-05-12 11:22:15 -05:00
b34ab5980e
HBASE-17887 Row-level consistency is broken for read
2017-05-12 19:45:07 +08:00
5e046151d6
HBASE-11013: Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions - revert, pending work in snapshot descriptor
2017-05-11 18:53:14 -07:00
b3dcfb659e
HBASE-17928 Shell tool to clear compaction queues (Guangxu Cheng)
2017-05-11 18:47:12 -07:00
d8d4ba7c59
HBASE-17928 Shell tool to clear compaction queues - revert pending work in snapshot descriptor
2017-05-11 18:43:59 -07:00
c833473619
Amend HBASE-18000 Make sure we always return the scanner id with ScanResponse (Karan Mehta and Duo Zhang)
...
Move getRegionScanner() call below builder.setScannerId() to handle a
corner case.
2017-05-11 14:35:23 -07:00
0ae0edcd63
HBASE-17917 Use pread by default for all user scan
2017-05-11 11:28:22 +08:00
c5cc81d8e3
HBASE-17343 Changing the default type of CompactingMemStore to be BASIC
2017-05-10 09:02:26 +03:00
3bcc475ee2
HBASE-8758 Error in RegionCoprocessorHost class preScanner method documentation
...
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-05-10 11:35:45 +08:00
815b0f853b
HBASE-17928 Shell tool to clear compaction queues (Guangxu Cheng)
2017-05-09 18:32:38 -07:00
c38bf12444
HBASE-18017 Reduce frequency of setStoragePolicy failure warnings
2017-05-09 12:08:05 -07:00
951b23a44c
HBASE-11013: Clone Snapshots on Secure Cluster Should provide option to apply Retained User Permissions
...
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-05-09 09:32:48 -07:00
51d4c68b7c
HBASE-18009 Move RpcServer.Call to a separated file
2017-05-09 14:56:03 +08:00
959deb0e5c
HBASE-17924 Consider sorting the row order when processing multi() ops before taking rowlocks
...
(Allan Yang)
2017-05-08 16:23:13 -07:00
3dc38a4ff1
HBASE-18015 Storage class aware block placement for procedure v2 WALs
2017-05-08 14:11:27 -07:00
e99ed99bee
HBASE-18000 Make sure we always return the scanner id with ScanResponse
...
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-05-05 11:38:34 -07:00
2026540ea3
HBASE-17667: Implement async flush/compact region methods
...
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2017-05-05 21:12:49 +08:00
01af27061e
HBASE-17990 Refactor TestFSUtils to use Before
...
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-05-04 14:40:40 -07:00
0fb1fd1200
HBASE-17964 ensure hbase-metrics-api is included in mapreduce job classpaths
...
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2017-05-04 10:39:14 -05:00
52119200bc
HBASE-17991 Add more details about compaction queue on /dump (Guangxu Cheng)
2017-05-04 08:24:58 -07:00
2de6b051f6
HBASE-16466 Snapshots support in VerifyReplication tool
...
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-05-03 14:22:18 -07:00
78f6799f4d
Addendum patch for HBASE-12870. Added check for null pointer.
...
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-05-03 14:03:16 -07:00
ff998ef74f
HBASE-17867: Implement async procedure RPC API(list/exec/abort/isFinished)
...
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2017-05-03 21:46:37 +08:00
91995749c2
HBASE-17958 Avoid passing unexpected cell to ScanQueryMatcher when optimize SEEK to SKIP
2017-05-03 21:28:43 +08:00
4dbd025ccf
HBASE-17263 Netty based rpc server impl
...
Signed-off-by: zhangduo <zhangduo@apache.org>
2017-05-03 17:23:26 +08:00
30d0c3fdde
HBASE-12870 "Major compaction triggered" and "Skipping major compaction" messages lack the region information
...
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-05-02 10:54:15 -07:00
17007685c1
HBASE-13074 Cleaned up usage of hbase.master.lease.thread.wakefrequency
...
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-05-02 23:12:02 +08:00
c8a7e80e0a
HBASE-14286 Correct typo in argument name for WALSplitter.writeRegionSequenceIdFile
...
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-05-02 10:01:55 +08:00
ef94de3eb7
HBASE-17957 Custom metrics of replicate endpoints don't prepend "source." to global metrics
...
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-05-01 08:11:59 -07:00
1848353fd6
HBASE-17879: Avoid NPE in snapshot.jsp when accessing without any request parameter
...
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-04-29 10:51:46 +08:00
ba12cdf138
HBASE-17975 TokenUtil should throw remote exception rather than squash it.
...
Signed-off-by: Josh Elser <elserj@apache.org>
Signed-off-by: Ted Yu <tedyu@apache.org>
Signed-off-by: Umesh Agashe <uagashe@cloudera.com>
2017-04-28 13:19:33 -05:00
73d80bb416
HBASE-17970 Set yarn.app.mapreduce.am.staging-dir when starting MiniMRCluster
...
Signed-off-by: Sean Busbey <busbey@apache.org>
2017-04-28 12:37:00 -05:00
03e8f6b56e
HBASE-17835 Spelling mistakes in the Java source
...
Signed-off-by: Sean Busbey <busbey@apache.org>
2017-04-28 11:25:57 -05:00
43f3fccb7b
HBASE-17920 TestFSHDFSUtils always fails against hadoop 3.0.0-alpha2
...
Signed-off-by: Sean Busbey <busbey@apache.org>
2017-04-28 11:25:14 -05:00
635c9db815
HBASE-17817 add table name to output (if available) when removing coprocessors
...
Amending-Author: Sean Busbey <busbey@apache.org>
Signed-off-by: Sean Busbey <busbey@apache.org>
2017-04-28 11:24:00 -05:00
5411d3ecb1
HBASE-17972 Remove mergePool from CompactSplitThread (Guangxu Cheng)
2017-04-28 06:52:10 -07:00
b401a35fdc
HBASE-17950 Write the chunkId also as Int instead of long into the first
...
byte of the chunk (Ram)
2017-04-28 14:44:46 +05:30
c4cbb419a3
HBASE-17757 Unify blocksize after encoding to decrease memory fragment
...
Signed-off-by: anoopsamjohn <anoopsamjohn@gmail.com>
2017-04-28 07:25:36 +05:30
4bc0eb31c3
HBASE-17865: Implement async listSnapshot/deleteSnapshot methods.
...
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2017-04-28 09:09:13 +08:00
880db3eee4
HBASE-17965 Canary tool should print the regionserver name on failure
...
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-04-26 18:30:13 -07:00
6bad35e728
HBASE-16942 Add FavoredStochasticLoadBalancer and FN Candidate generators
...
Signed-off-by: Francis Liu <toffer@apache.org>
2017-04-26 18:11:45 -07:00
053e61541e
HBASE-15583 Any HTableDescriptor we give out should be immutable
2017-04-27 03:22:29 +08:00
8973582bc6
HBASE-17956 Raw scan should ignore TTL
2017-04-26 15:03:53 +08:00
2557506415
HBASE-15143 Procedure v2 - Web UI displaying queues
...
Signed-off-by: Michael Stack <stack@apache.org>
2017-04-25 09:39:28 -07:00
72fac37981
HBASE-17952 The new options for PE tool do not work
2017-04-25 09:34:49 +08:00
435104af70
HBASE-17302 The region flush request disappeared from flushQueue - addendum (Guangxu Cheng)
2017-04-23 07:37:34 -07:00
9053ec6fe6
HBASE-17943 Addendum increases the threshold value of in-memory compaction for TestWalAndCompactingMemStoreFlush
2017-04-22 20:47:55 +08:00
e95cf479c7
HBASE-16314 Retry on table snapshot failure during full backup (Vladimir Rodionov)
2017-04-21 16:15:07 -07:00
d39f40e787
HBASE-17864: Implement async snapshot/cloneSnapshot/restoreSnapshot methods
...
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2017-04-21 18:57:43 +08:00
33dadc1a94
HBASE-17941 CellArrayMap#getCell may throw IndexOutOfBoundsException
...
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-04-21 11:35:39 +08:00
ea3a27b18d
HBASE-17943 The in-memory flush size is different for each CompactingMemStore located in the same region
2017-04-21 11:20:20 +08:00
49cba2c237
HBASE-17937 Memstore size becomes negative in case of expensive postPut/Delete Coprocessor call
...
Signed-off-by: zhangduo <zhangduo@apache.org>
2017-04-21 08:50:09 +08:00
40cc666ac9
HBASE-17915 Implement async replication admin methods
2017-04-20 18:13:03 +08:00
b1ef8dd43a
HBASE-17542 Move backup system table into separate namespace
2017-04-20 02:57:24 -07:00
0953c14470
HBASE-17940 HMaster can not start due to Jasper related classes conflict
2017-04-20 16:06:50 +08:00
3acd8e4644
HBASE-17929 addendum fix typo and modify printUsage
2017-04-19 21:03:09 +08:00
972e8c8c29
HBASE-16438 Create a cell type so that chunk id is embedded in it (Ram)
2017-04-19 15:28:03 +05:30
6e962d6f65
HBASE-17929 addendum add scan read type option
2017-04-19 11:58:44 +08:00
66b616d7a3
HBASE-17914 Create a new reader instead of cloning a new StoreFile when compaction
2017-04-19 09:26:33 +08:00
75d1e0361a
HBASE-17930 Avoid using Canary.sniff in HBaseTestingUtility
2017-04-18 18:07:25 +08:00
5eda5fb9d7
HBASE-17912 - Avoid major compactions on region server startup
...
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-04-17 19:41:19 -07:00
3c32032f5c
HBASE-17929 Add more options for PE tool
2017-04-18 09:52:34 +08:00
ecdfb82326
Revert "HBASE-16438 Create a cell type so that chunk id is embedded in it (Ram)"
...
This reverts commit c2c2178b2e
2017-04-18 00:00:12 +05:30
c2c2178b2e
HBASE-16438 Create a cell type so that chunk id is embedded in it (Ram)
2017-04-17 09:28:24 +05:30
c1ac3f7739
HBASE-17366 Run TestHFile#testReaderWithoutBlockCache failes
...
Signed-off-by: CHIA-PING TSAI <chia7712@gmail.com>
2017-04-17 10:34:17 +08:00
8db9760363
HBASE-17866: Implement async setQuota/getQuota methods
...
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2017-04-17 09:49:30 +08:00
7678855fac
HBASE-17904 Get runs into NoSuchElementException when using Read Replica, with hbase. ipc.client.specificThreadForWriting
...
to be true and hbase.rpc.client.impl to be org.apache.hadoop.hbase.ipc.RpcClientImpl (Huaxiang Sun)
2017-04-16 11:01:06 -07:00
c8461456d0
HBASE-17888: Added generic methods for updating metrics on submit and finish of a procedure execution
...
Signed-off-by: Michael Stack <stack@apache.org>
2017-04-14 11:51:08 -07:00
da5fb27eab
HBASE-16775 Fix flaky TestExportSnapshot#testExportRetry.
...
Reason for flakyness: Current test is probability based fault injection and triggers failure 3% of the time. Earlier when test used LocalJobRunner which didn't honor "mapreduce.map.maxattempts", it'd pass 97% time (when no fault is injected) and fail 3% time (when fault was injected). Point being, even when the test was complete wrong, we couldn't catch it because it was probability based.
This change will inject fault in a deterministic manner.
On design side, it encapsulates all testing hooks in ExportSnapshot.java into single inner class.
Change-Id: Icba866e1d56a5281748df89f4dd374bc45bad249
2017-04-12 11:11:31 -07:00
cf3215d343
HBASE-17897 StripeStoreFileManager#nonOpenRowCompare use the wrong comparison function
2017-04-12 11:54:42 +08:00
3aadc675b0
HBASE-17896 The FIXED_OVERHEAD of Segment is incorrect
2017-04-12 11:26:43 +08:00
Josh Elser
tedyu
Apekshit Sharma
Guanghao Zhang
zhangduo
anastas
Umesh Agashe
Jingcheng Du
huzheng
Chia-Ping Tsai
anoopsamjohn
Andrew Purtell
Roman Nikitchenko
Zach York
Sean Busbey
Sukumar Maddineni
Chinmay Kulkarni
binlijin
Jan Hentschel
Gábor Lipták
yoshida.shinya
Abhishek Kumar
QilinCao
Jonathan M Hsieh
Steen Manniche
Ramkrishna
Allan Yang
Karan Mehta
Thiruvel Thirumoolan
Balazs Meszaros
s9514171
Abhishek Singh Chouhan
gjacoby
huaxiang sun
Michael Stack