Issues incorporated and incompatible changes called out on the jira.
* Cleaned up references to HLog
* Deprecates HLogKey but maintains it for compatibility
- Moves all Writeable from WALKey to HLogKey
* Adds utility code to CoprocessorHost to help with evolving Coprocessor APIs
* RSRpcServices roll WAL call now requests the non-meta LogRoller roll all logs
- rolls actually happen asynchronously
- deprecated old api (and noted incompatible behavior change)
- modified api in new Admin interface to reflect lack of return values.
* Moved WAL user facing API to "WAL"
- only 1 sync offered
- WALTrailer removed from API
* make provider used by the WALFactory configurable.
* Move all WAL requests to use opaque ids instead of paths
* WALProvider provides API details for implementers and handles creation of WALs.
* Refactor WALActionsListener to have a basic implementation.
* turn MetricsWAL into a WALActionsListener.
* tests that needs FSHLog implementation details use them directly, others just reference provider + factory
- Some tests moved from Large to Medium based on run time.
* pull out wal disabling into its own no-op class
* update region open to delegate to WALFactory
* update performance test tool to allow for multiple regions
* Removed references to meta-specific wals within wal code
- replaced with generic suffixes
- WALFactory maintains a dedicated WALProvider for meta (and so knows about the distinction)
* maintain backwards compat on HLogPrettyPrinter and mark it deprecated.
- made WALPrettyPrinter IA.Private in favor of `bin/hbase wal`
* move WALUtil stuff that's implementation specific to said implementation
- WALUtil now acts as an integration point between the RegionServer and hte WAL code.
Incorporates contributions from v.himanshu and stack.
Conflicts:
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
hbase-server/src/main/java/org/apache/hadoop/hbase/migration/NamespaceUpgrade.java
hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ReplicationProtbufUtil.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java
hbase-server/src/main/java/org/apache/hadoop/hbase/util/HMerge.java
hbase-server/src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin2.java
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverScannerOpenHook.java
hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java
hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestInvocationRecordFilter.java
hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorder.java
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransaction.java
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollAbort.java
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMultiSlaveReplication.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationWALEntryFilters.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpointNoMaster.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationWALReaderManager.java
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSVisitor.java
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestSecureWAL.java
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALFiltering.java
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALMethods.java
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALReaderOnSecureWAL.java
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplitCompressed.java
src/main/docbkx/ops_mgt.xml
Remove deleteConnection from everywhere except the two tests that ensure
it does the right thing and from HCM and CM themselves.
Undoes deleteConnection from tests and from the web ui
HConnection#getTable (0.98, 0.99)
Backport
Replaced HTable under hbase-*/src/main/java. Skipped tests. Would take
till end of time to do all and some cases are cryptic. Also skipped
some mapreduce where HTable comes through in API. Can do both of
these stragglers in another issue.
Generally, if a utility class or standalone class, tried to pass in a
Connection rather than have the utility or standalone create its own
connection on each invocation; e.g. the Quota stuff. Where not possible,
noted where invocation comes from... if test or hbck, didn't worry about
it.
Some classes are just standalone and nothing to be done to avoid
a Connection setup per invocation (this is probably how it worked
in the new HTable...days anyways). Some classes are not used:
AggregationClient, FavoredNodes... we should just purge this stuff.
Doc on what short circuit connection does (I can just use it...
I thought it was just for short circuit but no, it switches dependent
on where you are connecting).
Changed HConnection to super Interface ClusterConnection where safe (
internal usage by private classes only).
Doc cleanup in example usage so we do new mode rather than the old
fashion.
Used java7 idiom that allows you avoid writing out finally to call close
on implementations of Closeable.
Added a RegistryFactory.. moved it out from being inner class.
Added a utility createGetClosestRowOrBeforeReverseScan method to Scan
to create a Scan that can ...
Renamed getShortCircuitConnection as getConnection – users don't need
to know what implementation does (that it can short-circuit RPC).
The old name gave pause. I was frightened to use it thinking it only
for short-circuit reading – that it would not do remote too.
Squashed commit of the following:
Move from HConnection to ClusterConnection or Connection
Use unmanaged connections where we use managed previous
(used the jdk7 https://docs.oracle.com/javase/7/docs/technotes/guides/language/try-with-resources.html idiom).
In ZKConfig, synchronize on Configuration rather than make a copy.
Making a copy we were dropping hbase configs in certain test context
(could not find the zk ensemble because default port).
In tests, some move to the new style connection setup but mostly
fixes for premature connection close or adding cleanup where it
was lacking.
Conflicts:
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMultiSlaveReplication.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java
Summary: If there are bugs in HDFS move and/or create we should protect against them by making sure that all files referenced end up in split daughters.
Test Plan: Unit tests cover splits pretty well
Subscribers: matteobertozzi
Differential Revision: https://reviews.facebook.net/D29373
Summary: This diff ports the Preemptive Fast Fail feature to OSS. In multi threaded clients, we use a feature developed on 0.89-fb branch called Preemptive Fast Fail. This allows the client threads which would potentially fail, fail fast. The idea behind this feature is that we allow, among the hundreds of client threads, one thread to try and establish connection with the regionserver and if that succeeds, we mark it as a live node again. Meanwhile, other threads which are trying to establish connection to the same server would ideally go into the timeouts which is effectively unfruitful. We can in those cases return appropriate exceptions to those clients instead of letting them retry.
Test Plan: Unit tests
Differential Revision: https://reviews.facebook.net/D24177
Signed-off-by: stack <stack@apache.org>
Conflicts:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClusterConnection.java
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionManager.java
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCaller.java
Summary: This is a follow up diff for the jira to add the functionality to update configs in an online fashion.
Test Plan: Unit test. This adds an end point on the regionserver. It just tries to call the end point and make sure the call succeeds.
Differential Revision: https://reviews.facebook.net/D24861
Signed-off-by: stack <stack@apache.org>
Conflicts:
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerOnlineConfigChange.java
* Implemented the waitUntilAllRegionsOnline in HBaseTestingUtility#createTable
* Add waitFor around #isTableAvailable call that previously didn't do anything
* Remove unused byte[] hex
Signed-off-by: stack <stack@apache.org>
Summary: This diff is intended to forward port HBASE-8805 and HBASE-8544 implemented by Gaurav Menghani in 89-fb. This improves operational efficiency in managing clusters that are serving production traffic. The idea is to have a central configuration which can manage notifying the configuration observers. The observers in turn should update their local state from the latest config. Minor caveats where configuration variables are corelated should be taken care of with additional care.
Test Plan: Unit tests
Differential Revision: https://reviews.facebook.net/D24681
Signed-off-by: stack <stack@apache.org>
Summary: # Clear the cache of the server when failed
Test Plan: Add testcase `TestHTableMultiplexerFlushCache` to reproduce the bug.
Differential Revision: https://reviews.facebook.net/D24603
Signed-off-by: Elliott Clark <elliott@fb.com>
Signed-off-by: Elliott Clark <eclark@apache.org>
stack
Elliott Clark
Sean Busbey
Enis Soztutar
Jeffrey Zhong
Ramkrishna
Elliott Clark
Andrew Purtell
tedyu
anoopsjohn
Sean Busbey
Nick Dimiduk
Matteo Bertozzi
Lars Hofhansl
Esteban Gutierrez
Niels Basjes
Adrian Muraru
Ashish Singhi
manukranthk
Dima Spivak
Ted Yu
Yi Deng
anoopsjohn
Patrick White
Nicolas Liochon
Rajeshbabu Chintaguntla
Jimmy Xiang