Summary:
Intgration tests don't currently have a lot of splits going on while there is IO.
This changes that by changing the split aglorithm and the max region hfile size. That should make things split more.
Additionally this allows ITBLL to start with just one region if hbase.test.pre-split-table is False.
Test Plan: Test on a cluster with ITBLL slow determinitic monkey and stress am monkey.
Differential Revision: https://reviews.facebook.net/D44181
* fix hbase-it's incorrect use of hbase-server's unfiltered NOTICE
* fix printed comment in resource-bundle NOTICE
* add dependency url for non-aggregated works so that we meet Category B requirements
* corrects license/notice for source distribution
* adds inception year to correct copyright in generated NOTICE files for jars
* updates project names in poms to use "Apache HBase" instead of "HBase" so jar NOTICE files will be correct
* uses append-resources to include supplemental info on jars with 3rd party works in source
* adds an hbase specific resource bundle for jars that include 3rd party works for binaries
** uses supplemental-model to fill in license gaps
** uses the above and a shade plugin transformation to build proper files for shaded jars.
** uses the above and the assembly plugin to build the proper files for bin assembly
* adds a NOTICE item for things copied out of Hadoop (TODO legal-discuss)
Conflicts:
NOTICE.txt
hbase-checkstyle/pom.xml
hbase-resource-bundle/pom.xml
hbase-server/pom.xml
hbase-thrift/pom.xml
pom.xml
* IA.Public accessible logger instances deprecated
* logger instances modified by tests left in place
* all others made private static final
Signed-off-by: Sean Busbey <busbey@apache.org>
Conflicts:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientIdGenerator.java
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerImpl.java
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java
hbase-it/src/test/java/org/apache/hadoop/hbase/IntegrationTestIngestWithEncryption.java
hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/TestStatusResource.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
hbase-server/src/test/java/org/apache/hadoop/hbase/TestTableDescriptor.java
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestRowCounter.java
hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildHole.java
hbase-server/src/test/java/org/apache/hadoop/hbase/util/hbck/TestOfflineMetaRebuildOverlap.java
Summary: This test subclasses the BigLinkedList test. It takes two hbase clusters as arguments, sets up tables, sets up replication, and runs the BigLinkedList generator. The verification portion of the loop checks that the sink of the replication has the data and it is correct.
Test Plan: ran the test on my laptop and a small live cluster
Reviewers: dimaspivak, eclark
Reviewed By: eclark
Subscribers: srikanth235, asameet
Differential Revision: https://reviews.facebook.net/D36423
Signed-off-by: Elliott Clark <eclark@apache.org>
Adds a number of lifecycle-mapping entries which
prevent errors from showing up in Eclipse on a fresh
import of HBase. For plugins defined in the top-level
pom, the mapping is added there; otherwise, the mapping
is pushed down to the child pom.
Signed-off-by: Sean Busbey <busbey@apache.org>
In our pre-1.0 API, HTable is considered a light-weight object that consumed by
a single thread at a time. The HTablePool class provided a means of sharing
multiple HTable instances across a number of threads. As an optimization,
HTable managed a "write buffer", accumulating edits and sending a "batch" all
at once. By default the batch was sent as the last step in invocations of
put(Put) and put(List<Put>). The user could disable the automatic flushing of
the write buffer, retaining edits locally and only sending the whole "batch"
once the write buffer has filled or when the flushCommits() method in invoked
explicitly. Explicit or implicit batch writing was controlled by the
setAutoFlushTo(boolean) method. A value of true (the default) had the write
buffer flushed at the completion of a call to put(Put) or put(List<Put>). A
value of false allowed for explicit buffer management. HTable also exposed the
buffer to consumers via getWriteBuffer().
The combination of HTable with setAutoFlushTo(false) and the HTablePool
provided a convenient mechanism by which multiple "Put-producing" threads could
share a common write buffer. Both HTablePool and HTable are deprecated, and
they are officially replaced in The new 1.0 API by Table and BufferedMutator.
Table, which replaces HTable, no longer exposes explicit write-buffer
management. Instead, explicit buffer management is exposed via BufferedMutator.
BufferedMutator is made safe for concurrent use. Where code would previously
retrieve and return HTables from an HTablePool, now that code creates and
shares a single BufferedMutator instance across all threads.
Conflicts:
hbase-client/src/main/java/org/apache/hadoop/hbase/client/HTable.java
hbase-client/src/test/java/org/apache/hadoop/hbase/client/TestAsyncProcess.java
hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedList.java
hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestBigLinkedListWithVisibility.java
hbase-it/src/test/java/org/apache/hadoop/hbase/test/IntegrationTestLoadAndVerify.java
hbase-it/src/test/java/org/apache/hadoop/hbase/trace/IntegrationTestSendTraceRequests.java
hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/PerformanceEvaluation.java
hbase-rest/src/test/java/org/apache/hadoop/hbase/rest/client/TestRemoteTable.java
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/MultiTableOutputFormat.java
hbase-server/src/test/java/org/apache/hadoop/hbase/PerformanceEvaluation.java
hbase-server/src/test/java/org/apache/hadoop/hbase/TestMultiVersions.java
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestClientPushback.java
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestCloneSnapshotFromClient.java
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMultiParallel.java
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRpcControllerFactory.java
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMaster.java
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestFSErrorsExposed.java
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerMetrics.java
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestScannerWithBulkload.java
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRolling.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationChangingPeerRegionservers.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationWithTags.java
hbase-server/src/test/java/org/apache/hadoop/hbase/security/visibility/TestVisibilityLabelsReplication.java
hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java
hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestFlushSnapshotFromClient.java
hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestRestoreFlushSnapshotFromClient.java
Summary: The reason of this change is to make us write implementation of ClusterManager not using ssh/unix signals.
Test Plan: The compilation is OK.
Reviewers: eclark, manukranthk
Differential Revision: https://reviews.facebook.net/D30201
Signed-off-by: Elliott Clark <eclark@apache.org>
Issues incorporated and incompatible changes called out on the jira.
* Cleaned up references to HLog
* Deprecates HLogKey but maintains it for compatibility
- Moves all Writeable from WALKey to HLogKey
* Adds utility code to CoprocessorHost to help with evolving Coprocessor APIs
* RSRpcServices roll WAL call now requests the non-meta LogRoller roll all logs
- rolls actually happen asynchronously
- deprecated old api (and noted incompatible behavior change)
- modified api in new Admin interface to reflect lack of return values.
* Moved WAL user facing API to "WAL"
- only 1 sync offered
- WALTrailer removed from API
* make provider used by the WALFactory configurable.
* Move all WAL requests to use opaque ids instead of paths
* WALProvider provides API details for implementers and handles creation of WALs.
* Refactor WALActionsListener to have a basic implementation.
* turn MetricsWAL into a WALActionsListener.
* tests that needs FSHLog implementation details use them directly, others just reference provider + factory
- Some tests moved from Large to Medium based on run time.
* pull out wal disabling into its own no-op class
* update region open to delegate to WALFactory
* update performance test tool to allow for multiple regions
* Removed references to meta-specific wals within wal code
- replaced with generic suffixes
- WALFactory maintains a dedicated WALProvider for meta (and so knows about the distinction)
* maintain backwards compat on HLogPrettyPrinter and mark it deprecated.
- made WALPrettyPrinter IA.Private in favor of `bin/hbase wal`
* move WALUtil stuff that's implementation specific to said implementation
- WALUtil now acts as an integration point between the RegionServer and hte WAL code.
Incorporates contributions from v.himanshu and stack.
Conflicts:
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java
hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
hbase-server/src/main/java/org/apache/hadoop/hbase/migration/NamespaceUpgrade.java
hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ReplicationProtbufUtil.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LogRoller.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java
hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/RegionReplicaReplicationEndpoint.java
hbase-server/src/main/java/org/apache/hadoop/hbase/util/HMerge.java
hbase-server/src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java
hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALSplitter.java
hbase-server/src/test/java/org/apache/hadoop/hbase/TestIOFencing.java
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin2.java
hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverScannerOpenHook.java
hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestFilter.java
hbase-server/src/test/java/org/apache/hadoop/hbase/filter/TestInvocationRecordFilter.java
hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorder.java
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportExport.java
hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestAtomicOperation.java
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransaction.java
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollAbort.java
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestLogRollingNoCluster.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestMultiSlaveReplication.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationEndpoint.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationSource.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationWALEntryFilters.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpointNoMaster.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationSourceManager.java
hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestReplicationWALReaderManager.java
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestFSVisitor.java
hbase-server/src/test/java/org/apache/hadoop/hbase/util/TestMergeTool.java
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestSecureWAL.java
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALFiltering.java
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALMethods.java
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALReaderOnSecureWAL.java
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplit.java
hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALSplitCompressed.java
src/main/docbkx/ops_mgt.xml
chenheng
tedyu
Reid
Sean Busbey
Enis Soztutar
Gary Helmling
Joseph Hwang
Sergey Soldatov
Sean Mackrory
Mikhail Antonov
stack
Jonathan M Hsieh
Samir Ahmic
Elliott Clark
Andrew Purtell
Stephen Yuan Jiang
Srikanth Srungarapu
Sean Busbey
Dima Spivak
Matteo Bertozzi
Rajeshbabu Chintaguntla
Nick Dimiduk
Matt Warhaftig
Usha Kuchibhotla
Rajesh Nishtala
zhangduo
Josh Elser
Devaraj Das
Ramkrishna
Elliott Clark
Yi Deng