Commit Graph

12134 Commits

Author SHA1 Message Date
stack 407aa4d496 Revert "HBASE-15994 Allow selection of RpcSchedulers Adds logging by the RpcExecutors of their run configs Changes the default RpcSchedulerFactory from SimpleRpcSchedulerFactory.class to RpcSchedulerFactoryImpl.class. RpcSchedulerFactoryImpl.class. defaults to using SimpleRpcSchedulerFactory.class and the SimpleRpcScheduler, as has been default up to this, unless you set "hbase.region.server.rpc.scheduler.class" to org.apache.hadoop.hbase.ipc.FifoRpcScheduler"
This reverts commit 3ac4a57fd2.
2016-06-08 20:22:50 -07:00
stack 3ac4a57fd2 HBASE-15994 Allow selection of RpcSchedulers Adds logging by the RpcExecutors of their run configs Changes the default RpcSchedulerFactory from SimpleRpcSchedulerFactory.class to RpcSchedulerFactoryImpl.class. RpcSchedulerFactoryImpl.class. defaults to using SimpleRpcSchedulerFactory.class and the SimpleRpcScheduler, as has been default up to this, unless you set "hbase.region.server.rpc.scheduler.class" to org.apache.hadoop.hbase.ipc.FifoRpcScheduler
Signed-off-by: stack <stack@apache.org>
2016-06-08 20:20:50 -07:00
Jonathan M Hsieh a7172d5611 HBASE-15959 Fix flaky test TestRegionServerMetrics.testMobMetrics (Huaxiang Sun and Jingcheng Du) 2016-06-08 19:07:13 -07:00
Matteo Bertozzi d9463bcce0 HBASE-15989 Remove hbase.online.schema.update.enable 2016-06-08 13:09:31 -07:00
Matteo Bertozzi d5d9b7d500 HBASE-15107 Procedure v2 - Procedure Queue with Region locks 2016-06-08 12:52:58 -07:00
Enis Soztutar d05a3722c8 HBASE-15600 Add provision for adding mutations to memstore or able to write to same region in batchMutate coprocessor hooks (Rajeshbabu and Enis) 2016-06-07 20:12:51 -07:00
stack 3a95552cfe HBASE-15948 Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers" Adds HADOOP-9955 RPC idle connection closing is extremely inefficient
Changes how we do accounting of Connections to match how it is done in Hadoop.
Adds a ConnectionManager class. Adds new configurations for this new class.

"hbase.ipc.client.idlethreshold" 4000
"hbase.ipc.client.connection.idle-scan-interval.ms" 10000
"hbase.ipc.client.connection.maxidletime" 10000
"hbase.ipc.client.kill.max", 10
"hbase.ipc.server.handler.queue.size", 100

The new scheme does away with synchronization that purportedly would freeze out
reads while we were cleaning up stale connections (according to HADOOP-9955)

Also adds in new mechanism for accepting Connections by pulling in as many
as we can at a time adding them to a Queue instead of doing one at a time.
Can help when bursty traffic according to HADOOP-9956. Removes a blocking
while Reader is busy parsing a request. Adds configuration
"hbase.ipc.server.read.connection-queue.size" with default of 100 for
queue size.

Signed-off-by: stack <stack@apache.org>
2016-06-07 16:42:21 -07:00
stack e66ecd7db6 Revert "HBASE-15948 Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers""
Revert mistaken commit...
This reverts commit e0b70c00e7.
2016-06-07 16:41:30 -07:00
stack 6d5a25935e Revert "HBASE-15967 Metric for active ipc Readers and make default fraction of cpu count"
Revert mistaken commit
This reverts commit 1125215aad.
2016-06-07 16:41:01 -07:00
Bryan Beaudreault 2da090f9a3 HBASE-15981 Remove references to disabling table in docs around stripe and date-tiered compactions
Signed-off-by: stack <stack@apache.org>
2016-06-07 13:10:35 -07:00
stack 1125215aad HBASE-15967 Metric for active ipc Readers and make default fraction of cpu count
Add new metric hbase.regionserver.ipc.runningReaders
Also make it so Reader count is a factor of processor count
2016-06-07 13:10:14 -07:00
stack e0b70c00e7 HBASE-15948 Port "HADOOP-9956 RPC listener inefficiently assigns connections to readers"
Adds HADOOP-9955 RPC idle connection closing is extremely inefficient
Then removes queue added by HADOOP-9956 at Enis suggestion

    Changes how we do accounting of Connections to match how it is done in Hadoop.
    Adds a ConnectionManager class. Adds new configurations for this new class.

    "hbase.ipc.client.idlethreshold" 4000
    "hbase.ipc.client.connection.idle-scan-interval.ms" 10000
    "hbase.ipc.client.connection.maxidletime" 10000
    "hbase.ipc.client.kill.max", 10
    "hbase.ipc.server.handler.queue.size", 100

    The new scheme does away with synchronization that purportedly would freeze out
    reads while we were cleaning up stale connections (according to HADOOP-9955)

    Also adds in new mechanism for accepting Connections by pulling in as many
    as we can at a time adding them to a Queue instead of doing one at a time.
    Can help when bursty traffic according to HADOOP-9956. Removes a blocking
    while Reader is busy parsing a request. Adds configuration
    "hbase.ipc.server.read.connection-queue.size" with default of 100 for
    queue size.
2016-06-07 13:10:14 -07:00
Sergey Soldatov da88b48240 HBASE-15957 RpcClientImpl.close never ends in some circumstances
Signed-off-by: Enis Soztutar <enis@apache.org>
2016-06-07 11:33:03 -07:00
Andrew Purtell 376ad0d986 HBASE-15698 Increment TimeRange not serialized to server (Ted Yu) 2016-06-06 22:17:00 -07:00
tedyu 7fd3532de6 HBASE-15803 ZooKeeperWatcher's constructor can leak a ZooKeeper instance with throwing ZooKeeperConnectionException when canCreateBaseZNode is true 2016-06-06 18:35:15 -07:00
Apekshit Sharma 15c03fd1c9 HBASE-15965
- Testing by executing a command will cover the exact path users will trigger, so its better then directly calling library functions in tests. Changing the tests to use @shell.command(:<command>, args) to execute them like it's a command coming from shell.

Norm change:
Commands should print the output user would like to see, but in the end, should also return the relevant value. This way:
- Tests can use returned value to check that functionality works
- Tests can capture stdout to assert particular kind of output user should see.
- We do not print the return value in interactive mode and keep the output clean. See Shell.command() function.

Bugs found due to this change:
- Uncovered bug in major_compact.rb with this approach. It was calling admin.majorCompact() which doesn't exist but our tests didn't catch it since they directly tested admin.major_compact()
- Enabled TestReplicationShell. If it's bad, flaky infra will take care of it.

Change-Id: I5d8af16bf477a79a2f526a5bf11c245b02b7d276
2016-06-06 17:50:22 -07:00
Enis Soztutar 3d7840a173 HBASE-15954 REST server should log requests with TRACE instead of DEBUG 2016-06-06 10:58:37 -07:00
Ramkrishna b21c56e795 HBASE-15174 Client Public API should not have PB objects in 2.0 (Ram) 2016-06-06 10:11:38 +05:30
Apekshit Sharma 70762faa98 HBASE-15849 Simplify the way we handle runtime of commands.
Functions format_simple_command and format_and_return_simple_command are used to print runtimes right now. They are called from within every single command and use Ruby's 'yield' magic.  Instead, we can simplify it using 'command_safe' function. Since command_safe wraps all commands, we can simply time before and after we call individual command.
If a command only wants to time a part of its logic, it can set instance variables start_time and end_time accordingly which is far more simpler to understand and work with than 'yield'.

Change-Id: Ibfacf3593175af22fc4f7d80896dd2f6d7c5dde3
2016-06-03 15:50:03 -07:00
Stephen Yuan Jiang de1b5ff776 HBASE-15955 Disable action in CatalogJanitor#setEnabled should wait for active cleanup scan to finish (Stephen Yuan Jiang) 2016-06-03 15:49:23 -07:00
Joseph Hwang 21e98271c3 HBASE-15883 Adding WAL files and tracking offsets in HBase.
Implemented ReplicationQueuesHBaseImpl that tracks WAL offsets and replication queues in an HBase table.
Only wrote the basic tracking methods, have not implemented claimQueue() or HFileRef methods yet.
Wrote a basic unit test for ReplicationQueueHBaseImpl that tests the implemented functions on a single Region Server

Signed-off-by: Elliott Clark <elliott@fb.com>
Signed-off-by: Elliott Clark <eclark@apache.org>
2016-06-03 15:23:10 -07:00
Apekshit 9a53d8b385 HBASE-15929 There are two tests named TestRegionServerMetrics. This causes slight issues when parsing console output for hanging tests and flaky tests analysis. Moving .../hbase/TestRegionServerMetrics.java to .../hbase/regionserver/TestRegionServerReadRequestMetrics.java. (Apekshit)
Change-Id: I379c15fe2c2c01bed53bddf7619d5f2a07c5640e
2016-06-03 15:13:49 -07:00
Apekshit b557f0bec6 HBASE-15949 Cleanup TestRegionServerMetrics.
@Before and @After to setup/teardown tables using @Rule to set table name based on testname.
Refactor out copy-pasted code fragments to single function.
(Apekshit)

Change-Id: Ic22e5027cc3952bab5ec30070ed20e98017db65a
2016-06-03 14:27:24 -07:00
Apekshit bdb46f01b9 HBASE-15845 Changes:
- Renaming hbase.rb to hbase_constants.rb because there are two hbase.rb files right now which is confusing.
- Remove omnipresence of formatter object since it is kind of a use-and-throw class. Commands should create
  an instance, use it to format the output and discard it.
- Some refactoring

Change-Id: If9ea9873904e0a39d199a6aa10e23864b86a2f09
2016-06-03 13:37:38 -07:00
Matteo Bertozzi f0c159b5fe HBASE-15927 Remove HMaster.assignRegion() 2016-06-03 12:36:25 -07:00
Ashish Singhi 0cbce07626 HBASE-15888 Extend HBASE-12769 for bulk load data replication 2016-06-03 18:42:00 +05:30
tedyu 72d3f2a868 HBASE-15939 Two shell test failures on master (Talat and Ted) 2016-06-02 15:44:18 -07:00
Lars Hofhansl fc890a2ecb HBASE-15881 Allow BZIP2 compression. 2016-06-02 12:19:02 -07:00
tedyu 7e5d530870 HBASE-15727 Canary Tool for Zookeeper (churro morales) 2016-06-02 10:15:08 -07:00
tedyu cd2588001c HBASE-15933 Addendum - make merge decision when sizes of both regions are known 2016-06-02 09:30:36 -07:00
Apekshit 9593a9f396 HBASE-15938 submit-patch.py: Don't crash if there are tests with same name. Refactor: Split out flaky dashboard html template to separate file. (Apekshit)
Change-Id: Ie5875bdefbf886984a57dfc85661be2ac9592a7b

Signed-off-by: stack <stack@apache.org>
2016-06-02 08:55:42 -07:00
Apekshit 4ffea7711a HBASE-15944 Spark test flooding mvn output. Redirect test logs to file. This doesn't fix the problem fully as I still see few logs being dumped in stdout. But it cleans up majority of the earlier dump. (Apekshit)
Change-Id: I6893301d154078a7cfb6b9af2eedc744deafb8d7

Signed-off-by: stack <stack@apache.org>
2016-06-02 08:44:08 -07:00
tedyu cfe868d56e HBASE-15933 NullPointerException may be thrown from SimpleRegionNormalizer#getRegionSize() 2016-06-02 01:55:28 -07:00
tedyu a0f49c9884 HBASE-15858 Some region server group shell commands don't work 2016-06-01 21:13:33 -07:00
Yu Li 53eb27bb60 HBASE-15931 Add log for long-running tasks in AsyncProcess 2016-06-02 12:00:42 +08:00
tedyu cbb95cd3a9 HBASE-15932 Shell test fails due to uninitialized constant 2016-06-01 10:35:00 -07:00
tedyu 015f2ef629 HBASE-15923 Shell rows counter test fails 2016-05-31 14:21:32 -07:00
Ronan Stokes 73ec33856d HBASE-15907 updates for HBase Shell pre-splitting docs
(cherry picked from commit 01adec574d9ccbdd6183466cb8ee6b43935d69ca)
2016-05-31 13:52:46 -07:00
Apekshit eb64cd9dd1 HBASE-15917 Addendum. Fix bug in report-flakies.py where hanging tests are not being added to flaky list. (Apekshit)
ADDENDUM !

Change-Id: I9c55932d0f9e65b72ec8d3ae714144536b2bfe0a

Signed-off-by: stack <stack@apache.org>
2016-05-31 10:16:40 -07:00
stack c80e232642 Remove the hbasecon banner logo 2016-05-31 10:14:40 -07:00
Apekshit 5ea2f09233 HBASE-15919 Modify docs to change from @Rule to @ClassRule. Also clarify that timeout limits are on test case level. (Apekshit)
Change-Id: Ifcd0264ea147bcb1100db74d92da95b643f4793f

Signed-off-by: stack <stack@apache.org>
2016-05-31 10:12:00 -07:00
Apekshit 75c2360543 HBASE-15918 Cleanup excludes/includes file after use in hbase-personality.sh to avoid asf license error. (Apekshit)
Change-Id: I3a47113ef9f598f90357b7cb580cbdc56f3c46f4

Signed-off-by: stack <stack@apache.org>
2016-05-30 21:30:19 -07:00
Apekshit f0a1e22441 HBASE-15917 Addendum. Add jija2 to python requirements. It's used by report-flakies.py to build the dashboard. (Apekshit)
ADDENDUM!!!

Change-Id: I8d30986e204463d1ea26b4a70e2f26915c99bdfc

Signed-off-by: stack <stack@apache.org>
2016-05-30 21:16:33 -07:00
Apekshit 1a27278624 HBASE-15915 Set timeouts on hanging tests. Tests run: 8, Time elapsed: 94.191 sec - in org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures Tests run: 103, Time elapsed: 69.057 sec - in org.apache.hadoop.hbase.regionserver.TestHRegion Tests run: 103, Time elapsed: 67.957 sec - in org.apache.hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush Tests run: 5, Time elapsed: 34.629 sec - in org.apache.hadoop.hbase.regionserver.TestRegionMergeTransactionOnCluster Tests run: 9, Time elapsed: 82.913 sec - in org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient Tests run: 9, Time elapsed: 80.458 sec - in org.apache.hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient (Apekshit)
Change-Id: Ia7b986ca6276ff5498f588c0b1b9c570e2a9d798

Signed-off-by: stack <stack@apache.org>
2016-05-30 21:13:55 -07:00
Apekshit 0923346b61 HBASE-15917 Flaky tests dashboard. (Apekshit)
Change-Id: If0e4299b173d3d76e0bdb12a3de080e5b51d8748

Signed-off-by: stack <stack@apache.org>
2016-05-30 20:54:01 -07:00
Daniel Vimont e6d613de70 HBASE-15835 - Prevent HBaseTestingUtility#startMiniCluster from throwing _HMasterAddress already in use_ RuntimeException when another HBase thread is running
Signed-off-by: stack <stack@apache.org>
2016-05-30 20:05:19 -07:00
Jurriaan Mous a1f0c1cbb3 HBASE-15875 Remove HTable references and HTableInterface
Signed-off-by: stack <stack@apache.org>
2016-05-30 19:56:54 -07:00
Youngjoon Kim 47176049f8 HBASE-15897 Fix a wrong comment about QOS order
Signed-off-by: stack <stack@apache.org>
2016-05-29 14:35:35 -07:00
Jurriaan Mous cdd532da8a HBASE-15610 Remove deprecated HConnection for 2.0 thus removing all PB references for 2.0
Signed-off-by: stack <stack@apache.org>
2016-05-29 07:50:55 -07:00
Apekshit 74442fde0f HBASE-15909 Use Yetus' patch naming rules in submit-patch.py. (Apekshit)
Change-Id: I4d71c94da550efe8b24ea73edcde73d9bba84b8a

Signed-off-by: stack <stack@apache.org>
2016-05-28 21:55:17 -07:00