Commit Graph

1347 Commits

Author SHA1 Message Date
Mike Drob 468d4fcd73 HBASE-20282 Clean up tooling docs/help 2018-03-30 09:00:46 -07:00
Vasudevan 1687dc5ddb HBASE-20197 Review of ByteBufferWriterOutputStream.java (BELUGA BEHR)
Signed-off-by: Vasudevan <ramkrishna.s.vasudevan@gmail.com>
2018-03-30 12:20:09 +05:30
Yu Li b72e19e3b9 HBASE-20159 Support using separate ZK quorums for client 2018-03-29 03:20:52 +08:00
Josh Elser c3d82a283d HBASE-20223 Update to hbase-thirdparty 2.1.0
Remove commons-cli and commons-collections4 use. Account
for the newer internal protobuf version of 3.5.1.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Mike Drob <mdrob@apache.org>
2018-03-26 16:07:39 -04:00
anoopsamjohn b5881dbd3f HBASE-17819 Reduce the heap overhead for BucketCache. 2018-03-25 16:39:21 +05:30
Chia-Ping Tsai 6aba045aae HBASE-19504 Add TimeRange support into checkAndMutate
Signed-off-by: Michael Stack <stack@apache.org>
2018-03-24 00:05:22 +08:00
Chia-Ping Tsai dd9e46bbf5 HBASE-20212 Make all Public classes have InterfaceAudience category
Signed-off-by: tedyu <yuzhihong@gmail.com>
Signed-off-by: Michael Stack <stack@apache.org>
2018-03-22 18:09:54 +08:00
Chia-Ping Tsai 95596e8ba7 HBASE-20119 Introduce a pojo class to carry coprocessor information in order to make TableDescriptorBuilder accept multiple cp at once
Signed-off-by: Ted Yu <yuzhihong@gmail.com>
Signed-off-by: Michael Stack <stack@apache.org>
2018-03-16 01:26:08 +08:00
Chance Li 47b227b55b HBASE-19389 Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
Signed-off-by: Yu Li <liyu@apache.org>
2018-03-14 18:36:58 +08:00
Yu Li c8ecfc5461 Revert "HBASE-19389 Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted"
This reverts commit c6d89b494c.
2018-03-14 18:35:22 +08:00
Yu Li c6d89b494c HBASE-19389 Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted
Signed-off-by: Yu Li <liyu@apache.org>
2018-03-14 18:33:39 +08:00
anoopsamjohn 7bd39250e7 HBASE-20078 MultiByteBuff : bug in reading primitives when individual buffers are too small. 2018-03-14 12:10:22 +05:30
Chia-Ping Tsai a58f65212c
HBASE-20171 Remove o.a.h.h.ProcedureState 2018-03-12 09:26:25 -07:00
zhangduo 1de08ded58 HBASE-19543 Abstract a replication storage interface to extract the zk specific code 2018-03-09 20:55:48 +08:00
Sean Busbey 71cc7869db HBASE-20155 update branch-2 version to 2.1.0-SNAPSHOT
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2018-03-08 08:44:30 -08:00
Mike Drob af9a108a93 HBASE-19987 error-prone 2.2.0 2018-03-07 21:21:05 -06:00
Sean Busbey 9927c2e14a HBASE-20070 refactor website generation
* rely on git plumbing commands when checking if we've built the site for a particular commit already
* switch to forcing '-e' for bash
* add command line switches for: path to hbase, working directory, and publishing
* only export JAVA/MAVEN HOME if they aren't already set.
* add some docs about assumptions
* Update javadoc plugin to consistently be version 3.0.0
* avoid duplicative site invocations on reactor modules
* update use of cp command so it works both on linux and mac
* manually skip enforcer plugin during build
* still doing install of all jars due to MJAVADOC-490, but then skip rebuilding during aggregate reports.
* avoid the pager on git-diff by teeing to a log file, which also helps later reviewing in the case of big changesets.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Misty Stanley-Jones <misty@apache.org>

 Conflicts:
	hbase-backup/pom.xml
	hbase-spark-it/pom.xml
2018-03-02 09:51:43 -06:00
Michael Stack a2de29560f HBASE-20113 Move branch-2 version from 2.0.0-beta-2-SNAPSHOT to 2.0.0-beta-2 2018-03-01 15:46:38 -08:00
Michael Stack 44544c7db0 HBASE-20069 fix existing findbugs errors in hbase-server 2018-02-26 10:55:53 -08:00
Michael Stack 8b3ae58e18 HBASE-20043 ITBLL fails against hadoop3
Fix MoveRandomRegionOfTableAction. It depended on old AM behavior.
Make it do explicit move as is required in AMv3; w/o it, it was just
closing region causing test to fail.

Fix pom so hadoop3 profile specifies a different netty3 version.

Bunch of logging format change that came of trying trying to read
the spew from this test.
2018-02-24 17:29:24 -08:00
Sean Busbey a66fa85a16 Revert "HBASE-19835 Use explicit casting to avoid side effects"
This reverts commit bc080e7500.

 Conflicts:
	hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
	hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Segment.java

patch reverted changes that happened in parallel without explanation. see jira.
2018-02-23 23:04:34 -06:00
zhangduo 4ddfdaffdc HBASE-20048 Revert serial replication feature 2018-02-23 13:54:10 +08:00
tedyu 66ba8aa58c HBASE-20031 Unable to run integration test using mvn due to missing HBaseClassTestRule 2018-02-21 12:04:17 -08:00
eshcar a458d7c400 HBASE-18294 Reduce global heap pressure: flush based on heap occupancy 2018-02-18 12:58:52 +02:00
Michael Stack 8be0696320 HBASE-20014 TestAdmin1 Times out 2018-02-16 20:55:29 -08:00
zhangduo ce5f5e6bcd HBASE-20000 Remove the quantum logic in FairQueue, always put high priority queue in front 2018-02-15 16:40:21 +08:00
Michael Stack a23a7743c9 HBASE-19986 If HBaseTestClassRule timesout a test, thread dump; ADDENDUM; white-space, checkstyle, and rb feedback by Duo 2018-02-12 18:12:46 -08:00
Michael Stack e94f082776 HBASE-19986 If HBaseTestClassRule timesout a test, thread dump 2018-02-12 15:13:42 -08:00
Aman Poonia bc080e7500 HBASE-19835 Use explicit casting to avoid side effects
Signed-off-by: tedyu <yuzhihong@gmail.com>
2018-02-08 12:33:08 -08:00
Michael Stack c93ba37dfc HBASE-19956 Remove category as a consideration timing out tests; set all test to timeout at 10minutes regardless 2018-02-07 20:10:36 -08:00
Michael Stack 8f1e01b6e5 HBASE-19951 Cleanup the explicit timeout value for test method 2018-02-07 16:39:54 -08:00
zhangduo e71b0bf19d HBASE-19929 Call RS.stop on a session expired RS may hang 2018-02-07 15:53:08 +08:00
Michael Stack bac4687345 HBASE-19919 Tidying up logging 2018-02-02 22:42:30 -08:00
Mike Drob a165bd766a HBASE-19841 Every HTU should be local until DFS starts 2018-02-01 20:23:56 -06:00
Michael Stack ced502801f HBASE-19911 Convert some tests from small to medium because they are timing out: TestNettyRpcServer, TestClientClusterStatus; ADDENDUM2 Up timeout for smalltests from 30 seconds to 60 seconds -- a pause on jenkins can mess up smalltests 2018-02-01 08:24:54 -08:00
zhangduo 28d13c54a9 HBASE-19887 Do not overwrite the surefire junit listener property in the pom of sub modules 2018-01-31 20:49:25 +08:00
Michael Stack 90a75fb052 HBASE-19888 Move branch-2 version from 2.0.0-beta-1 to 2.0.0-beta-2-SNAPSHOT 2018-01-29 14:17:54 -08:00
Duo Zhang bbf3bae72a
HBASE-19873 Add a CategoryBasedTimeout ClassRule for all UTs 2018-01-29 12:41:14 -08:00
Michael Stack 2e2e7845ab
HBASE-19840 Flakey TestMetaWithReplicas; ADDENDUM to fix Checksyte 2018-01-29 12:41:13 -08:00
Michael Stack 77607e4961
HBASE-19840 Flakey TestMetaWithReplicas
Fix two issues:

 # Meta Replicas can all be assigned to the same server. This
 will call the test to hang when we do our kill of the server
 hosting meta because there'll be no replicas to read from
 as test intends. Check is to look for this condition on
 startup and adjust if we come across it. Replicas cross-cut
 assignment. They need work.
 # Other issue was shutdown. The master started toward the
 end of the test may not have come up fully by the time
 shutdown is called. We could be stuck assigning the
 meta replicas. Have shutdown shutdown the procedure
 executor engine.

There is other cleanup and notes in the below.

M HMaster
 Remove the silly stops in startup now we have real
 means of shutting down Master during init.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java
 This replica stuff was doing stuff it shouldn't be doing
 like setting core Master state flags. It may have made
 sense once but now meta is assigned by a Pv2 Procedure
 so the flag setting in here is meddlesome. Clear out
 methods no longer needed.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
 Remove unused methods.
 Changes local variable names so they align w/ our naming elsewhere in
 code base.

M hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestMetaWithReplicas.java
 Check for all replicas on the one server.
2018-01-29 09:26:41 -08:00
Chia-Ping Tsai a36e9c76c2 HBASE-19877 hbase-common and hbase-zookeeper don't add the log4j.properties to the resource path for testing 2018-01-29 16:52:13 +08:00
Michael Stack c1470714bb HBASE-19794 TestZooKeeper hangs
Kill backup master first

Add some cleanup around NamespaceManager

Shorten the timeout waiting on namespace manager as workaround
until we have better soln for interrupting ongoing client rpcs.
Do it in general for all tests.

Signed-off-by: zhangduo <zhangduo@apache.org>
2018-01-21 20:59:31 +08:00
Apekshit Sharma cdfd12c101 HBASE-19823 Make RawCellBuilderFactory LimitedPrivate.UNITTEST 2018-01-19 14:53:27 -08:00
Chia-Ping Tsai 3e6f80dcd5
HBASE-19746 Add default impl to Cell#getType 2018-01-11 15:04:25 -08:00
Apekshit Sharma 25e4bf8f37 HBASE-19773 Documenting how to get instance of type RawCellBuilder 2018-01-11 13:17:30 -08:00
Xiang Li bc4e49ffaa HBASE-19702 Improve RSGroupInfo constructors
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2018-01-09 12:55:30 +08:00
zhangduo 4c31374a90
HBASE-19731 TestFromClientSide#testCheckAndDeleteWithCompareOp and testNullQualifier are flakey 2018-01-08 10:34:57 -08:00
Beluga Behr 88c7493948
HBASE-19651 Remove LimitInputStream 2018-01-05 17:01:50 -06:00
Michael Stack e7da907c3c
Revert "HBASE-19651 Remove LimitInputStream"
Revert to fix author.

This reverts commit 4c19e1f4e3.
2018-01-05 16:59:59 -06:00
Michael Stack 4c19e1f4e3
HBASE-19651 Remove LimitInputStream
Signed-off-by: Beluga Behr <dam6923@gmail.com>
2018-01-05 08:38:12 -06:00
BELUGA BEHR b6bff5f8e2 HBASE-19683 Remove Superfluous Methods From String Class (BELUGA BEHR).
* Remove isEmpty method
* Remove repeat
Use the Apache Commons implementations instead.
2018-01-02 11:34:52 -08:00
tedyu 3f1cfc8f08 HBASE-19679 Superusers Logging and Data Structures (BELUGA BEHR) 2018-01-01 14:19:04 -08:00
tedyu c394f3919e HBASE-19678 HBase Admin security capabilities should be represented as a Set - revert due to wrong issue 2018-01-01 14:16:46 -08:00
tedyu c2ca90f0fb HBASE-19678 HBase Admin security capabilities should be represented as a Set (BELUGA BEHR) 2017-12-31 11:32:14 -08:00
Xiang Li 5b3513a5ee HBASE-19672 Correct comments for default values of major compaction in SortedCompactionPolicy#getNextMajorCompactTime()
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-12-30 16:28:09 -08:00
Michael Stack b4e3798bb9 HBASE-19660 Up default retries from 10 to 15 and blocking store files limit from 10 to 16 2017-12-28 15:02:23 -08:00
Michael Stack a8ef734ddd HBASE-19651 Remove LimitInputStream
Signed-off-by: Beluga Behr <dam6923@gmail.com>
2017-12-28 13:08:46 -08:00
Mike Drob 64cb777a8a HBASE-19552 find-and-replace thirdparty offset 2017-12-28 12:01:25 -06:00
Chia-Ping Tsai 97f0aad66e HBASE-19626 Rename Cell.DataType to Cell.Type 2017-12-28 12:59:28 +08:00
Michael Stack d6d8369655
HBASE-19648 Move branch-2 version from 2.0.0-beta-1-SNAPSHOT to 2.0.0-beta-1 2017-12-27 14:41:19 -08:00
Chia-Ping Tsai 7dee1bcd31 HBASE-19644 add the checkstyle rule to reject the illegal imports 2017-12-28 04:17:45 +08:00
Chia-Ping Tsai 2468f30029 HBASE-19628 ByteBufferCell should extend ExtendedCell 2017-12-27 23:02:18 +08:00
Vasudevan 0454a389a3 HBASE-19629 RawCell#getTags should return the Iterator<Tag> in order to avoid iterating through whole tag array at once
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-12-27 21:30:08 +08:00
Peter Somogyi 15ed748286 HBASE-19545 Replace getBytes(StandardCharsets.UTF_8) with Bytes.toBytes
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-12-27 20:10:50 +08:00
BELUGA BEHR 5f548146af HBASE-19487 Remove IterablesUtil Class
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-12-25 16:15:29 +08:00
Guanghao Zhang 1e490dcc11 HBASE-19602 Cleanup the usage of ReplicationAdmin from document 2017-12-23 19:05:04 +08:00
Michael Stack 95663f6b11
HBASE-19148 Reevaluate default values of configurations; minor ADDENDUM2
Fix a description.
2017-12-22 08:38:13 -08:00
Michael Stack 18987ed867
HBASE-19148 Reevaluate default values of configurations
Removed unused:
    <name>hbase.fs.tmp.dir</name>
Added hbase.master.loadbalance.bytable

Edit of description text. Moved stuff around  to put configs beside each
other.

M hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerCommandLine.java

 Emit some hbase configs in log on startup.

Signed-off-by: Michael Stack <stack@apache.org>
2017-12-21 14:07:47 -08:00
Balazs Meszaros 992b5d8630 HBASE-10092 Move up on to log4j2
Changes:
- replaced commons-logging to slf4j everywhere
- log.XXX(Throwable) calls were replaced with log.XXX(t.toString(), t)
- log.XXX(Object) calls were replaced with log.XXX(Objects.toString(obj))
- log.fatal() calls were replaced with log.error(HBaseMarkers.FATAL, ...)
- programmatic log4j configuration was removed from the unit test

This commit does not affect the current logging configurations, because log4j
is still on the classpath. slf4j-log4j12 binds log4j to slf4j.

Signed-off-by: Michael Stack <stack@apache.org>
2017-12-20 22:58:12 -08:00
Jan Hentschel 606ff3357d HBASE-19539 Removed unnecessary semicolons in hbase-common 2017-12-19 20:27:32 +01:00
Michael Stack 9acfb8ae67
HBASE-19122 Suspect methods on Cell to be deprecated; ADDENDUM2 to fix
more compile error
2017-12-18 16:13:00 -08:00
Peter Somogyi dbf0d427e2
HBASE-19549 Change path comparison in CommonFSUtils
Also change makeQualified(FileSystem fs)
to makeQualified(URI defaultUri, Path workingDir)

Signed-off-by: Michael Stack <stack@apache.org>
2017-12-18 15:28:29 -08:00
Michael Stack 495bedea35
HBASE-19122 Suspect methods on Cell to be deprecated 2017-12-18 15:23:03 -08:00
Chia-Ping Tsai b646d7249b HBASE-18750 Cleanup the docs saying "HTable use write buffer" 2017-12-18 17:44:28 +08:00
Peter Somogyi 4c2b2b3e6a
HBASE-19497 Fix findbugs and error-prone warnings in hbase-common (branch-2)
Signed-off-by: Apekshit Sharma <appy@apache.org>
2017-12-15 16:18:11 -08:00
Peter Somogyi 2043c337d8
HBASE-19498 Fix findbugs and error-prone warnings in hbase-client (branch-2)
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Apekshit Sharma <appy@apache.org>
2017-12-15 15:57:01 -08:00
Mike Drob 23a9059cb2 HBASE-18838 Fix hadoop3 check-shaded-invariants 2017-12-15 13:20:54 -06:00
Mike Drob 2952cc7dea HBASE-19289 Add flag to disable stream capability enforcement
Signed-off-by: Josh Elser <elserj@apache.org>
2017-12-14 12:19:59 -06:00
Chia-Ping Tsai d2ffd5d7ed HBASE-19484 (addendum) NPE in ExtendedCell#write 2017-12-14 02:32:28 +08:00
BELUGA BEHR 7c871849de HBASE-19472 Remove ArrayUtil Class
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-12-13 19:08:54 +08:00
tedyu 0e47ded1a5 HBASE-19394 Support multi-homing env for the publication of RS status with multicast (hbase.status.published) (Toshihiro Suzuki) 2017-12-12 07:48:40 -08:00
Chia-Ping Tsai 6010daf8b0 HBASE-19484 The value array written by ExtendedCell#write is out of bounds 2017-12-11 11:01:25 +08:00
BELUGA BEHR 3c84272b53 HBASE-19464 Replace StringBuffer with StringBuilder for hbase-common
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-12-10 22:47:54 +08:00
Apekshit Sharma e8ba7b2320 HBASE-19457 Debugging flaky TestTruncateTableProcedure
- Adds debug logging for future ease
- Removes 60s timeout since testRecoveryAndDoubleExecutionPreserveSplits is only halfway after a minute.
- Adds some comments
- Logging change: Some places report "regionState=" while others just "state=".
  State machine procs also have "state=" in their logs. Let me change all region related logging to "regionState=" so that
  1) it's consistent everywhere, 2) more filtered results when searching through logs.
2017-12-08 17:25:44 -08:00
anoopsamjohn f8e2323323 HBASE-19357 Bucket cache no longer L2 for LRU cache. 2017-12-07 15:06:30 +05:30
Chia-Ping Tsai 4833e63d17 HBASE-19430 Remove the SettableTimestamp and SettableSequenceId 2017-12-07 13:34:38 +08:00
zhangduo 03cb581585 HBASE-19410 Move zookeeper related UTs to hbase-zookeeper and mark them as ZKTests 2017-12-06 16:38:58 +08:00
anoopsamjohn 78a6e0532e Revert "HBASE-19357 Bucket cache no longer L2 for LRU cache."
This reverts commit d952c5df9f.
2017-12-06 11:36:19 +05:30
anoopsamjohn d952c5df9f HBASE-19357 Bucket cache no longer L2 for LRU cache. 2017-12-06 11:26:10 +05:30
Chia-Ping Tsai 2ab0470b43 HBASE-19431 The tag array written by IndividualBytesFieldCell#write is out of bounds 2017-12-05 15:42:29 +08:00
Chia-Ping Tsai 73fd3d32a1 HBASE-19298 CellScanner and CellScannable should be declared as IA.Public
Signed-off-by: Michael Stack <stack@apache.org>
2017-12-02 15:48:00 -08:00
Yuqi Gu d32ba0977f HBASE-19386 Enable Arm64 unaligned support for HBase
On Arm64, java.nio.Bits.unaligned() wrongly returns false due to a JDK bug.
This causes HBase UnsafeAvailChecker wrongly returning false on Arm64.
And it slso cause FuzzyRowFilter Unit test failed.
Fix it by providing a hard-code to enbale Arm64 unaligned support.

Jira: HBASE-19386

Change-Id: I3ab821dacbe42b18cd515080da1fa3dc1f1e1d28
Signed-off-by: Yuqi Gu <yuqi.gu@arm.com>
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-11-30 00:42:58 -08:00
Michael Stack 9419709995 HBASE-19383 [1.2] java.lang.AssertionError: expected:<2> but was:<1> at
org.apache.hadoop.hbase.TestChoreService.testTriggerNowFailsWhenNotScheduled(TestChoreService.java:707)
2017-11-29 20:45:58 -08:00
Guanghao Zhang 9e7ab9ffe2
HBASE-19359 Revisit the default config of hbase client retries number 2017-11-29 10:33:20 -08:00
Josh Elser 4fef4cfc30 HBASE-19267 Remove compiler-plugin mapping executions as it breaks Java8 detection
It seems like the original reason this execution filter was added is no
longer an issue for 2.0. Actually, these entries actually preclude
Eclipse from correctly using the Java8 source/target version that we
have specified (which creates numerous compilation errors in Eclipse)

Signed-off-by: Guanghao Zhang <zghao@apache.org>
2017-11-28 14:54:32 -05:00
Vasudevan 6ac6ae3fa2 HBASE-19092 Make Tag IA.LimitedPrivate and expose for CPs (Ram)
Signed-off-by: Chia-ping Tsai, Anoop Sam John, Stack
2017-11-24 12:38:42 +05:30
zhangduo 9c29a6211b HBASE-19311 Promote TestAcidGuarantees to LargeTests and start mini cluster once to make it faster 2017-11-22 14:10:17 +08:00
Peter Somogyi bcd367e293
HBASE-19315 Incorrect snapshot version is used for 2.0.0-beta-1
Signed-off-by: Michael Stack <stack@apache.org>
2017-11-21 10:41:50 -08:00
Lars Hofhansl 3d4fd90947 HBASE-12091 Optionally ignore edits for dropped tables for replication. 2017-11-14 17:16:56 -08:00
Tamas Penzes 7a69ebc73e HBASE-18601: Update Htrace to 4.2
Updated HTrace version to 4.2
Created TraceUtil class to wrap htrace methods. Uses try with resources.

Signed-off-by: Balazs Meszaros <balazs.meszaros@cloudera.com>
Signed-off-by: Michael Stack <stack@apache.org>
2017-11-13 10:38:36 -08:00
anoopsamjohn 3ee76f8573 HBASE-19187 Remove option to create on heap bucket cache. 2017-11-11 12:41:09 +05:30
Apekshit Sharma a6d8023ff5 HBASE-19128 Purge Distributed Log Replay from codebase, configurations, text; mark the feature as unsupported, broken. 2017-11-07 17:48:52 -08:00
Umesh Agashe fa3cc6c3bc
HBASE-18961 doMiniBatchMutate() is split into smaller member methods of BatchOperation and it's sub-classes
There is no functionality change except for below:
* Variable lastIndexExclusive was getting incremented while locking rows corresponding to input
  operations. As a result when getRowLockInternal() method throws TimeoutIOException only operations
  in range [nextIndexToProcess, lastIndexExclusive) was getting marked as FAILED before raising
  exception up the call stack. With these changes all operations are getting marked as FAILED.
* Cluster Ids of first mutation is used consistently for entire batch. Previous behavior was to use
  cluster ids of first mutation in a mini-batch

Signed-off-by: Michael Stack <stack@apache.org>
2017-11-07 10:01:28 -08:00
Michael Stack f13cf56f1c
HBASE-19197 Move version on branch-2 from 2.0.0-alpha4 to 2.0.0-beta-1.SNAPSHOT 2017-11-06 20:46:38 -08:00
Josh Elser f4a4144f35 HBASE-19111 Add CellUtil#isPut and deprecate methods returning/expecting non public-api data
KeyValue.Type, and its corresponding byte value, are not public API. We
shouldn't have methods that are expecting them. Added a basic sanity
test for isPut and isDelete.

Signed-off-by: Ramkrishna <ramkrishna.s.vasudevan@intel.com>
2017-11-06 15:37:16 -05:00
Mike Drob cfddfcf23c HBASE-19160 expose CellComparator as IA.Public 2017-11-06 10:15:55 -06:00
Michael Stack f8c58930aa
HBASE-19179 Remove hbase-prefix-tree 2017-11-04 10:09:46 -07:00
Sean Busbey a9f0c5d4e2 HBASE-18784 if available, query underlying outputstream capabilities where we need hflush/hsync.
* pull things that don't rely on HDFS in hbase-server/FSUtils into hbase-common/CommonFSUtils
* refactor setStoragePolicy so that it can move into hbase-common/CommonFSUtils, as a side effect update it for Hadoop 2.8,3.0+
* refactor WALProcedureStore so that it handles its own FS interactions
* add a reflection-based lookup of stream capabilities
* call said lookup in places where we make WALs to make sure hflush/hsync is available.
* javadoc / checkstyle cleanup on changes as flagged by yetus

Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-11-02 21:54:16 -05:00
Apekshit Sharma d69570a485 HBASE-18925 Update mockito dependency from mockito-all:1.10.19 to mockito-core:2.1.0 for JDK8 support.
Last mockito-all release was in Dec'14. Mockito-core has had many releases since then.

From mockito's site:
- "Mockito does not produce the mockito-all artifact anymore ; this one was primarily
aimed at ant users, and contained other dependencies. We felt it was time to move on
and remove such artifacts as they cause problems in dependency management system like
maven or gradle."
- anyX() and any(SomeType.class) matchers now reject nulls and check type.
2017-11-01 14:38:50 -07:00
eshcar 526d2826f5 HBASE-16417: In-memory MemStore Policy for Flattening and Compactions 2017-11-01 16:48:40 +02:00
Michael Stack 426ab1e931 HBASE-19135 TestWeakObjectPool time out 2017-10-31 16:16:56 -07:00
Andrew Purtell 285efb1f1b HBASE-19129 TestChoreService is flaky
Increase the delta to compensate for environmental variance.
2017-10-31 00:08:37 +00:00
Michael Stack efb5d7b24d
HBASE-18995 Move methods that are for internal usage from CellUtil to Private util class (Ramkrishna Vasudevan); ADDENDUM add file I forgot to add 2017-10-27 20:35:51 -07:00
Michael Stack 250a8bc7ad
HBASE-18995 Move methods that are for internal usage from CellUtil to Private util class (Ramkrishna Vasudevan) 2017-10-27 17:51:32 -07:00
Ramkrishna 1b70751c9c HBASE-19090 Add config 'hbase.systemtables.compacting.memstore.type' to
hbase-default.xml
2017-10-26 12:43:20 +05:30
Apekshit Sharma 3ce7ab3c70 HBASE-19073 Cleanup CoordinatedStateManager
- Merged BaseCSM class into CSM interface
- Removed config hbase.coordinated.state.manager.class
- Since state manager is not pluggable anymore, we don't need start/stop/initialize to setup unknown classes. Our internal ZkCSM now requires Server in constructor itself. Makes the dependency clearer too.
- Removed CSM from HRegionServer and HMaster constructor. Although it's a step back from dependency injection, but it's more consistent with our current (not good)  pattern where we initialize everything in the ctor itself.

Change-Id: Ifca06bb354adec5b11ea1bad4707e014410491fc
2017-10-24 20:00:31 -07:00
Apekshit Sharma d6982414c1 HBASE-19053 Split out o.a.h.h.http from hbase-server into a separate module
Change-Id: Ie3a688b789104df7feaf34ac9fb326a79d6a3960
2017-10-23 23:25:26 -07:00
Ramkrishna 2cb64fb467 HBSE-18945 Make a IA.LimitedPrivate interface for CellComparator (Ram) 2017-10-17 23:19:21 +05:30
Chia-Ping Tsai a63d79837d HBASE-18966 Use non-sync TimeRangeTracker as a replacement for TimeRange in ImmutableSegment 2017-10-14 23:16:48 +08:00
Peter Somogyi 507a3f9425
HBASE-18108 Procedure WALs are archived but not cleaned; fix
The archived Procedure WALs are moved to <hbase_root>/oldWALs/masterProcedureWALs
directory. TimeToLiveProcedureWALCleaner class was added which
regularly cleans the Procedure WAL files from there.

The TimeToLiveProcedureWALCleaner is now added to
hbase.master.logcleaner.plugins to clean the 2 WALs in one run.

A new config parameter is added hbase.master.procedurewalcleaner.ttl
which specifies how long a Procedure WAL should stay in the
archive directory.

Signed-off-by: Michael Stack <stack@apache.org>
2017-10-11 14:30:45 -07:00
zhangduo d15549738a HBASE-18878 Use Optional in return types.
These functions have been changed to return Optional<T> instead of T, where T = old return type.
- ObserverContext#getCaller
- RpcCallContext#getRequestUser
- RpcCallContext#getRequestUserName
- RpcServer#getCurrentCall
- RpcServer#getRequestUser
- RpcServer#getRequestUserName
- RpcServer#getRemoteAddress
- ServerCall#getRequestUser

Change-Id: Ib7b4e6be637283755f55755dd4c5124729f7052e
Signed-off-by: Apekshit Sharma <appy@apache.org>
2017-10-04 15:59:13 -07:00
Chia-Ping Tsai 2a5b6bc681 HBASE-18927 Add the DataType which is subset of KeyValue#Type to CellBuilder for building cell 2017-10-04 23:45:57 +08:00
Sean Busbey 35094bf4d5 HBASE-18933 set version number to 2.0.0-alpha4-SNAPSHOT following release of alpha3 2017-10-04 07:57:49 -05:00
Ramkrishna 66fb60d4a4 HBASE-18649 Deprecate KV Usage in MR to move to Cells in 3.0 (Ram) 2017-10-04 14:44:19 +05:30
Chia-Ping Tsai 2cc1ea3d2e HBASE-18753 Introduce the unsynchronized TimeRangeTracker 2017-10-01 21:38:05 +08:00
anastas 95405fbd83 HBASE-18010: CellChunkMap integration into CompactingMemStore, merge to Release 2.0 2017-10-01 11:01:10 +03:00
Andy Yang 0658252ed6 HBASE-13844 Move static helper methods from KeyValue into CellUtils
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-09-29 13:20:24 +08:00
Chia-Ping Tsai f380676d49 HBASE-18835 The return type of ExtendedCell#deepClone should be ExtendedCell rather than Cell 2017-09-18 22:38:42 +08:00
zhangduo d90f77ab7d HBASE-14004 [Replication] Inconsistency between Memstore and WAL may result in data in remote cluster that is not in the origin 2017-09-15 19:22:05 +08:00
Michael Stack 7660f9e86a HBASE-18819 Set version number to 2.0.0-alpha3 from 2.0.0-alpha3-SNAPSHOT 2017-09-14 12:38:46 -07:00
Sean Busbey d576e5a32d HBASE-17823 Migrate to Apache Yetus Audience Annotations
Includes partial backport of hbase-build-configuration module

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Misty Stanley-Jones <misty@apache.org>
2017-09-12 23:15:50 -05:00
Ramkrishna b651b46d36 HBASE-18795 - Expose KeyValue.getBuffer() for tests alone (Ram) 2017-09-12 15:25:10 +05:30
Peter Somogyi 8172251a9d HBASE-10240 Remove 0.94->0.96 migration code
Change-Id: I70cdbd1f1159ca923dc79221601e6190db2b4de1

Signed-off-by: Michael Stack <stack@apache.org>
2017-09-11 12:08:20 -07:00
Yun Zhao 7af0c44af3 HBASE-18662 The default values for many configuration items in the code are not consistent with hbase-default.xml
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-09-11 21:12:11 +08:00
Chia-Ping Tsai 1ee4923730 Revert "HBASE-18662 The default values for many configuration items in the code are not consistent with hbase-default.xml"
need more checks
This reverts commit 89f978beb8.
2017-09-09 22:55:33 +08:00
Yun Zhao 89f978beb8 HBASE-18662 The default values for many configuration items in the code are not consistent with hbase-default.xml
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-09-09 18:11:14 +08:00
Balazs Meszaros c48dc02b76 HBASE-18106 Redo ProcedureInfo and LockInfo
Main changes:
- ProcedureInfo and LockInfo were removed, we use JSON instead of them
- Procedure and LockedResource are their server side equivalent
- Procedure protobuf state_data became obsolate, it is only kept for
reading previously written WAL
- Procedure protobuf contains a state_message field, which stores the internal
state messages (Any type instead of bytes)
- Procedure.serializeStateData and deserializeStateData were changed slightly
- Procedures internal states are available on client side
- Procedures are displayed on web UI and in shell in the following jruby format:
  { ID => '1', PARENT_ID = '-1', PARAMETERS => [ ..extra state information.. ] }

Signed-off-by: Michael Stack <stack@apache.org>
2017-09-08 11:56:28 -07:00
Michael Stack f08dd5d342 HBASE-18779 Move CompareOperator to hbase-client module 2017-09-08 10:18:32 -07:00
Michael Stack 7f70b6c8e3 HBASE-18769 Make CompareFilter use generic CompareOperator instead of
internal enum
2017-09-08 10:05:51 -07:00
Apekshit Sharma 56cba5e450 HBASE-14997 Move compareOp and Comparators out of filter to client package
Add a new generic CompareOperator enum and use it instead of the filter
CompareOp enum everywhere BUT inside CompareFilter.
2017-09-07 11:03:22 -07:00
Peter Somogyi 33711fd481 HBASE-18704 Upgrade hbase to commons-collections 4
Upgrade commons-collections:3.2.2 to commons-collections4:4.1
Add missing dependency for hbase-procedure, hbase-thrift
Replace CircularFifoBuffer with CircularFifoQueue in WALProcedureStore and TaskMonitor

Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
(cherry picked from commit 137b105c67)
2017-09-07 10:39:13 -05:00
Michael Stack 7592cb8d34 HBASE-18768 Move TestTableName to hbase-common from hbase-server
Signed-off-by: Michael Stack <stack@apache.org>
2017-09-06 18:12:14 -07:00
Umesh Agashe bd219c0fb8 HBASE-18674 upgrading to commons-lang3
Signed-off-by: Michael Stack <stack@apache.org>
2017-09-05 09:46:49 -07:00
tedyu c762753b4b HBASE-18743 HFiles in use by a table which has the same name and namespace with a default table cloned from snapshot may be deleted when that snapshot and default table are deleted (wenbang) 2017-09-03 07:11:16 -07:00
Michael Stack 9dd458a8c7 HBASE-18739 Make all TimeRange Constructors InterfaceAudience Private 2017-09-01 12:37:04 -07:00
Michael Stack 43c4bc5761 HBASE-18723 [pom cleanup] Do a pass with dependency:analyze; remove unused and explicity list the dependencies we exploit
Do a pass with dependency:analyze; remove unused and
explicity list the dependencies we exploit.
Remove the parent dependencies set which had junit, mockito,
log4j, and findbugs annotations (had to put junit back
temporarily in subsequent version of this patch TODO). Listing in
parent set meant these libs were dependencies for all modules
which in practice was not the case. Edited all modules so
those that need any from this parent set now do explicit listing.

Ran the dependency:analyze over the project. Acted on most
suggested removals and requests for explicit listing. Some
grey areas remain around transitives that come in with
hadoop -needs better excludes, another project- and that
the dependency:analyze tool is not always accurate in its
reporting.
2017-09-01 08:03:35 -07:00
Chia-Ping Tsai a32642e6f6 HBASE-18721 Cleanup unused configs and private declaration 2017-08-31 23:40:18 +08:00
Guangxu Cheng a0f980b573 HBASE-18461 Build broken If the username contains a backslash
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-08-30 19:06:53 -07:00
Sean Mackrory a722c7ecdf HBASE-15947 Classes used only for tests included in main code base
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-08-30 10:40:45 +08:00
Michael Stack 5053ed9545 HBASE-18692 [compat 1-2] ByteBufferUtils.copyFromBufferToBuffer goes from void to int 2017-08-28 10:01:18 -07:00
Chia-Ping Tsai 0142c09217 HBASE-18519 Use builder pattern to create cell 2017-08-28 14:19:45 +08:00
Mike Drob 6252ae22cf HBASE-18656 First issues found by error-prone 2017-08-24 12:16:14 -05:00
Michael Stack 6230d21506 HBASE-18658 Purge hokey hbase Service implementation; use (internal) Guava Service instead
Removes hbase Service. Moves the single user, ClusterSchemaServiceImpl
to use relocated internal Guava Service instead.
2017-08-24 09:25:42 -07:00
Michael Stack 8c84793d22 HBASE-18224 Upgrade Jetty 2017-08-23 23:14:13 -07:00
Michael Stack b24e33312a HBASE-18594 Release hbase-2.0.0-alpha2; ADDENDUM update version from 2.0.0-alpha2 to 2.0.0-alpha3-SNAPSHOT 2017-08-23 11:07:41 -07:00
Umesh Agashe fad968d99f HBASE-18575 [AMv2] Fixed and enabled TestRestartCluster#testRetainAssignmentOnRestart on master
* Fixed ServerCrashProcedure to set forceNewPlan to false for instances AssignProcedure. This enables balancer to find most suitable target server
* Fixed and enabled TestRestartCluster#testRetainAssignmentOnRestart on master
* Renamed method ServerName@isSameHostnameAndPort() to isSameAddress()

Signed-off-by: Michael Stack <stack@apache.org>
2017-08-23 10:11:54 -07:00
Chia-Ping Tsai bb2b6b8662 HBASE-18471 The DeleteFamily cell is skipped when StoreScanner seeks to next column 2017-08-19 02:14:54 +08:00
Mike Drob 0ded122b1e HBASE-18303 Clean up @Parameter boilerplate 2017-08-14 14:24:37 -05:00
Michael Stack add9974515 HBASE-18595 Set version in branch-2 from 2.0.0-alpha2-SNAPSHOT to 2.0.0-alpha2 2017-08-14 10:28:44 -07:00
dongtao.zhang 8da096b231 HBASE-18262 name of parameter quote need update
Signed-off-by: Michael Stack <stack@apache.org>
2017-08-10 14:49:45 -07:00
Josh Elser 50d9301136 HBASE-18023 Update row threshold warning from 1k to 5k (addendum) 2017-07-25 18:35:12 -04:00
rgidwani d461bec6c2 HBASE-15816 Provide client with ability to set priority on Operations
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-07-21 17:12:21 -07:00
Xiang Li 946289113a HBASE-18389 Remove byte[] from formal parameter of sizeOf() of ClassSize, ClassSize.MemoryLayout and ClassSize.UnsafeLayout
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-07-22 02:16:09 +08:00
Michael Stack d5c6e11016 HBASE-17908 Upgrade guava
Pull in guava 22.0 by using the shaded version up in new hbase-thirdparty project.

In poms, exclude guava everywhere except on hadoop-common. Do this so
we minimize transitive includes. hadoop-common is needed because hadoop
Configuration uses guava doing preconditions.

Everywhere we used guava, instead use shaded so fix a load of imports.

Stopwatch API changed as did hashing and toStringHelper which is now
in MoreObjects class. Otherwise, minimal changes to come up on 22.0
2017-07-21 15:41:52 +01:00
anoopsamjohn 8606cda357 HBASE-16993 BucketCache throw java.io.IOException: Invalid HFile block magic when configuring hbase.bucketcache.bucket.sizes. 2017-07-20 22:59:52 +05:30
Ramkrishna e095d3964b HBASE-17738 BucketCache startup is slow - addendum (Ram) 2017-07-20 22:40:30 +05:30
Ramkrishna 19154c8ea8 HBASE-17738 BucketCache startup is slow (Ram) 2017-07-19 21:53:06 +05:30
Sean Busbey 9cc9404261 HBASE-18187 update version to 2.0.0-alpha-2-SNAPSHOT 2017-07-14 11:20:32 -05:00
Chia-Ping Tsai 9daac09f6e HBASE-18365 Eliminate the findbugs warnings for hbase-common 2017-07-13 19:33:06 +08:00
Phil Yang 1ead5f9103 HBASE-17931 Assign system tables to servers with highest version 2017-07-06 17:28:40 +08:00
Peter Somogyi eacbdb061e HBASE-18264 Update pom plugins
Update plugins in main and subprojects
Unified versions to use variable instead of direct values

Affected plugins:
- apache-rat-plugin 0.11 -> 0.12
- asciidoctor-maven-plugin 1.5.2.1 -> 1.5.5
- asciidoctorj-pdf 1.5.0-alpha.6 -> 1.5.0-alpha.15
- build-helper-maven-plugin 1.9.1 -> 3.0.0
- buildnumber-maven-plugin 1.3 -> 1.4
- exec-maven-plugin 1.2.1/1.4.0 -> 1.6.0
- extra-enforcer-rules 1.0-beta-3 -> 1.0-beta-6
- findbugs-maven-plugin 3.0.0 -> 3.0.4
- jamon-maven-plugin 2.4.1 -> 2.4.2
- maven-bundle-plugin 2.5.3 -> 3.3.0
- maven-compiler-plugin 3.2/3.5.1 -> 3.6.1
- maven-eclipse-plugin 2.9 -> 2.10
- maven-shade-plugin 2.4.1 -> 3.0.0
- maven-surefire-plugin 2.18.1 -> 2.20
- maven-surefire-report-plugin 2.7.2 -> 2.20
- scala-maven-plugin 3.2.0 -> 3.2.2
- spotbugs 3.1.0-RC1 -> 3.1.0-RC3
- wagon-ssh 2.2 -> 2.12
- xml-maven-plugin 1.0 -> 1.0.1

- maven-assembly-plugin 2.4 -> 2.6(inherited)
- maven-dependency-plugin 2.4 -> 2.10 (inherited)
- maven-enforcer-plugin 1.3.1 -> 1.4.1 (inherited)
- maven-javadoc-plugin 2.10.3 -> 2.10.4 (inherited)
- maven-resources-plugin 2.7 (inherited)
- maven-site-plugin 3.4 -> 3.5.1 (inherited)

Change-Id: I84539f555be498dff18caed1e3eea1e1aeb2143a

Signed-off-by: Michael Stack <stack@apache.org>
2017-07-03 19:43:42 -07:00
David Harju d9dd319390 HBASE-18023 Log multi-* requests for more than threshold number of rows
Signed-off-by: Josh Elser <elserj@apache.org>
2017-06-24 15:34:02 -04:00
tedyu 3ff84a2da9 HBASE-18226 Disable reverse DNS lookup at HMaster and use the hostname provided by RegionServer (Duo Xu) 2017-06-21 08:15:27 -07:00
Sean Busbey d3ba357c7c HBASE-16242 Manually resolve transitive Apache Avro dependency to consistently be ver 1.7.7.
Signed-off-by: Jerry He <jerryjch@apache.org>
2017-06-19 16:07:33 -05:00
Andrew Purtell 6af3ee44c0 HBASE-18219 Fix typo in constant HConstants.HBASE_CLIENT_MEAT_REPLICA_SCAN_TIMEOUT 2017-06-14 16:03:50 -07:00
Michael Stack adfb48eeb8 HBASE-18190 Set version in branch-2 to 2.0.0-alpha-1 2017-06-07 21:10:01 -07:00
anoopsamjohn 8bfa8aaaca HBASE-18030 Per Cell TTL tags may get duplicated with increments/Append causing tags length overflow. 2017-06-06 12:25:15 +05:30
Michael Stack 3975bbd008 HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi) Move to a new AssignmentManager, one that describes Assignment using a State Machine built on top of ProcedureV2 facility.
This doc. keeps state on where we are at w/ the new AM:
https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.vfdoxqut9lqn
Includes list of tests disabled by this patch with reasons why.

Based on patches from Matteos' repository and then fix up to get it all to pass cluster
tests, filling in some missing functionality, fix of findbugs, fixing bugs, etc..
including:

    1. HBASE-14616 Procedure v2 - Replace the old AM with the new AM.
    The basis comes from Matteo's repo here:
    689227fcbf

    Patch replaces old AM with the new under subpackage master.assignment.
    Mostly just updating classes to use new AM -- import changes -- rather
    than the old. It also removes old AM and supporting classes.
    See below for more detail.

    2. HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi)
    3622cba4e3

    Adds running of remote procedure. Adds batching of remote calls.
    Adds support for assign/unassign in procedures. Adds version info
    reporting in rpc. Adds start of an AMv2.

    3. Reporting of remote RS version is from here:
    ddb4df3964.patch

    4. And remote dispatch of procedures is from:
    186b9e7c4d

    5. The split merge patches from here are also melded in:
    9a3a95a2c2
    and d6289307a0

We add testing util for new AM and new sets of tests.

Does a bunch of fixup on logging so its possible to follow a procedures' narrative by grepping
procedure id. We spewed loads of log too on big transitions such as master fail; fixed.

Fix CatalogTracker. Make it use Procedures doing clean up of Region data on split/merge.
Without these changes, ITBLL was failing at larger scale (3-4hours 5B rows) because we were
splitting split Regions among other things (CJ would run but wasn't
taking lock on Regions so havoc).

    Added a bunch of doc. on Procedure primitives.

    Added new region-based state machine base class. Moved region-based
    state machines on to it.

    Found bugs in the way procedure locking was doing in a few of the
    region-based Procedures. Having them all have same subclass helps here.

    Added isSplittable and isMergeable to the Region Interface.

    Master would split/merge even though the Regions still had
    references. Fixed it so Master asks RegionServer if Region
    is splittable.

    Messing more w/ logging. Made all procedures log the same and report
    the state the same; helps when logging is regular.

    Rewrote TestCatalogTracker. Enabled TestMergeTableRegionProcedure.

    Added more functionality to MockMasterServices so can use it doing
    standalone testing of Procedures (made TestCatalogTracker use it
    instead of its own version).

    Add to MasterServices ability to wait on Master being up -- makes
    it so can Mock Master and start to implement standalone split testing.
    Start in on a Split region standalone test in TestAM.

    Fix bug where a Split can fail because it comes in in the middle of
    a Move (by holding lock for duration of a Move).

    Breaks CPs that were watching merge/split. These are run by Master now
    so you need to observe on Master, not on RegionServer.

    Details:

    M hbase-client/src/main/java/org/apache/hadoop/hbase/ClusterStatus.java
    Takes List of regionstates on construction rather than a Set.
    NOTE!!!!! This is a change in a public class.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
    Add utility getShortNameToLog

    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
    M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ShortCircuitMasterConnection.java
    Add support for dispatching assign, split and merge processes.

    M hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java
    Purge old overlapping states: PENDING_OPEN, PENDING_CLOSE, etc.

    M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java
    Lots of doc on its inner workings. Bug fixes.

    M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
    Log and doc on workings. Bug fixes.

    A hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java
    Dispatch remote procedures every 150ms or 32 items -- which ever
    happens first (configurable). Runs a timeout thread. This facility is
    not on yet; will come in as part of a later fix. Currently works a
    region at a time. This class carries notion of a remote procedure and of a buffer full of these.
    "hbase.procedure.remote.dispatcher.threadpool.size" with default = 128
    "hbase.procedure.remote.dispatcher.delay.msec" with default = 150ms
    "hbase.procedure.remote.dispatcher.max.queue.size" with default = 32

    M hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java
    Add in support for merge. Remove no-longer used methods.

    M hbase-protocol-shaded/src/main/protobuf/Admin.proto b/hbase-protocol-shaded/src/main/protobuf/Admin.proto
    Add execute procedures call ExecuteProcedures.

    M hbase-protocol-shaded/src/main/protobuf/MasterProcedure.proto
    Add assign and unassign state support for procedures.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/client/VersionInfoUtil.java
    Adds getting RS version out of RPC
    Examples: (1.3.4 is 0x0103004, 2.1.0 is 0x0201000)

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
    Remove periodic metrics chore. This is done over in new AM now.
    Replace AM with the new. Host the procedures executor.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java
    Have AMv2 handle assigning meta.

    M hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
    Extract version number of the server making rpc.

    A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignProcedure.java
    Add new assign procedure. Runs assign via Procedure Dispatch.
    There can only be one RegionTransitionProcedure per region running at the time,
    since each procedure takes a lock on the region.

    D hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignCallable.java
    D hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
    D hbase-server/src/main/java/org/apache/hadoop/hbase/master/BulkAssigner.java
    D hbase-server/src/main/java/org/apache/hadoop/hbase/master/GeneralBulkAssigner.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/GeneralBulkAssigner.java
    Remove these hacky classes that were never supposed to live longer than
    a month or so to be replaced with real assigners.

    D hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStateStore.java
    D hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java
    D hbase-server/src/main/java/org/apache/hadoop/hbase/master/UnAssignCallable.java

    A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
    A procedure-based AM (AMv2).

    TODO
     - handle region migration
     - handle meta assignment first
     - handle sys table assignment first (e.g. acl, namespace)
     - handle table priorities
      "hbase.assignment.bootstrap.thread.pool.size"; default size is 16.
      "hbase.assignment.dispatch.wait.msec"; default wait is 150
      "hbase.assignment.dispatch.wait.queue.max.size"; wait max default is 100
      "hbase.assignment.rit.chore.interval.msec"; default is 5 * 1000;
      "hbase.assignment.maximum.attempts"; default is 10;

     A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MoveRegionProcedure.java
     Procedure that runs subprocedure to unassign and then assign to new location

     A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
     Manage store of region state (in hbase:meta by default).

     A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
     In-memory state of all regions. Used by AMv2.

     A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java
     Base RIT procedure for Assign and Unassign.

     A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/UnassignProcedure.java
     Unassign procedure.

     A hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
     Run region assignement in a manner that pays attention to target server version.
     Adds "hbase.regionserver.rpc.startup.waittime"; defaults 60 seconds.
2017-05-31 17:49:11 -07:00
Andrew Purtell 564c193d61 HBASE-18118 Default storage policy if not configured cannot be "NONE" 2017-05-26 11:08:46 -07:00
Michael Stack a3c5a74487 Revert "HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi)"
Revert a mistaken commit!!!

This reverts commit dc1065a85d.
2017-05-24 23:31:36 -07:00
Michael Stack dc1065a85d HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi)
Move to a new AssignmentManager, one that describes Assignment using
a State Machine built on top of ProcedureV2 facility.

This doc. keeps state on where we are at w/ the new AM:
https://docs.google.com/document/d/1eVKa7FHdeoJ1-9o8yZcOTAQbv0u0bblBlCCzVSIn69g/edit#heading=h.vfdoxqut9lqn
Includes list of tests disabled by this patch with reasons why.

Based on patches from Matteos' repository and then fix up to get it all to pass cluster
tests, filling in some missing functionality, fix of findbugs, fixing bugs, etc..
including:

1. HBASE-14616 Procedure v2 - Replace the old AM with the new AM.
The basis comes from Matteo's repo here:
689227fcbf

Patch replaces old AM with the new under subpackage master.assignment.
Mostly just updating classes to use new AM -- import changes -- rather
than the old. It also removes old AM and supporting classes.
See below for more detail.

2. HBASE-14614 Procedure v2 - Core Assignment Manager (Matteo Bertozzi)
3622cba4e3

Adds running of remote procedure. Adds batching of remote calls.
Adds support for assign/unassign in procedures. Adds version info
reporting in rpc. Adds start of an AMv2.

3. Reporting of remote RS version is from here:
ddb4df3964.patch

4. And remote dispatch of procedures is from:
186b9e7c4d

5. The split merge patches from here are also melded in:
9a3a95a2c2
and d6289307a0

We add testing util for new AM and new sets of tests.

Does a bunch of fixup on logging so its possible to follow a procedures' narrative by grepping
procedure id. We spewed loads of log too on big transitions such as master fail; fixed.

Fix CatalogTracker. Make it use Procedures doing clean up of Region data on split/merge.
Without these changes, ITBLL was failing at larger scale (3-4hours 5B rows) because we were
splitting split Regions among other things (CJ would run but wasn't
taking lock on Regions so havoc).

Added a bunch of doc. on Procedure primitives.

Added new region-based state machine base class. Moved region-based
state machines on to it.

Found bugs in the way procedure locking was doing in a few of the
region-based Procedures. Having them all have same subclass helps here.

Added isSplittable and isMergeable to the Region Interface.

Master would split/merge even though the Regions still had
references. Fixed it so Master asks RegionServer if Region
is splittable.

Messing more w/ logging. Made all procedures log the same and report
the state the same; helps when logging is regular.

Rewrote TestCatalogTracker. Enabled TestMergeTableRegionProcedure.

Added more functionality to MockMasterServices so can use it doing
standalone testing of Procedures (made TestCatalogTracker use it
instead of its own version).

Add to MasterServices ability to wait on Master being up -- makes
it so can Mock Master and start to implement standalone split testing.
Start in on a Split region standalone test in TestAM.

Fix bug where a Split can fail because it comes in in the middle of
a Move (by holding lock for duration of a Move).

Breaks CPs that were watching merge/split. These are run by Master now
so you need to observe on Master, not on RegionServer.

Details:

M hbase-client/src/main/java/org/apache/hadoop/hbase/ClusterStatus.java
Takes List of regionstates on construction rather than a Set.
NOTE!!!!! This is a change in a public class.

M hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java
Add utility getShortNameToLog

M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
M hbase-client/src/main/java/org/apache/hadoop/hbase/client/ShortCircuitMasterConnection.java
Add support for dispatching assign, split and merge processes.

M hbase-client/src/main/java/org/apache/hadoop/hbase/master/RegionState.java
Purge old overlapping states: PENDING_OPEN, PENDING_CLOSE, etc.

M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/Procedure.java
Lots of doc on its inner workings. Bug fixes.

M hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/ProcedureExecutor.java
Log and doc on workings. Bug fixes.

A hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/RemoteProcedureDispatcher.java
Dispatch remote procedures every 150ms or 32 items -- which ever
happens first (configurable). Runs a timeout thread. This facility is
not on yet; will come in as part of a later fix. Currently works a
region at a time. This class carries notion of a remote procedure and of a buffer full of these.
"hbase.procedure.remote.dispatcher.threadpool.size" with default = 128
"hbase.procedure.remote.dispatcher.delay.msec" with default = 150ms
"hbase.procedure.remote.dispatcher.max.queue.size" with default = 32

M hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java
Add in support for merge. Remove no-longer used methods.

M hbase-protocol-shaded/src/main/protobuf/Admin.proto b/hbase-protocol-shaded/src/main/protobuf/Admin.proto
Add execute procedures call ExecuteProcedures.

M hbase-protocol-shaded/src/main/protobuf/MasterProcedure.proto
Add assign and unassign state support for procedures.

M hbase-server/src/main/java/org/apache/hadoop/hbase/client/VersionInfoUtil.java
Adds getting RS version out of RPC
Examples: (1.3.4 is 0x0103004, 2.1.0 is 0x0201000)

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
Remove periodic metrics chore. This is done over in new AM now.
Replace AM with the new. Host the procedures executor.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterMetaBootstrap.java
Have AMv2 handle assigning meta.

M hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
Extract version number of the server making rpc.

A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignProcedure.java
Add new assign procedure. Runs assign via Procedure Dispatch.
There can only be one RegionTransitionProcedure per region running at the time,
since each procedure takes a lock on the region.

D hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignCallable.java
D hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
D hbase-server/src/main/java/org/apache/hadoop/hbase/master/BulkAssigner.java
D hbase-server/src/main/java/org/apache/hadoop/hbase/master/GeneralBulkAssigner.java b/hbase-server/src/main/java/org/apache/hadoop/hbase/master/GeneralBulkAssigner.java
Remove these hacky classes that were never supposed to live longer than
a month or so to be replaced with real assigners.

D hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStateStore.java
D hbase-server/src/main/java/org/apache/hadoop/hbase/master/RegionStates.java
D hbase-server/src/main/java/org/apache/hadoop/hbase/master/UnAssignCallable.java

A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/AssignmentManager.java
A procedure-based AM (AMv2).

TODO
 - handle region migration
 - handle meta assignment first
 - handle sys table assignment first (e.g. acl, namespace)
 - handle table priorities
  "hbase.assignment.bootstrap.thread.pool.size"; default size is 16.
  "hbase.assignment.dispatch.wait.msec"; default wait is 150
  "hbase.assignment.dispatch.wait.queue.max.size"; wait max default is 100
  "hbase.assignment.rit.chore.interval.msec"; default is 5 * 1000;
  "hbase.assignment.maximum.attempts"; default is 10;

 A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/MoveRegionProcedure.java
 Procedure that runs subprocedure to unassign and then assign to new location

 A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStateStore.java
 Manage store of region state (in hbase:meta by default).

 A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionStates.java
 In-memory state of all regions. Used by AMv2.

 A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/RegionTransitionProcedure.java
 Base RIT procedure for Assign and Unassign.

 A hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/UnassignProcedure.java
 Unassign procedure.

 A hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/RSProcedureDispatcher.java
 Run region assignement in a manner that pays attention to target server version.
 Adds "hbase.regionserver.rpc.startup.waittime"; defaults 60 seconds.
2017-05-24 20:47:25 -07:00
Yu Li d047cc9ecc HBASE-18085 Prevent parallel purge in ObjectPool 2017-05-24 15:42:39 +08:00
Josh Elser 56cbc68b0d HBASE-18075 (addendum) Correct javadoc for valid namespaces 2017-05-22 13:28:57 -04:00
Josh Elser 709f5a1980 HBASE-18075 Support non-latin table names and namespaces 2017-05-21 22:24:12 -04:00
tedyu 958cd2d1b7 HBASE-18035 Meta replica does not give any primaryOperationTimeout to primary meta region (huaxiang sun) 2017-05-18 15:56:41 -07:00
tedyu d137991ccc HBASE-18058 Zookeeper retry sleep time should have an upper limit (Allan Yang) 2017-05-18 15:50:59 -07:00
Andrew Purtell 6b60ba8ade HBASE-18043 Institute a hard limit for individual cell size that cannot be overridden by clients 2017-05-15 18:03:33 -07:00
zhangduo 341223d86c HBASE-18012 Move RpcServer.Connection to a separated file 2017-05-15 18:07:38 +08:00
huzheng 4bc0eb31c3 HBASE-17865: Implement async listSnapshot/deleteSnapshot methods.
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2017-04-28 09:09:13 +08:00
Vikas Vishwakarma b81e00f5ea HBASE-17877 Improve HBase's byte[] comparator.
Signed-off-by: Lars Hofhansl <larsh@apache.org>
2017-04-27 13:21:07 -07:00
Balazs Meszaros 2557506415 HBASE-15143 Procedure v2 - Web UI displaying queues
Signed-off-by: Michael Stack <stack@apache.org>
2017-04-25 09:39:28 -07:00
Colm O hEigeartaigh a3b6f4addc HBASE-17944 - Removed unused JDK version parsing from ClassSize.
Signed-off-by: Sean Busbey <busbey@apache.org>
2017-04-21 09:16:23 -05:00
huzheng d39f40e787 HBASE-17864: Implement async snapshot/cloneSnapshot/restoreSnapshot methods
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2017-04-21 18:57:43 +08:00
Ramkrishna 972e8c8c29 HBASE-16438 Create a cell type so that chunk id is embedded in it (Ram) 2017-04-19 15:28:03 +05:30
Ramkrishna ecdfb82326 Revert "HBASE-16438 Create a cell type so that chunk id is embedded in it (Ram)"
This reverts commit c2c2178b2e.
2017-04-18 00:00:12 +05:30
Ramkrishna c2c2178b2e HBASE-16438 Create a cell type so that chunk id is embedded in it (Ram) 2017-04-17 09:28:24 +05:30
Umesh Agashe c8461456d0 HBASE-17888: Added generic methods for updating metrics on submit and finish of a procedure execution
Signed-off-by: Michael Stack <stack@apache.org>
2017-04-14 11:51:08 -07:00
CHIA-PING TSAI 3aadc675b0 HBASE-17896 The FIXED_OVERHEAD of Segment is incorrect 2017-04-12 11:26:43 +08:00
CHIA-PING TSAI df96d328fb HBASE-17872 The MSLABImpl generates the invaild cells when unsafe is not availble 2017-04-09 23:28:34 +08:00
Chia-Ping Tsai 18c5ecf6ed HBASE-17881 Remove the ByteBufferCellImpl 2017-04-07 21:14:19 +08:00