Commit Graph

310 Commits

Author SHA1 Message Date
Duo Zhang 2fc879e863
HBASE-27434 Use ${revision} as placeholder for maven version to make it easier to control the version from command line (#4836)
Signed-off-by: GeorryHuang <huangzhuoyue@apache.org>
2022-10-24 11:50:48 +08:00
Duo Zhang 63cdd026f0
HBASE-27401 Clean up current broken 'n's in our javadoc (#4812)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-10-06 18:17:34 +08:00
Michael Stack 1b5403cf7d
HBASE-27340 Artifacts with resolved profiles (#4740) (#4761)
(Forward port from branch-2; simplified by the fact that there
is no hadoop-2.0 profile on master branch)

    Make it so our published poms carry the minimum needed to run
    an hbase; the published pom has no profiles -- the profiles
    specified at build time are resolved, their dependencies inlined,
    and then they are stripped -- and no build-time, or plugins
    dependencies or properties, etc. Resultant poms have explicit
    hadoop lib versions baked in -- no more being able to choose
    hbase with hadoop2 or haddop3 at downstream build time by setting
    a '-Dhadoop.profile=X.0'.

    Pattern is to add profiles when none in sub-modules when
    the flatten plugin complains it can't resolve an hadoop
    dependency's 'version' (e.g. hadoop-common, hadoop-hdfs).
    Adding the profile in the sub-module make it so the flatten
    plugin can figure 'hadoop.version' definitively.
    (In master there is only the hadoop-3.0 profile).

    Another spin on the above happens when profiles already exist
    in submodule but the flatten plugin is complaining it can't
    figure figure version on an hadoop dependency NOT under
    profiles. Below, we move the delinquent hadoop dependency under
    existing profiles (minikdc was the usual dependency outside
    profiles in sub-modules that flatten complained about).

    Sometimes, moving an hadoop dependency under a profile, there
    would be excludes on the local dependency. If the parent pom
    excludes section was missing the local excludes, we added them
    up to the parent module so all excluding is done up there in
    the parent profile dependencyManagement section.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-09-06 23:06:13 +08:00
Duo Zhang acf144717b HBASE-27220 Apply the spotless format change in HBASE-27208 to our code base
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-07-19 10:00:43 +08:00
Duo Zhang 9bcb7493e0
HBASE-27165 Set version as 3.0.0-alpha-4-SNAPSHOT in master (#4582)
Signed-off-by: Xin Sun <ddupgs@gmail.com>
2022-06-29 11:04:21 +08:00
Duo Zhang 02990894ab
HBASE-27121 Set version as 3.0.0-alpha-3 in master in prep for first RC of 3.0.0-alpha-3 (#4535)
Signed-off-by: Xin Sun <ddupgs@gmail.com>
2022-06-15 10:51:31 +08:00
Nick Dimiduk ad64a9baae HBASE-26986 Trace a one-shot execution of a Master procedure
This one adds a tracing context to the Master's ProcedureV2 execution. This ties all the spans
that result from a procedure execution back to a procedure by name. Maybe in a follow-on issue, we
can serialize span context along with procedures and display procedure start/stop/retry events and
parent-child hierarchy in tracing visualization.

Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-05-16 11:39:21 +02:00
Duo Zhang 9c8c9e7fbf HBASE-26899 Run spotless:apply
Closes #4312
2022-05-01 22:15:09 +08:00
Andrew Purtell 10471944bd
HBASE-26582 Prune use of Random and SecureRandom objects (#4118)
Avoid the pattern where a Random object is allocated, used once or twice, and
then left for GC. This pattern triggers warnings from some static analysis tools
because this pattern leads to poor effective randomness. In a few cases we were
legitimately suffering from this issue; in others a change is still good to
reduce noise in analysis results.

Use ThreadLocalRandom where there is no requirement to set the seed to gain
good reuse.

Where useful relax use of SecureRandom to simply Random or ThreadLocalRandom,
which are unlikely to block if the system entropy pool is low, if we don't need
crypographically strong randomness for the use case. The exception to this is
normalization of use of Bytes#random to fill byte arrays with randomness.
Because Bytes#random may be used to generate key material it must be backed by
SecureRandom.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
2022-03-08 13:49:02 -08:00
Duo Zhang 4bea1e8c37
HBASE-26723 Switch to use log4j2.properties file to configure log4j2 (#4096)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2022-03-02 21:08:32 +08:00
Duo Zhang 3f59f21be0
HBASE-26621 Set version as 3.0.0-alpha-3-SNAPSHOT in master (#3978)
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2021-12-24 14:20:32 +08:00
Duo Zhang 8bca21b47d
HBASE-26558 Set version as 3.0.0-alpha-2 in master in prep for first RC of 3.0.0-alpha-2 (#3935)
Signed-off-by: Geoffrey Jacoby <gjacoby@apache.org>
2021-12-11 20:52:35 +08:00
Wei-Chiu Chuang c74366c498
HBASE-26049 Remove DfsBuilderUtility (#3444)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-07-26 02:30:22 -07:00
Duo Zhang 16721239e7
HBASE-26100 Set version as 3.0.0-alpha-2-SNAPSHOT in master (#3508)
Signed-off-by: Yulin Niu <niuyulin@apache.org>
2021-07-20 23:04:08 +08:00
Duo Zhang d30cc27097
HBASE-26081 Copy HBTU to hbase-testing-util, rename the HBTU related classes in hbase-server and mark them as IA.LimitedPrivate (#3478)
Signed-off-by: Michael Stack <stack@apache.org>
2021-07-19 09:29:08 +08:00
Duo Zhang 5118321ec9
HBASE-26059 Set version as 3.0.0-alpha-1 in master in prep for first RC of 3.0.0-alpha-1 (#3453)
Signed-off-by: Pankaj Kumar <pankajkumar@apache.org>
2021-07-02 07:50:41 +08:00
Viraj Jasani d11dc81721
HBASE-26012 Improve logging and dequeue logic in DelayQueue (#3397)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2021-06-24 15:27:11 +05:30
Andrew Purtell 335305e0cf
HBASE-25911 Replace calls to System.currentTimeMillis with EnvironmentEdgeManager.currentTime (#3302)
We introduced EnvironmentEdgeManager as a way to inject alternate clocks
for unit tests. In order for this to be effective, all callers that would
otherwise use System.currentTimeMillis() must call
EnvironmentEdgeManager.currentTime() instead, except the implementers of
EnvironmentEdge.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2021-06-01 09:57:48 -07:00
Duo Zhang ba3610d097
HBASE-19577 Use log4j2 instead of log4j for logging (#1708)
Signed-off-by: stack <stack@apache.org>
2021-03-20 09:21:25 +08:00
Bo Cui 600be60a4b
HBASE-25447 remoteProc is suspended due to OOM ERROR (#2824)
Some OMME can not cause the JVM to exit, like "java.lang.OutOfMemoryError: Direct buffer memory", "java.lang.OutOfMemoryError: unable to create new native thread", as they dont call vmError#next_OnError_command. So abort HMaster when uncaught exception occurs in TimeoutExecutor, the new active Hmaster will resume the suspended procedure.

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.com>
Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
2021-01-04 21:04:38 +05:30
Andrew Purtell 1726160839
HBASE-24640 Purge use of VisibleForTesting (#2695)
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-11-25 14:10:12 -08:00
Joseph295 a8096b3ac3
HBASE-25048 [HBCK2] Bypassed parent procedures are not updated in store (#2410)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-10-04 21:39:27 -07:00
Duo Zhang 1bb19e0cdd
HBASE-25037 Lots of thread pool are changed to non daemon after HBASE-24750 which causes trouble when shutting down (#2407)
Signed-off-by: Viraj Jasani <vjasani@apache.org>
2020-09-16 21:11:47 +08:00
Guanghao Zhang 54fe81eb56
HBASE-24912 Enlarge MemstoreFlusherChore/CompactionChecker period for unit test (#2285)
Signed-off-by: stack <stack@apache.org>
2020-08-21 12:41:20 +08:00
Viraj Jasani 0b604d921a
HBASE-24750 : All ExecutorService should use guava ThreadFactoryBuilder
Closes #2196

Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: Ted Yu <tyu@apache.org>
2020-08-07 20:24:36 +05:30
niuyulin 56f32ea895 HBASE-24669 Logging of ppid should be consistent across all occurrences
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-07-29 06:37:34 -07:00
Duo Zhang c303f9d329
HBASE-24408 Introduce a general 'local region' to store data on master (#1753)
Signed-off-by: stack <stack@apache.org>
2020-05-23 15:59:51 +08:00
Duo Zhang 8601416ee8
HBASE-24309 Avoid introducing log4j and slf4j-log4j dependencies for modules other than hbase-assembly (#1640)
Signed-off-by: stack <stack@apache.org>
2020-05-12 12:03:30 +08:00
Duo Zhang 1f66806c96
HBASE-24170 Remove hadoop-2.0 profile (#1495)
Signed-off-by: stack <stack@apache.org>
2020-04-16 18:57:40 +08:00
Duo Zhang f3ee9b8aa3
HBASE-24000 Simplify CommonFSUtils after upgrading to hadoop 2.10.0 (#1335)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Nick Dimiduk <ndimiduk@apache.org>
2020-03-26 17:05:31 +08:00
Duo Zhang c5aa2bc121
HBASE-23077 move entirely to spotbugs (#1265)
Signed-off-by: Sean Busbey <busbey@apache.org>
2020-03-12 10:18:09 +08:00
Michael Stack aaf9a148a0 HBASE-23956 Use less resources running tests (#1266)
Add being able to configure netty thread counts. Enable socket reuse
(should not have any impact).

hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/BlockingRpcConnection.java
 Rename the threads we create in here so they are NOT named same was
 threads created by Hadoop RPC.

hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/DefaultNettyEventLoopConfig.java
hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcClient.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/AsyncFSWAL.java
 Allow configuring eventloopgroup thread count (so can override for
 tests)

hbase-examples/src/main/java/org/apache/hadoop/hbase/client/example/HttpProxyExample.java
 Enable socket resuse.

hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/NettyRpcServer.java
 Enable socket resuse and config for how many threads to use.

hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
hbase-server/src/main/java/org/apache/hadoop/hbase/util/ModifyRegionUtils.java
 Thread name edit; drop the redundant 'Thread' suffix.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/HFileReplicator.java
 Make closeable and shutdown executor when called.

hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSink.java
 Call close on HFileReplicator

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java
 HDFS creates lots of threads. Use less of it so less threads overall.

hbase-server/src/test/resources/hbase-site.xml
hbase-server/src/test/resources/hdfs-site.xml
 Constrain resources when running in test context.

hbase-server/src/test/resources/log4j.properties
 Enable debug on netty to see netty configs in our log

pom.xml
 Add system properties when we launch JVMs to constrain thread counts in
 tests

 Signed-off-by: Duo Zhang <zhangduo@apache.org>
2020-03-11 10:37:50 -07:00
stack 6777e2c2d1 HBASE-23899 [Flakey Test] Stabilizations and Debug
A miscellaney. Add extra logging to help w/ debug to a bunch of tests.
Fix some issues particular where we ran into mismatched filesystem
complaint. Some modernizations, removal of unnecessary deletes
(especially after seeing tests fail in table delete), and cleanup.
Recategorized one tests because it starts four clusters in the one
JVM from  medium to large. Finally, zk standalone server won't come
on occasion; added debug and thread dumping to help figure why (
manifests as test failing in startup saying master didn't launch).

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java
  Fixes occasional mismatched filesystems where the difference is file:// vs file:///
  or we pick up hdfs schema when it a local fs test. Had to do this
  vetting of how we do make qualified on a Path in a few places, not
  just here as a few tests failed with this same issue. Code in here is
  used by a lot of tests that each in turn suffered this mismatch.

  Refactor for clarity

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotV1NoCluster.java
  Unused import.

hbase-procedure/src/test/java/org/apache/hadoop/hbase/procedure2/store/wal/TestWALProcedureStore.java
  This test fails if tmp dir is not where it expects because tries to
  make rootdir there. Give it a rootdir under test data dir.

hbase-server/src/test/java/org/apache/hadoop/hbase/TestZooKeeper.java
  This change is probably useless. I think the issue is actually
  a problem addressed later where our test for zk server being
  up gets stuck and never times out.

hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestSplitOrMergeStatus.java
 Move off deprecated APIs.

hbase-server/src/test/java/org/apache/hadoop/hbase/master/balancer/BalancerTestBase.java
 Log when we fail balance check for DEBUG Currently just says 'false'

hbase-server/src/test/java/org/apache/hadoop/hbase/master/procedure/TestSplitWALProcedure.java
 NPEs on way out if setup failed.

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompaction.java
 Add logging when assert fails to help w/ DEBUG

hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerAbortTimeout.java
 Don't bother removing stuff on teardown. All gets thrown away anyways.
 Saw a few hangs in here in the teardown where hdfs was down before
 expected messing up shutdown.

hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Add timeout on socket; was seeing check for zk server getting stuck
 and never timing out (test time out in startup)

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotWithTemporaryDirectory.java
 Write to test data dir instead.
 Be careful about how we make qualified paths.

hbase-mapreduce/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableInputFormatScanBase.java
 Remove snowflake configs.

hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationStatus.java
 Add a hacky pause. Tried adding barriers but didn't work. Needs deep
 dive.

hbase-zookeeper/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
 Remove code copied from zk and use zk methods directly instead.
 A general problem is that zk cluster doesn't come up occasionally but
 no clue why. Add thread dumping and state check.
2020-02-28 13:39:38 -08:00
binlijin 24823ecfc9
HBASE-23682 Fix NPE when disable DeadServerMetricRegionChore (#1026)
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
2020-02-08 15:20:23 -08:00
Michael Stack 661abeb730 HBASE-23780 Edit of test classifications (#1109)
These classifications come of running at various fork counts.. A test
may complete quick if low fork count but if it is accessing disk, it
will run much slower if fork count is high. This edit accommodates
some of this phenomenon.

Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-02-03 10:28:19 -08:00
Duo Zhang 167892ce64
HBASE-23680 RegionProcedureStore missing cleaning of hfile archive (#1022)
Signed-off-by: stack <stack@apache.org>
2020-01-18 20:30:47 +08:00
Michael Stack 8b7b097905 HBASE-23687 DEBUG logging cleanup (#1040)
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-01-14 22:07:55 -08:00
binlijin 06eff551c3
HBASE-23615 Use a dedicated thread for executing WorkerMonitor in Pro… (#961)
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: virajjasani <34790606+virajjasani@users.noreply.github.com>
2019-12-31 10:03:01 +08:00
Duo Zhang 0ba84d8e95
HBASE-23617 Add a stress test tool for region based procedure store (#962)
Signed-off-by: stack <stack@apache.org>
2019-12-27 22:28:12 +08:00
Duo Zhang 1b049a2d34
HBASE-23326 Implement a ProcedureStore which stores procedures in a HRegion (#941)
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: stack <stack@apache.org>
2019-12-25 12:02:12 +08:00
stack ca6e67a6de HBASE-23315 Miscellaneous HBCK Report page cleanup
* Add a bit of javadoc around SerialReplicationChecker.
 * Miniscule edit to the profiler jsp page and then a bit of doc on how to make it work that might help.
 * Add some detail if NPE getting BitSetNode to help w/ debug.
 * Change HbckChore to log region names instead of encoded names; helps doing diagnostics; can take region name and query in shell to find out all about the region according to hbase:meta.
 * Add some fix-it help inline in the HBCK Report page – how to fix.
 * Add counts in procedures page so can see if making progress; move listing of WALs to end of the page.
2019-11-19 07:34:24 -08:00
belugabehr a3efa5911d HBASE-23102: Improper Usage of Map putIfAbsent (#828)
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
2019-11-17 10:13:52 +00:00
Michael Stack bbfc73789f HBASE-23247 [hbck2] Schedule SCPs for 'Unknown Servers' (#791)
Signed-off-by: Sean Busbey <busbey@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-11-04 09:01:56 -08:00
stack ea5c572963 Revert "HBASE-22917 Proc-WAL roll fails saying someone else has already created log (#544)"
This reverts commit 538a4c51ff.
2019-10-31 08:10:34 -07:00
Pankaj 538a4c51ff HBASE-22917 Proc-WAL roll fails saying someone else has already created log (#544)
Signed-off-by: stack <stack@apache.org>
2019-10-30 12:58:29 -07:00
Karthik Palanisamy 257ccad31c HBASE-23208 Unit formatting in Master & RS UI
Signed-off-by: binlijin <binlijin@gmail.com>
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-10-29 10:18:58 -05:00
Jan Hentschel e3a54e7035
HBASE-22763 Fixed remaining Checkstyle issue in hbase-procedure
Signed-off-by: stack <stack@apache.org>
2019-07-30 09:54:05 +02:00
stack be432b7c45 HBASE-22652 Flakey TestLockManager; test timed out after 780 seconds
Signed-off-by: Duo Zhang <Apache9@apache.org>
2019-07-03 07:49:31 -07:00
stack bdf9d56f2d HBASE-22652 Flakey TestLockManager; test timed out after 780 seconds
Signed-off-by: Sean Busbey <busbey@apache.org>
2019-07-02 22:15:40 -07:00
Andrew Purtell a30b186568
HBASE-22449 https everywhere in Maven metadata (#247) 2019-05-21 12:34:25 -07:00