Running `ServerKillingChaosMonkey` via `RESTApiClusterManager` for any
duration of time slowly leaks region servers. I see failures on the
RESTApi side go unreported on the ChaosMonkey side. It seems like
`RuntimeException`s are being thrown and lost.
`PolicyBasedChaosMonkey` uses a primitive means of thread management
anyway. Update to use a thread pool, thread groups, and an
uncaughtExceptionHandler.
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
* Specify the version
* by using apt-get instead of pip install
* Remove comment blocks
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Lars Hofhansl <larsh@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Writing a test for this is tricky. There is enough coverage for
functional tests. Only concern is performance, but there is enough
logging for it to detect timed out/badly performing sync calls.
Additionally, this patch decouples the ZK event processing into it's
own thread rather than doing it in the EventThread's context. That
avoids deadlocks and stalls of the event thread.
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
(cherry picked from commit 84e246f9b1)
(cherry picked from commit 2379a25f0c)
`RollingBatchRestartRsAction` doesn't handle failure cases when
tracking its list of dead servers. The original author believed that a
failure to restart would result in a retry. However, by removing the
dead server from the failed list, that state is lost, and retry never
occurs. Because this action doesn't ever look back to the current
state of the cluster, relying only on its local state for the current
action invocation, it never realizes the abandoned server is still
dead. Instead, be more careful to only remove the dead server from the
list when the `startRs` invocation claims to have been successful.
Signed-off-by: stack <stack@apache.org>
(cherry picked from commit 0dae377f53)
* Add chaos monkey action for suspend/resume region servers
* Add these to relevant chaos monkeys
branch-1-backport-note: Graceful regionserver restart action wasn't
backported due to a dependency of "RegionMover" script. Can be done
later if needed.
Signed-off-by: Balazs Meszaros <meszibalu@apache.org>
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
While we never expect table descriptors to be missing, a corrupt meta
can result in the master crashing before regions get assigned. We can
guard against that happening with a simple null-check.
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Closes#1908
Implements `ClusterManager` that relies on the new
`ShellExecEndpointCoprocessor` for remote shell command execution.
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Co-authored-by: Nick Dimiduk <ndimiduk@apache.org>
This one surfaced as a flake test but turns out to be a legit bug
in FIFOCompaction code. FifoCompaction does not check if an empty
store file is already being compacted by an in-flight compaction
request and still enqueues. It can potentially race with a running
compaction (as in this test case, see jira for the exact exception).
Fixes the edge case and cleans up the test code a bit.
Signed-off-by: Andrew Purtell <apurtell@apache.org>
HBASE-24479: Deflake TestCompaction#testStopStartCompaction
Polling of active compaction count is racy. Tightened the asserts
to be more reliable.
Reid Chan <reidchan@apache.org>
This utility is useful for any module that wants to detect
dynamic config changes. Having it to hbase-common makes it
accessible to all the other modules.
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Nick Dimiduk
Mohammad Arshad
Zheng Wang
sguggilam
WenFeiYi
lhofhansl
Reid Chan
Bo Cui
Geoffrey Jacoby
David Manning
Bharath Vissapragada
Abhishek Singh Chouhan
Viraj Jasani
BukrosSzabolcs
Josh Elser
satanson
Anoop Sam John
Andrew Purtell
thangTang
Ravi Kishore Valeti
jacob-leblanc
xincunSong
wenbang
Javier Akira Luca de Tena
Alexander Myasnikov
niuyulin
Sandeep Pal