Remove the deprecated fields, which can be removed in 3.0.0. Marked the
constant OLDEST_TIMESTAMP as InterfaceAudience.Private as it is only use
in classes, which are also marked as InterfaceAudience.Private.
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
There are code paths in which we throw non-IOExceptions when
initializing a WAL reader. However, we only close the InputStream to the
WAL filesystem when the exception is an IOException. Close it if it is
open in all cases.
Co-authored-by: Josh Elser <jelser@cloudera.com>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
The issue is that FileInputStream is created with try-with-resources, so its close() is called right after the try sentence.
FileInputStream is a finalize class, when this object is garbage collected, its close() is called again.
To avoid this double-free resources, add guard against it.
Signed-off-by: stack <stack@apache.org>
Don't have every handler update regionserver metrics on each
scan#nextRaw; instead, do a batch update just before Scan
returns. Otherwise, all running handlers end up contending
on metrics update.
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
Update of regionserver metrics counters moved out to caller where
can be done as a batch update instead of per-next.
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServer.java
Class doc to encourage batch updating metrics.
Remove the single update as unused anymore.
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
Count calls to nextRaw. Update regionserver count in finally block when
scan is done rather than per nextRaw call. Move all metrics updates to
finally.
Signed-off-by: Reid Chan <reidchan@apache.org>
Signed-off-by: Baiqiang Zhao <ZhaoBQ>
Introduces a new metric that tracks number of replication sources that are stuck in initialization.
Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Add a toString to all split policy implementations listing name and
vitals. Use this toString in the Region open message. Ditto for flush
policy for the Region.
Signed-off-by: Huaxiang Sun<huaxiangsun@apache.org>
Turn down the amount we log. If you want to see the full exception
enable TRACE-level logging.
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: shahrs87
Close the stream using a try-with-resources block.
Reviewed-by: Aman Poonia <aman.poonia.29@gmail.com>
Signed-off-by: Wei-Chiu Chuang <weichiu@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
ServerCall.java: calling wrapWithSasl() was moved to getResponse(), so
the SASL wrapping is delayed until the reply is sent back to the client.
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
ProtobufLogReader#readNext may be called by code that attempts to advance
the reader but does not necessarily expect to succeed, for example
WALEntryStream#tryAdvanceEntry. Much of the logging in this method is
at TRACE level. Other logging at WARN level will be frequently emitted, as
often as several per minute, and this will cause false positive assessment
from operators that they are experiencing a bug. Fix the mixed intent with
respect to log levels in readNext. Log at only DEBUG level or below.
Signed-off-by: Sean Busbey <busbey@apache.org>
Plumbs the configuration needed to enable core thread timeout on non-critical thread pools.
Currently only enabled for thread pools with op-codes RS_LOG_REPLAY_OPS, RS_PARALLEL_SEEK, MASTER_SNAPSHOT_OPERATIONS, MASTER_MERGE_OPERATIONS. Others can be added later as
needed.
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
* HBASE-25542 Add client detail to scan name so when lease expires, we have clue on who was scanning
When we create a scanner lease, record client ip and port (removed
unnecessary store of scannerName).
Signed-off-by: Clara Xiong <clarax98007@gmail.com>
Retain assignment will be useful in non-cloud scenario where RegionServer and Datanode are deployed in same machine and will avoid remote read.
Signed-off-by: Guanghao Zhang <zghao@apache.org>
Signed-off-by: Anoop Sam John <anoopsamjohn@apache.org>
Adds "hbase.master.executor.merge.dispatch.threads" and defaults to 2.
Also adds additional logging that includes the number of split plans
and merge plans computed for each normalizer run.
Signed-off-by: Wellington Chevreuil <wchevreuil@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Adding StoreContext which contains the metadata about the HStore. This
metadata can be used across the HFileWriter/Readers and other HStore
consumers without the need of passing around the complete store and
exposing its internals.
Co-authored-by: Abhishek Khanna <akkhanna@amazon.com>
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Zach York <zyork@apache.org>
Some OMME can not cause the JVM to exit, like "java.lang.OutOfMemoryError: Direct buffer memory", "java.lang.OutOfMemoryError: unable to create new native thread", as they dont call vmError#next_OnError_command. So abort HMaster when uncaught exception occurs in TimeoutExecutor, the new active Hmaster will resume the suspended procedure.
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: stack <stack@apache.com>
Signed-off-by: Pankaj Kumar<pankajkumar@apache.org>
* HBASE-25379 Make retry pause time configurable for regionserver short operation RPC (reportRegionStateTransition/reportProcedureDone)
* HBASE-25379 RemoteProcedureResultReporter also should retry after the configured pause time
* Addressed the review comments
Signed-off-by: Yulin Niu <niuyulin@apache.org>
* Using ContiguousCellFormat as a marker alone
* Commit the new file
* Fix the comparator logic that was an oversight
* Fix the sequenceId check order
* Adding few more static methods that helps in scan flow like query
matcher where we have more cols
* Remove ContiguousCellFormat and ensure compare() can be inlined
* applying negation as per review comment
* Fix checkstyle comments
* fix review comments
* Address review comments
* Fix the checkstyle issues
* Fix javadoc
Signed-off-by: stack <stack@apache.org>
Signed-off-by: AnoopSamJohn <anoopsamjohn@apache.org>
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
* HBASE-25277 postScannerFilterRow impacts Scan performance a lot in HBase 2.x
1. Added a check for Object class in RegionCoprocessorHost to avoid wrong initialization of hasCustomPostScannerFilterRow
2. Removed dummy implementation of postScannerFilterRow from AccessController, VisibilityController & ConstraintProcessor (which are not required currently)
Signed-off-by Ramkrishna S Vasudevan <ramkrishna@apache.org>
Signed-off-by Anoop Sam John <anoopsamjohn@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Add a bit of a wait before testing if online replicas match the zk count. It might take a
while for all replicas to come online.
Signed-off-by: huaxiangsun <huaxiangsun@apache.org>
Check TEST_SKIP_REPORTING_TRANSITION and if true, skip trying to talk to
update master on state transition -- i.e. reportFileArchivalForQuotas --
as we allow for reportRegionStateTransition (For tests only)
Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
Network identities should be bound late. Remote addresses should be
resolved at the last possible moment, just before connect(). Network
identity mappings can change, so our code should not inappropriately
cache them. Otherwise we might miss a change and fail to operate normally.
Revert "HBASE-14544 Allow HConnectionImpl to not refresh the dns on errors"
Removes hbase.resolve.hostnames.on.failure and related code. We always
resolve hostnames, as late as possible.
Preserve InetSocketAddress caching per RPC connection. Avoids potential
lookups per Call.
Replace InetSocketAddress with Address where used as a map key. If we want
to key by hostname and/or resolved address we should be explicit about it.
Using Address chooses mapping by hostname and port only.
Add metrics for potential nameservice resolution attempts, whenever an
InetSocketAddress is instantiated for connect; and metrics for failed
resolution, whenever InetSocketAddress#isUnresolved on the new instance
is true.
* Use ServerName directly to build a stub key
* Resolve and cache ISA on a RpcChannel as late as possible, at first call
* Remove now invalid unit test TestCIBadHostname
We resolve DNS at the latest possible time, at first call, and do not
resolve hostnames for creating stubs at all, so this unit test cannot
work now.
Reviewed-by: Mingliang Liu <liuml07@apache.org>
Signed-off-by: Duo Zhang <zhangduo@apache.org>
This PR is a follow-up of HBASE-25181 (#2539), where several issues were
discussed on the PR:
1. Currently we use PBKDF2WithHmacSHA1 key generation algorithm to generate a
secret key for HFile / WalFile encryption, when the user is defining a string
encryption key in the hbase shell. This algorithm is not secure enough and
not allowed in certain environments (e.g. on FIPS compliant clusters). We are
changing it to PBKDF2WithHmacSHA384. It will not break backward-compatibility,
as even the tables created by the shell using the new algorithm will be able
to load (e.g. during bulkload / replication) the HFiles serialized with the
key generated by an old algorithm, as the HFiles themselves already contain
the key necessary for their decryption.
Smaller issues fixed by this commit:
2. Improve the documentation e.g. with the changes introduced by HBASE-25181
and also by some points discussed on the Jira ticket of HBASE-25263.
3. In EncryptionUtil.createEncryptionContext the various encryption config
checks should throw IllegalStateExceptions instead of RuntimeExceptions.
4. Test cases in TestEncryptionTest.java should be broken down into smaller
tests.
5. TestEncryptionDisabled.java should use ExpectedException JUnit rule to
validate exceptions.
closes#2676
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
Jan Hentschel
d-c-manning
lujiefsi
Toshihiro Suzuki
Josh Elser
Michael Stack
caroliney14
Andrew Purtell
huaxiangsun
XinSun
Duo Zhang
Baiqiang Zhao
shahrs87
haxiaolin
bsglz
Sandeep Pal
DivyeshChandra
bitterfox
Peter Somogyi
meiyi
Wellington Ramos Chevreuil
Anoop Sam John
Narges Shadab
niuyulin
Bharath Vissapragada
Rahul Kumar
Akshay Sudheer
Balazs Meszaros
pustota2009
GeorryHuang
YutSean
stack
robin7roy
Pankaj
Bo Cui
Aman Poonia
Anjan Das
Christine Feng
Viraj Jasani
Tak Lon (Stephen) Wu
xijiawen
mokai
申胜利
ramkrish86
Huang Zhuoyue
Mohammad Arshad
scotthunt
Sandeep Pal
Huaxiang Sun
Lokesh Khurana
Qi Yu
Mate Szalay-Beko