209 Commits

Author SHA1 Message Date
Nick Dimiduk
158a06d54b
HBASE-23792 [Flakey Test] TestExportSnapshotNoCluster.testSnapshotWithRefsExportFileSystemState (#1125)
1. Survive flakey rerunning by converting the static BeforeClass stuff
   into instance-level Before.
2. Break the test method into two, one for running over each of the
   snapshot manifest versions.

Signed-off-by: stack <stack@apache.org>
2020-02-04 16:08:37 -08:00
Michael Stack
12f4e0977c
HBASE-23780 Edit of test classifications (#1109)
These classifications come of running at various fork counts.. A test
may complete quick if low fork count but if it is accessing disk, it
will run much slower if fork count is high. This edit accommodates
some of this phenomenon.


Signed-off-by: Bharath Vissapragada <bharathv@apache.org>
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-02-03 09:16:47 -08:00
stack
62875a229a Revert "Revert "HBASE-23705 Add CellComparator to HFileContext (#1062)""
Revert of a revert, i.e., putting this patch back.

This reverts commit f4b0ad9f6af455ee889560acb79a0e7c68ab1e2b.
2020-01-27 07:40:56 -08:00
stack
f4b0ad9f6a Revert "HBASE-23705 Add CellComparator to HFileContext (#1062)"
This reverts commit f817293fb4328922462a1ef25d56cdab425853e1.
2020-01-26 20:49:03 -08:00
Michael Stack
f817293fb4 HBASE-23705 Add CellComparator to HFileContext (#1062)
Codecs don't have access to what CellComparator to use.  Backfill.

M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java
 Adds a new compareRows with default implementation that takes a ByteBuffer.
 Needed by the index in a block encoder implementation.

M hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparatorImpl.java
 Adds implementation for meta of new compareRows method. Adds utility
 method for figuring comparator based off tablename.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/AbstractDataBlockEncoder.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/BufferedDataBlockEncoder.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexCodecV1.java
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexSeekerV1.java
 Comparator is in context. Remove redundant handling.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/DataBlockEncoder.java
 Comparator is in context. Remove redundant handling. Clean javadoc.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDecodingContext.java
 Clean javadoc.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/RowIndexEncoderV1.java
 Cache context so can use it to get comparator to use later.

M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContext.java
 Cache cellcomparator to use. Javdoc on diff between HFileContext and
 HFileInfo.
M hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileContextBuilder.java
 Add CellComparator

M hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/HFileOutputFormat2.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderImpl.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterImpl.java
M hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileWriter.java
 Remove comparator caching. Get from context instead.

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/FixedFileTrailer.java
 Skip a reflection if we can.

M hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileInfo.java
 Javadoc. Removed unused filed.

Signed-off-by: Anoop Sam John <anoopsamjohn@apacher.org>
Signed-off-by: Ramkrishna <ramkrishna.s.vasudevan@intel.com>
Signed-off-by: Jan Hentschel <janh@apache.org>
2020-01-22 20:38:24 -08:00
Jan Hentschel
085450ba4e
HBASE-23635 Reduced number of Checkstyle violations in hbase-mapreduce
Signed-off-by: Viraj Jasani <vjasani@apache.org>
Signed-off-by: stack <stack@apache.org>
2020-01-04 01:09:16 +01:00
Sean Busbey
79134c8a3e HBASE-23239 Reporting on status of backing MOB files from client-facing cells (#785)
* Adds a new MapReduce job that builds a report of health of mob files
* Also builds background information on mob system use
* add a basic mob architecture in the ref guide to explain how mob takes the reference hfile value and finds the actual cell contents
* add a troubleshooting mob section to the ref guide to explain how to do a mob reference scan.

Signed-off-by: Peter Somogyi <psomogyi@apache.org>
(cherry picked from commit bc2f16274930e272767ce20ede27eb8fa4117241)
2019-12-16 13:04:10 -06:00
Mingliang Liu
95f25fed86 HBASE-22607. TestExportSnapshotNoCluster fails intermittently 2019-11-20 08:41:07 -08:00
Geoffrey Jacoby
da06aa7f1f HBASE-23251 - Add Column Family and Table Names to HFileContext and use in HFileWriterImpl logging (#796)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: Xu Cang <xucang@apache.org>
Signed-off-by: Zheng Hu <openinx@gmail.com>
2019-11-11 16:22:09 +08:00
chenxu14
235d03bc07 HBASE-22888 Share some stuffs with the initial reader when new stream reader created (#581)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
2019-11-08 14:30:49 +08:00
Shardul Singh
d1864ae8af HBASE-22980 HRegionPartioner getPartition() method incorrectly partitions the regions of the table. (#590)
Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Guangxu Cheng <guangxucheng@gmail.com>
2019-11-07 10:08:03 +08:00
Michael Stack
471538ca9e HBASE-23221 Polish the WAL interface after HBASE-23181 (#774)
Removes the closeRegion flag added by HBASE-23181 and instead
relies on reading meta WALEdit content. Modified how qualifier is
written when the meta WALEdit is for a RegionEventDescriptor
so the 'type' is added to the qualifer so can figure type
w/o having to deserialize protobuf value content: e.g.
HBASE::REGION_EVENT::REGION_CLOSE

Added doc on WALEdit and tried to formalize the 'meta' WALEdit
type and how it works. Needs complete redo in part as suggested
by HBASE-8457. Meantime, some doc and cleanup.

Also changed the LogRoller constructor to remove redundant param.
Because of constructor change, need to change also
TestFailedAppendAndSync, TestWALLockup, TestAsyncFSWAL &
WALPerformanceEvaluation.java

Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Lijin Bin <binlijin@apache.org>
2019-10-30 22:10:51 -07:00
Duo Zhang
3dba799837 HBASE-23181 Blocked WAL archive: "LogRoller: Failed to schedule flush of XXXX, because it is not online on us" (#753)
Signed-off-by: Lijin Bin <binlijin@apache.org>
Signed-off-by: stack <stack@apache.org>
2019-10-26 20:47:29 +08:00
Andrew Purtell
36c4239f37
HBASE-23139 MapReduce jobs lauched from convenience distribution are nonfunctional (#705)
Add hbase-shaded-gson to base job dependencies.

Conflicts:
	hbase-mapreduce/src/main/java/org/apache/hadoop/hbase/mapreduce/TableMapReduceUtil.java
2019-10-08 19:13:48 -07:00
langdamao
f40c745499 HBASE-22887 Fix HFileOutputFormat2 writer roll (#554)
Signed-off-by: langdamao <lang--lang--lang@163.com>
2019-10-08 16:47:49 -07:00
Pankaj
58319b8cc6 HBASE-22928 ScanMetrics counter update may not happen in case of exception in TableRecordReaderImpl
Signed-off-by: Reid Chan <reidchan@apache.org>
2019-09-02 10:48:32 +08:00
Viraj Jasani
e28a7c29c5 HBASE-22863 Cleanup transitive Jackson1 vulnerable dependencies(forward-port HBASE-22728) (#505)
Signed-off-by: Duo Zhang <zhangduo@apache.org>
Signed-off-by: Reid Chan <reidchan@apache.org>
2019-08-21 22:16:52 +08:00
syedmurtazahassan
c5b29265a3
HBASE-22725 Remove all remaining Javadoc warnings
Signed-off-by: stack <stack@apache.org>
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2019-08-11 22:02:53 +02:00
dingwei-2017
6e3b5947a0 HBASE-22773 when set blockSize option in Performance Evaluation tool, error occurs:ERROR: Unrecognized option/command: --blockSize=131072
Signed-off-by: huzheng <openinx@gmail.com>
2019-08-05 11:41:10 +08:00
Wellington Chevreuil
f160dd3c10 HBASE-21773 rowcounter utility should respond to pleas for help
Signed-off-by: stack <stack@apache.org>
2019-07-26 21:29:48 +01:00
syedmurtazahassan
169abb48c4
HBASE-22603 Javadoc Warnings related to @link tag
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2019-07-23 14:33:01 +02:00
syedmurtazahassan
955e643e1d
HBASE-22586 Javadoc Warnings related to @param tag
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2019-07-13 16:56:53 +02:00
syedmurtazahassan
b6e66699a9
HBASE-22571 Javadoc Warnings related to @return tag
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
2019-07-13 13:48:45 +02:00
syedmurtazahassan
8fa7800389 HBASE-22572 Javadoc Warnings: @link reference not found (#306)
Signed-off-by: Jan Hentschel <jan.hentschel@ultratendency.com>
Signed-off-by: stack <stack@apache.org>
2019-07-01 21:18:11 -07:00
Jan Hentschel
6cff25cc23
HBASE-22344 Documented the deprecation of public and limited private APIs (#208)
Signed-off-by: stack <stack@apache.org>
2019-06-17 22:29:51 +02:00
syedmurtazahassan
4400ab1852 HBASE-22481 Javadoc Warnings reference not found 2019-06-08 20:26:25 +02:00
zhangduo
09465f48a8 HBASE-22524 Refactor TestReplicationSyncUpTool 2019-06-04 08:19:00 +08:00
Guanghao
90c13d5970 HBASE-22003 Fix flaky test TestVerifyReplication.testHBase14905 2019-05-28 17:23:43 +08:00
Andrew Purtell
2c55bd9344
HBASE-22449 https everywhere in Maven metadata (#247) 2019-05-21 12:38:42 -07:00
wellington
d4438ae886
HBASE-20305 adding options to skip deletes/puts on target when running SyncTable
Signed-off-by: tedyu <yuzhihong@gmail.com>
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2019-05-14 15:27:09 -07:00
Sean Busbey
4862a596ef HBASE-22083 move eclipse settings into a profile.
Signed-off-by: stack <stack@apache.org>

 Conflicts:
	hbase-backup/pom.xml
	hbase-hadoop-compat/pom.xml
	hbase-protocol/pom.xml
2019-04-25 14:38:38 -05:00
Jan Hentschel
c40e6e2339 HBASE-22231 Removed unused and '*' import 2019-04-23 12:53:52 +02:00
wellington
c2f48e01db
HBASE-20586: add support for clusters on different realms (with cross-realm authentication)
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2019-04-15 11:23:50 -07:00
meiyi
61bc0b4741 HBASE-22084 Rename AccessControlLists to PermissionStorage
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2019-04-12 14:22:57 +08:00
Bahram Chechrazy
86483bf961 HBASE-22194 Snapshot unittests fail on Windows due to invalid file path uri
Signed-off-by: Sergey Shelukhin <sershe@apache.org>
2019-04-11 11:34:27 -07:00
Guanghao
66fffaa6d0 HBASE-22163 Should not archive the compacted store files when region warmup 2019-04-08 22:20:05 +08:00
zhangduo
359cd73e25 HBASE-22099 Backport HBASE-21895 "Error prone upgrade" to branch-2
Signed-off-by: Michael Stack <stack@apache.org>
2019-03-30 14:28:15 +08:00
stack
939a29b41e HBASE-22052 pom cleaning; filter out jersey-core in hadoop2 to match hadoop3 and remove redunant version specifications
This is a reapply of a reverted commit. This commit includes
HBASE-22059 amendment and subsequent ammendments to HBASE-22052.
See HBASE-22052 for full story.

jersey-core is problematic. It was transitively included from hadoop
and polluting our CLASSPATH with an implementation of a 1.x version
of the javax.ws.rs.core.Response Interface from jsr311-api when we
want the javax.ws.rs-api 2.x version.

    M hbase-endpoint/pom.xml
    M hbase-http/pom.xml
    M hbase-mapreduce/pom.xml
    M hbase-rest/pom.xml
    M hbase-server/pom.xml
    M hbase-zookeeper/pom.xml
     Remove redundant version specification (and the odd property define
     done already up in parent pom).
    M hbase-it/pom.xml
    M hbase-rest/pom.xml
     Exclude jersey-core explicitly.

    M hbase-procedure/pom.xml
     Remove redundant version and classifier.

    M pom.xml
     Add jersey-core exclusions to all dependencies that pull it in
     except hadoop-minicluster. mr tests fail w/o the jersey-core
     so let it in for minicluster and then in modules, exclude it
     where it causes damage as in hbase-it.
2019-03-25 09:28:39 -04:00
chenyechao
b6afc5fbc1 HBASE-21810 bulkload support set hfile compression on client
Signed-off-by: Guanghao Zhang <zghao@apache.org>
2019-03-12 17:54:24 +08:00
Wellington Chevreuil
bfe7751814
HBASE-21959 - CompactionTool should close the store it uses for compacting files, in order to properly archive compacted files.
Signed-off-by: Xu Cang <xucang@apache.org>
2019-03-08 14:06:15 -08:00
subrat.mishra
2f9ab7d95a HBASE-21871 Added support to specify a peer table name in VerifyReplication tool
Signed-off-by: Toshihiro Suzuki <brfrn169@gmail.com>
2019-03-06 23:29:08 +09:00
Guanghao Zhang
607ac735c4 HBASE-21922 BloomContext#sanityCheck may failed when use ROWPREFIX_DELIMITED bloom filter 2019-02-23 23:29:53 +08:00
Duo Zhang
761aef6d9d HBASE-20587 Replace Jackson with shaded thirdparty gson
Signed-off-by: Michael Stack <stack@apache.org>
2019-02-22 16:40:45 +08:00
huzheng
df8fa5ef23 HBASE-18484 VerifyRep by snapshot does not work when Yarn/SourceHBase/PeerHBase located in three different HDFS clusters 2019-02-12 11:06:11 +08:00
Toshihiro Suzuki
3902693f61 HBASE-21201 Support to run VerifyReplication MR tool without peerid
Signed-off-by: Josh Elser <elserj@apache.org>
2019-02-10 16:16:37 +09:00
Guanghao Zhang
16665b6e93 HBASE-21799 Update branch-2 version to 2.3.0-SNAPSHOT 2019-01-29 21:53:21 +08:00
zhangduo
2d4819dbed HBASE-21782 LoadIncrementalHFiles should not be IA.Public
Signed-off-by: Michael Stack <stack@apache.org>
2019-01-27 11:26:00 +08:00
Duo Zhang
1ac090b3f0 HBASE-21762 Move some methods in ClusterConnection to Connection
Signed-off-by: Peter Somogyi <psomogyi@apache.org>
2019-01-25 21:14:07 +08:00
huzheng
fdfbd9a21c HBASE-21657 PrivateCellUtil#estimatedSerializedSizeOf has been the bottleneck in 100% scan case 2019-01-16 09:47:27 +08:00
stack
8add7915de HBASE-21646 Flakey TestTableSnapshotInputFormat; DisableTable not completing... Amendment to fix checkstyle complaint
Includes fix for checkstyle complaint.
2018-12-28 14:47:50 -08:00