Commit Graph

186 Commits

Author SHA1 Message Date
huzheng ee19f2b36f HBASE-21445 CopyTable by bulkload will write hfile into yarn's HDFS 2018-11-10 10:19:45 +08:00
ffernandez92 12ea0cf373 HBASE-20766 Typo in VerifyReplication error.
closes #80.

Signed-off-by: Sean Busbey <busbey@apache.org>
(cherry picked from commit 8f8d571779)
2018-09-26 14:55:21 -07:00
Guangxu Cheng fd68e7593e
HBASE-20636 Introduce two bloom filter type : ROWPREFIX and ROWPREFIX_DELIMITED
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Amending-Author: Andrew Purtell <apurtell@apache.org>
2018-09-21 16:06:34 -07:00
Tyler Mi d40348e8cd HBASE-21098 Improve Snapshot Performance with Temporary Snapshot Directory when rootDir on S3
Signed-off-by: Zach York <zyork@apache.org>
Signed-off-by: Mingliang Liu <liuml07@apache.org>
2018-09-11 16:31:56 -07:00
Guanghao Zhang d39be48f04 HBASE-21127 TableRecordReader need to handle cursor result too 2018-09-05 10:41:02 +08:00
Mike Drob 2a9162a503
HBASE-20884 Replace Base64 with j.u.Base64
Signed-off-by: Andrew Purtell <apurtell@apache.org>
Signed-off-by: tedyu <yuzhihong@gmail.com>
2018-07-13 13:00:23 -05:00
jingyuntian a02581e72f HBASE-20769 getSplits() has a out of bounds problem in TableSnapshotInputFormatImpl
Signed-off-by: huzheng <openinx@gmail.com>
2018-06-27 14:31:41 +08:00
huzheng 2c02fb66a0 Revert "HBASE-20769 getSplits() has a out of bounds problem in TableSnapshotInputFormatImpl"
This reverts commit 48cd82e134.
2018-06-27 14:30:46 +08:00
jingyuntian 48cd82e134 HBASE-20769 getSplits() has a out of bounds problem in TableSnapshotInputFormatImpl 2018-06-26 15:35:05 +08:00
Josh Elser 81937df3bb HBASE-20579 Include original exception in wrapped exception
Signed-off-by: Ted Yu <yuzhihong@gmail.com>
2018-06-02 22:16:29 -04:00
tedyu 5a200b5902 HBASE-20668 Avoid permission change if ExportSnapshot's copy fails 2018-06-01 14:39:01 -07:00
jingyuntian c3c9a4a595 HBASE-20579 Improve snapshot manifest copy in ExportSnapshot
Signed-off-by: tedyu <yuzhihong@gmail.com>
2018-05-18 06:43:53 -07:00
Mike Drob 421ed6ca51 HBASE-20521 change getConf as first sequence instead of jobContext in TableOutputFormat.checkOutputSpec, add unit tests 2018-05-08 22:18:48 -05:00
Mike Drob d91908b0d4 Revert "change getConf as first sequence instead of jobContext in TableOutputFormat.checkOutputSpec, add unit tests"
This reverts commit 9e68719014.
2018-05-08 22:18:42 -05:00
michael.jin 9e68719014 change getConf as first sequence instead of jobContext in TableOutputFormat.checkOutputSpec, add unit tests 2018-05-08 10:25:56 -05:00
Peter Somogyi e51ced4f17 HBASE-20376 RowCounter and CellCounter documentations are incorrect 2018-04-12 10:05:45 +02:00
Balazs Meszaros d0060394bf HBASE-20386 [DOC] Align WALPlayer help text and refguide 2018-04-11 09:36:26 -07:00
Yuki Tawara 1ec708bf6c HBASE-20361 Non-successive TableInputSplits may wrongly be merged by auto balancing feature
Signed-off-by: tedyu <yuzhihong@gmail.com>
2018-04-08 10:58:56 -07:00
michael.jin 10ed3f41f5 HBASE-20295 fix NullPointException in TableOutputFormat.checkOutputSpecs 2018-04-07 09:30:52 -07:00
Josh Elser c3d82a283d HBASE-20223 Update to hbase-thirdparty 2.1.0
Remove commons-cli and commons-collections4 use. Account
for the newer internal protobuf version of 3.5.1.

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Mike Drob <mdrob@apache.org>
2018-03-26 16:07:39 -04:00
Jan Hentschel 52014b9423 HBASE-13300 Fixed casing in set/getTimeStamp for Mutations 2018-03-23 17:23:55 -07:00
Chia-Ping Tsai dd9e46bbf5 HBASE-20212 Make all Public classes have InterfaceAudience category
Signed-off-by: tedyu <yuzhihong@gmail.com>
Signed-off-by: Michael Stack <stack@apache.org>
2018-03-22 18:09:54 +08:00
Mike Drob 5e2f8e4023 HBASE-20180 Avoid Class::newInstance 2018-03-14 12:23:38 -05:00
zhangduo b84fbde175 HBASE-19687 Move the logic in ReplicationZKNodeCleaner to ReplicationChecker and remove ReplicationZKNodeCleanerChore 2018-03-09 20:55:48 +08:00
zhangduo 525fef572e HBASE-19633 Clean up the replication queues in the postPeerModification stage when removing a peer 2018-03-09 20:55:48 +08:00
huzheng 1e36a84afc HBASE-19622 Reimplement ReplicationPeers with the new replication storage interface 2018-03-09 20:55:48 +08:00
Guanghao Zhang eae251d203 HBASE-19573 Rewrite ReplicationPeer with the new replication storage interface 2018-03-09 20:55:48 +08:00
Michael Stack 03f9cb89a6 HBASE-20110 Findbugs in zk and mr caused nightly #409 branch-2 to fail 2018-02-28 23:38:35 -08:00
Artem Ervits a5443c18d2 HBASE-20023 CompactionTool command line examples are incorrect
Signed-off-by: tedyu <yuzhihong@gmail.com>
2018-02-20 08:13:59 -08:00
eshcar a458d7c400 HBASE-18294 Reduce global heap pressure: flush based on heap occupancy 2018-02-18 12:58:52 +02:00
Chia-Ping Tsai b0bf6f504e HBASE-19720 Rename WALKey#getTabnename to WALKey#getTableName 2018-02-02 20:02:04 +08:00
huzheng 0b62528dbf HBASE-18452 VerifyReplication by Snapshot should cache HDFS token before submit job for kerberos env 2018-01-04 14:01:41 +08:00
Chia-Ping Tsai 654edc5fa5 HBASE-19596 RegionMetrics/ServerMetrics/ClusterMetrics should apply to all public classes 2018-01-04 13:05:21 +08:00
Michael Stack 4c0fba6a85
HBASE-19588 Additional jar dependencies needed for mapreduce
PerformanceEvaluation

Signed-off-by: Albert Chu <chu11@llnl.gov>
2018-01-03 21:36:05 -06:00
huzheng 16e8422855 HBASE-18806 VerifyRep by snapshot need not to restore snapshot for each mapper 2018-01-04 10:12:53 +08:00
brandboat 9e141d12a4 HBASE-8518 Get rid of hbase.hstore.compaction.complete setting
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-12-30 04:32:12 +08:00
Mike Drob 64cb777a8a HBASE-19552 find-and-replace thirdparty offset 2017-12-28 12:01:25 -06:00
Chia-Ping Tsai 2468f30029 HBASE-19628 ByteBufferCell should extend ExtendedCell 2017-12-27 23:02:18 +08:00
Vasudevan 0454a389a3 HBASE-19629 RawCell#getTags should return the Iterator<Tag> in order to avoid iterating through whole tag array at once
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-12-27 21:30:08 +08:00
Peter Somogyi 15ed748286 HBASE-19545 Replace getBytes(StandardCharsets.UTF_8) with Bytes.toBytes
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-12-27 20:10:50 +08:00
Balazs Meszaros 992b5d8630 HBASE-10092 Move up on to log4j2
Changes:
- replaced commons-logging to slf4j everywhere
- log.XXX(Throwable) calls were replaced with log.XXX(t.toString(), t)
- log.XXX(Object) calls were replaced with log.XXX(Objects.toString(obj))
- log.fatal() calls were replaced with log.error(HBaseMarkers.FATAL, ...)
- programmatic log4j configuration was removed from the unit test

This commit does not affect the current logging configurations, because log4j
is still on the classpath. slf4j-log4j12 binds log4j to slf4j.

Signed-off-by: Michael Stack <stack@apache.org>
2017-12-20 22:58:12 -08:00
Xiang Li c739e3ecec HBASE-15482 Provide an option to skip calculating block locations for SnapshotInputFormat
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-12-19 15:52:42 -08:00
Peter Somogyi dbf0d427e2
HBASE-19549 Change path comparison in CommonFSUtils
Also change makeQualified(FileSystem fs)
to makeQualified(URI defaultUri, Path workingDir)

Signed-off-by: Michael Stack <stack@apache.org>
2017-12-18 15:28:29 -08:00
Michael Stack 495bedea35
HBASE-19122 Suspect methods on Cell to be deprecated 2017-12-18 15:23:03 -08:00
BELUGA BEHR 7c871849de HBASE-19472 Remove ArrayUtil Class
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-12-13 19:08:54 +08:00
Chia-Ping Tsai 4833e63d17 HBASE-19430 Remove the SettableTimestamp and SettableSequenceId 2017-12-07 13:34:38 +08:00
TAK LON WU b2f9b7bc19 HBASE-19023 Replace hbase-server with hbase-mapreduce for HBase and MapReduce chapter
RowCounter and other related HBase's MapReduce classes have been moved
to hbase-mapreduce component by HBASE-18640, related chapter was
out-of-date and this fix replaced hbase-server with hbase-mapreduce
to correct those commands

Also this change moved RowCounter_Counters.properties
to hbase-mapreduce package as well

JIRA https://issues.apache.org/jira/browse/HBASE-19023

Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-12-06 09:04:02 -08:00
Guangxu Cheng 142e6bb9de HBASE-19432 Roll the specified writer in HFileOutputFormat2
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-12-05 22:28:27 -08:00
tedyu 749361e304 HBASE-19355 Missing dependency on hbase-zookeeper module causes CopyTable to fail 2017-11-27 17:40:42 -08:00
Apekshit Sharma e0c4f374b5 HBASE-19114 Split out o.a.h.h.zookeeper from hbase-server and hbase-client
- Moved DrainingServerTracker and RegionServerTracker to hbase-server:o.a.h.h.master.
- Moved SplitOrMergeTracker to oahh.master (because it depends on a PB)
- Moving hbase-client:oahh.zookeeper.*  to hbase-zookeeper module.  After HBASE-19200, hbase-client doesn't need them anymore (except 3 classes).
- Renamed some classes to use a consistent naming for classes - ZK instead of mix of ZK, Zk , ZooKeeper. Couldn't rename following public classes: MiniZooKeeperCluster, ZooKeeperConnectionException. Left RecoverableZooKeeper for lack of better name. (suggestions?)
- Sadly, can't move tests out because they depend on HBaseTestingUtility (which defeats part of the purpose - trimming down hbase-server tests. We need to promote more use of mocks in our tests)
2017-11-17 13:23:28 -08:00
Michael Stack 3ad300a2b0
HBASE-19245 MultiTableInputFormatBase#getSplits creates a Connection per Table
We make one Connection only instead of a Connection per table (Change is
just moving one line it involves right-shifting body of the function)
2017-11-14 21:48:06 -08:00
Tamas Penzes 7a69ebc73e HBASE-18601: Update Htrace to 4.2
Updated HTrace version to 4.2
Created TraceUtil class to wrap htrace methods. Uses try with resources.

Signed-off-by: Balazs Meszaros <balazs.meszaros@cloudera.com>
Signed-off-by: Michael Stack <stack@apache.org>
2017-11-13 10:38:36 -08:00
Mike Drob 0d4f33ca27 HBASE-19195 error-prone fixes for client, mr, and server 2017-11-08 12:04:46 -06:00
tedyu 50f30668b8 HBASE-19160 expose CellComparator as IA.Public - addendum fixes compilation for branch-2 2017-11-06 11:32:26 -08:00
Mike Drob cfddfcf23c HBASE-19160 expose CellComparator as IA.Public 2017-11-06 10:15:55 -06:00
Chia-Ping Tsai 061a73db6d HBASE-19185 ClassNotFoundException: com.fasterxml.jackson.* 2017-11-05 23:31:05 +08:00
Michael Stack f8c58930aa
HBASE-19179 Remove hbase-prefix-tree 2017-11-04 10:09:46 -07:00
tedyu 20b7120afb HBASE-18870 Hbase Backup should set the details to MR job name (Vishal Khandelwal) 2017-10-30 19:57:58 -07:00
Michael Stack 250a8bc7ad
HBASE-18995 Move methods that are for internal usage from CellUtil to Private util class (Ramkrishna Vasudevan) 2017-10-27 17:51:32 -07:00
Tamas Penzes fc581270a8 HBASE-13346: Clean up Filter package for post 1.0 s/KeyValue/Cell/g
Added filterCell method to Filter, it calls filterKeyValue by default
Deprecated filterKeyValue in Filter, bud added default functionality to return Filter.ReturnCode.INCLUDE.
Added filterKeyValue (calling filterCell) to Filters extending FilterBase to be backward compatible.
renamed filterKeyValue to filterCell in all implementations
changed all internal calls to use filterCell instead of filterKeyValue
changed tests too

This way the change is simple and backward compatible.
Any existing custom filter should work since they override filterKeyValue
and the implementation is called by Filter.filterCell.

Moved FilterWrapper to hbase-server

Signed-off-by: anoopsamjohn <anoopsamjohn@gmail.com>
2017-10-27 11:17:53 +05:30
tedyu 998bc32ec9 HBASE-18945 Make a IA.LimitedPrivate interface for CellComparator - addendum fixes compilation for branch-2 2017-10-17 16:30:53 -07:00
Ramkrishna 2cb64fb467 HBSE-18945 Make a IA.LimitedPrivate interface for CellComparator (Ram) 2017-10-17 23:19:21 +05:30
Yu Li cf550c2be1 HBASE-19016 Coordinate storage policy property name for table schema and bulkload 2017-10-17 10:25:27 +08:00
Ramkrishna 66fb60d4a4 HBASE-18649 Deprecate KV Usage in MR to move to Cells in 3.0 (Ram) 2017-10-04 14:44:19 +05:30
Yi Liang 4475ba88c1 HBASE-16894 Create more than 1 split per region, generalize HBASE-12590
Signed-off-by: Andrew Purtell <apurtell@apache.org>
2017-10-03 17:15:39 -07:00
libisthanks 0db82daa91 HBASE-18814 Make ScanMetrics enabled and add counter <HBase Counters, ROWS_SCANNED> into the MapReduce Job over snapshot
Signed-off-by: Ashu Pachauri <ashu@apache.org>
2017-10-02 18:25:34 -07:00
libisthanks 7ee44a820f HBASE-18090 Improve TableSnapshotInputFormat to allow more multiple mappers per region
Signed-off-by: Ashu Pachauri <ashu@apache.org>
2017-09-30 02:11:22 -07:00
Ashu Pachauri c256532ce3 Revert "HBASE-18814 Improve TableSnapshotInputFormat to allow more multiple mappers per region" due to wrong jira id.
This reverts commit 55987efdcf.
2017-09-30 02:00:16 -07:00
libisthanks 55987efdcf HBASE-18814 Improve TableSnapshotInputFormat to allow more multiple mappers per region
Signed-off-by: Ashu Pachauri <ashu@apache.org>
2017-09-29 16:21:51 -07:00
Andy Yang 0658252ed6 HBASE-13844 Move static helper methods from KeyValue into CellUtils
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-09-29 13:20:24 +08:00
Chia-Ping Tsai 6693f45faf HBASE-18839 Apply RegionInfo to code base 2017-09-28 20:19:41 +08:00
zhangduo d26b8f8ddd HBASE-18826 Use HStore instead of Store in our own code base and remove unnecessary methods in Store interface 2017-09-28 15:41:56 +08:00
shaofengshi e97738f0c2 HBASE-18885 HFileOutputFormat2 hardcodes default FileOutputCommitter
Signed-off-by: tedyu <yuzhihong@gmail.com>
2017-09-26 19:58:51 -07:00
zhangduo 06d8a23531 HBASE-18825 Use HStoreFile instead of StoreFile in our own code base and remove unnecessary methods in StoreFile interface 2017-09-25 09:48:33 +08:00
Yi Liang a66f9dcda5
HBASE-18803: Mapreduce job get failed caused by NoClassDefFoundError: org/apache/commons/lang3/ArrayUtils
Signed-off-by: Michael Stack <stack@apache.org>
2017-09-18 12:36:05 -07:00
zhangduo 46d993b0ae HBASE-18453 CompactionRequest should not be exposed to user directly 2017-09-14 20:37:46 +08:00
Michael Stack 274bbb64c6 HBASE-14998 Unify synchronous and asynchronous methods in Admin and cleanup
This is based on patch sent me by Balazs Meszaros. The good stuff in
here is from him. This patch does less than his ambition. It changes
Admin class only. Can work on making AsyncAdmin cohere in a follow-on.

 * Deprecates getAlterStatus. Everywhere else we talk of 'modify' rather
   'alter' and should use Future returned from async instead.
 * isTableAvailable(TableName, byte [][]) has been deprecated to be
   removed; use the overrie instead. This is a weird method.
 * Changed listTableDescriptor to getDescriptor.
 * Renamed other like methods to have same pattern (deprecating the old):
    balancer => balance
    setBalancerRunning => balancerSwitch
    setNormalizerRunning => normalizerSwitch
    enableCatalogJanitor => catalogJanitorSwitch
    setCleanerChoreRunning => cleanerChoreSwitch
    setSplitOrMergeEnabled => splitOrMergeEnabledSwitch

 * Renamed (with deprecation of old) runCatalogScan => runCatalogJanitor.
 * Reviewed generated javadoc and made some edits; purged reference to
   hbase issues from our API, fixed param names, etc.
 * Made all the enable services methods have same pattern.
 * Renamed takeSnapshotAsync as snapshotAsync (with deprecation of old)
 * Renamed execProcedureWithRet as execProcedureWithReturn (with
   deprecation)

Signed-off-by: Michael Stack <stack@apache.org>
2017-09-13 17:19:56 -07:00
Sean Busbey d576e5a32d HBASE-17823 Migrate to Apache Yetus Audience Annotations
Includes partial backport of hbase-build-configuration module

Signed-off-by: Michael Stack <stack@apache.org>
Signed-off-by: Misty Stanley-Jones <misty@apache.org>
2017-09-12 23:15:50 -05:00
Michael Stack 1d278d3de6 HBASE-16479 Move WALEdit from hbase.regionserver.wal package to hbase.wal package 2017-09-11 14:42:35 -07:00
Michael Stack 7f70b6c8e3 HBASE-18769 Make CompareFilter use generic CompareOperator instead of
internal enum
2017-09-08 10:05:51 -07:00
Umesh Agashe bd219c0fb8 HBASE-18674 upgrading to commons-lang3
Signed-off-by: Michael Stack <stack@apache.org>
2017-09-05 09:46:49 -07:00
Chun-Hao Tang 27db1298f7 HBASE-18746 Throw exception with job.getStatus().getFailureInfo() when ExportSnapshot fails
Signed-off-by: Chia-Ping Tsai <chia7712@gmail.com>
2017-09-05 02:05:11 +08:00
zhangduo a37417c254 HBASE-18699 Copy LoadIncrementalHFiles to another package and mark the old one as deprecated 2017-09-03 19:49:47 +08:00
Chia-Ping Tsai 96de8d71fc HBASE-15806 An endpoint-based export tool 2017-08-30 14:07:06 +08:00
Michael Stack 74c517a8bc HBASE-18698 MapreduceDependencyClasspathTool does not include hbase-server as a dependency
Move Driver to be the main-class in hbase-mapreduce jar rather than
in the hbase-server jar.

Reference the hbase-server and shaded protobuf so they get bundled
when you do 'hbase mapredcp'.
2017-08-29 13:10:24 -07:00
Apekshit Sharma 59d034106f HBASE-18640 Move mapreduce out of hbase-server into separate module.
- Moves out o.a.h.h.{mapred, mapreduce} to new hbase-mapreduce module which depends
  on hbase-server because of classes like *Snapshot{Input,Output}Format.java, WALs, replication, etc
- hbase-backup depends on it for WALPlayer and MR job stuff
- A bunch of tools needed to be pulled into hbase-mapreduce becuase of their dependencies on MR.
  These are: CompactionTool, LoadTestTool, PerformanceEvaluation, ExportSnapshot
  This is better place of them than hbase-server. But ideal place would be in separate hbase-tools module.
- There were some tests in hbase-server which were digging into these tools for static util funtions or
  confs. Moved these to better/easily shared place. For eg. security related stuff to HBaseKerberosUtils.
- Note that hbase-mapreduce has secondPartExecution tests. On my machine they took like 20 min, so maybe
  more on apache jenkins. That's basically equal reduction of runtime of hbase-server tests, which is a
  big win!

Change-Id: Ieeb7235014717ca83ee5cb13b2a27fddfa6838e8
2017-08-26 01:55:33 -07:00