hbase/CHANGES.md

39 KiB
Executable File

HBASE Changelog

Release 2.1.0 - Unreleased (as of 2018-07-10)

INCOMPATIBLE CHANGES:

JIRA Summary Priority Component
HBASE-20691 Storage policy should allow deferring to HDFS Blocker Filesystem Integration, wal
HBASE-20270 Turn off command help that follows all errors in shell Major shell
HBASE-20501 Change the Hadoop minimum version to 2.7.1 Blocker community, documentation
HBASE-20406 HBase Thrift HTTP - Shouldn't handle TRACE/OPTIONS methods Major security, Thrift
HBASE-20159 Support using separate ZK quorums for client Major Client, Operability, Zookeeper
HBASE-20148 Make serial replication as a option for a peer instead of a table Major Replication

NEW FEATURES:

JIRA Summary Priority Component
HBASE-20833 Modify pre-upgrade coprocessor validator to support table level coprocessors Major Coprocessors
HBASE-15809 Basic Replication WebUI Critical Replication, UI
HBASE-19735 Create a minimal "client" tarball installation Major build, Client
HBASE-20656 Validate pre-2.0 coprocessors against HBase 2.0+ Major tooling
HBASE-20592 Create a tool to verify tables do not have prefix tree encoding Minor Operability, tooling
HBASE-20046 Reconsider the implementation for serial replication Major Replication
HBASE-19397 Design procedures for ReplicationManager to notify peer change event from master Major proc-v2, Replication

IMPROVEMENTS:

JIRA Summary Priority Component
HBASE-20806 Split style journal for flushes and compactions Minor .
HBASE-20474 Show non-RPC tasks on master/regionserver Web UI by default Major UI
HBASE-20826 Truncate responseInfo attributes on RpcServer WARN messages Major rpc
HBASE-20450 Provide metrics for number of total active, priority and replication rpc handlers Major metrics
HBASE-20810 Include the procedure id in the exception message in HBaseAdmin for better debugging Major Admin, proc-v2
HBASE-20040 Master UI should include "Cluster Key" needed to use the cluster as a replication sink Minor Replication, Usability
HBASE-20095 Redesign single instance pool in CleanerChore Critical .
HBASE-19164 Avoid UUID.randomUUID in tests Major test
HBASE-20739 Add priority for SCP Major Recovery
HBASE-20737 put collection into ArrayList instead of addAll function Trivial .
HBASE-20695 Implement table level RegionServer replication metrics Minor metrics
HBASE-20733 QABot should run checkstyle tests if the checkstyle configs change Minor build, community
HBASE-20625 refactor some WALCellCodec related code Minor wal
HBASE-19852 HBase Thrift 1 server SPNEGO Improvements Major Thrift
HBASE-20579 Improve snapshot manifest copy in ExportSnapshot Minor mapreduce
HBASE-20444 Improve version comparison logic for HBase specific version string and add unit tests Major util
HBASE-20594 provide utility to compare old and new descriptors Major .
HBASE-20640 TestQuotaGlobalsSettingsBypass missing test category and ClassRule Critical test
HBASE-20478 move import checks from hbaseanti to checkstyle Minor test
HBASE-20548 Master fails to startup on large clusters, refreshing block distribution Major .
HBASE-20488 PE tool prints full name in help message Minor shell
HBASE-20567 Pass both old and new descriptors to pre/post hooks of modify operations for table and namespace Major .
HBASE-20545 Improve performance of BaseLoadBalancer.retainAssignment Major Balancer
HBASE-16191 Add stop_regionserver and stop_master to shell Major .
HBASE-20536 Make TestRegionServerAccounting stable and it should not use absolute number Minor .
HBASE-20523 PE tool should support configuring client side buffering sizes Minor .
HBASE-20527 Remove unused code in MetaTableAccessor Trivial .
HBASE-20507 Do not need to call recoverLease on the broken file when we fail to create a wal writer Major wal
HBASE-20484 Remove the unnecessary autoboxing in FilterListBase Trivial .
HBASE-20327 When qualifier is not specified, append and incr operation do not work (shell) Minor shell
HBASE-20389 Move website building flags into a profile Minor build, website
HBASE-20379 shadedjars yetus plugin should add a footer link Major test
HBASE-20243 [Shell] Add shell command to create a new table by cloning the existent table Minor shell
HBASE-20286 Improving shell command compaction_state Minor shell
HBASE-19488 Move to using Apache commons CollectionUtils Trivial .
HBASE-20197 Review of ByteBufferWriterOutputStream.java Minor .
HBASE-20047 AuthenticationTokenIdentifier should provide a toString Minor Usability
HBASE-19024 Configurable default durability for synchronous WAL Critical wal
HBASE-19389 Limit concurrency of put with dense (hundreds) columns to prevent write handler exhausted Critical Performance
HBASE-20186 Improve RSGroupBasedLoadBalancer#balanceCluster() to be more efficient when calculating cluster state for each rsgroup Minor rsgroup
HBASE-19449 Minor logging change in HFileArchiver Trivial .
HBASE-20120 Remove some unused classes/ java files from hbase-server Minor .

BUG FIXES:

JIRA Summary Priority Component
HBASE-20854 Wrong retries number in RpcRetryingCaller's log message Minor Client, logging
HBASE-20784 Will lose the SNAPSHOT suffix if we get the version of RS from ServerManager Minor master, UI
HBASE-20822 TestAsyncNonMetaRegionLocator is flakey Major asyncclient
HBASE-20808 Wrong shutdown order between Chores and ChoreService Minor .
HBASE-20789 TestBucketCache#testCacheBlockNextBlockMetadataMissing is flaky Major .
HBASE-20829 Remove the addFront assertion in MasterProcedureScheduler.doAdd Major Replication
HBASE-20825 Fix pre and post hooks of CloneSnapshot and RestoreSnapshot for Access checks Major security
HBASE-20812 Add defaults to Table Interface so implementors don't have to Major .
HBASE-20817 Infinite loop when executing ReopenTableRegionsProcedure Blocker Region Assignment
HBASE-20792 info:servername and info:sn inconsistent for OPEN region Blocker Region Assignment
HBASE-20769 getSplits() has a out of bounds problem in TableSnapshotInputFormatImpl Major .
HBASE-20732 Shutdown scan pool when master is stopped. Minor .
HBASE-20785 NPE getting metrics in PE testing scans Major Performance
HBASE-20795 Allow option in BBKVComparator.compare to do comparison without sequence id Major .
HBASE-20777 RpcConnection could still remain opened after we shutdown the NettyRpcServer Major rpc
HBASE-20403 Prefetch sometimes doesn't work with encrypted file system Major .
HBASE-20635 Support to convert the shaded user permission proto to client user permission object Major .
HBASE-20778 Make it so WALPE runs on DFS Major test
HBASE-20775 TestMultiParallel is flakey Major Region Assignment
HBASE-20752 Make sure the regions are truly reopened after ReopenTableRegionsProcedure Major proc-v2
HBASE-18622 Mitigate API compatibility concerns between branch-1 and branch-2 Blocker API
HBASE-20767 Always close hbaseAdmin along with connection in HBTU Major test
HBASE-20642 IntegrationTestDDLMasterFailover throws 'InvalidFamilyOperationException Major .
HBASE-20742 Always create WAL directory for region server Major wal
HBASE-20708 Remove the usage of RecoverMetaProcedure in master startup Blocker proc-v2, Region Assignment
HBASE-20723 Custom hbase.wal.dir results in data loss because we write recovered edits into a different place than where the recovering region server looks for them Critical Recovery, wal
HBASE-20681 IntegrationTestDriver fails after HADOOP-15406 due to missing hamcrest-core Major integration tests
HBASE-20561 The way we stop a ReplicationSource may cause the RS down Major Replication
HBASE-19377 Compatibility checker complaining about hash collisions Major community
HBASE-20689 Docker fails to install rubocop for precommit Blocker build
HBASE-20707 Move MissingSwitchDefault check from checkstyle to error-prone Major build
HBASE-20699 QuotaCache should cancel the QuotaRefresherChore service inside its stop() Major .
HBASE-20590 REST Java client is not able to negotiate with the server in the secure mode Critical REST, security
HBASE-20683 Incorrect return value for PreUpgradeValidator Critical .
HBASE-20684 org.apache.hadoop.hbase.client.Scan#setStopRow javadoc uses incorrect method Trivial Client, documentation
HBASE-20678 NPE in ReplicationSourceManager#NodeFailoverWorker Minor .
HBASE-20670 NPE in HMaster#isInMaintenanceMode Minor .
HBASE-20634 Reopen region while server crash can cause the procedure to be stuck Critical .
HBASE-12882 Log level for org.apache.hadoop.hbase package should be configurable Major .
HBASE-20668 Avoid permission change if ExportSnapshot's copy fails Major .
HBASE-18116 Replication source in-memory accounting should not include bulk transfer hfiles Major Replication
HBASE-20602 hbase.master.quota.observer.ignore property seems to be not taking effect Minor documentation
HBASE-20664 Variable shared across multiple threads Major .
HBASE-20659 Implement a reopen table regions procedure Major .
HBASE-20582 Bump up JRuby version because of some reported vulnerabilities Major dependencies, shell
HBASE-20533 Fix the flaky TestAssignmentManagerMetrics Major .
HBASE-20597 Use a lock to serialize access to a shared reference to ZooKeeperWatcher in HBaseReplicationEndpoint Minor Replication
HBASE-20633 Dropping a table containing a disable violation policy fails to remove the quota upon table delete Major .
HBASE-20645 Fix security_available method in security.rb Major .
HBASE-20612 TestReplicationKillSlaveRSWithSeparateOldWALs sometimes fail because it uses an expired cluster conn Major .
HBASE-20648 HBASE-19364 "Truncate_preserve fails with table when replica region > 1" for master branch Major .
HBASE-20588 Space quota change after quota violation doesn't seem to take in effect Major regionserver
HBASE-20616 TruncateTableProcedure is stuck in retry loop in TRUNCATE_TABLE_CREATE_FS_LAYOUT state Major amv2
HBASE-20638 nightly source artifact testing should fail the stage if it's going to report an error on jira Major test
HBASE-20624 Race in ReplicationSource which causes walEntryFilter being null when creating new shipper Major Replication
HBASE-20601 Add multiPut support and other miscellaneous to PE Minor tooling
HBASE-20627 Relocate RS Group pre/post hooks from RSGroupAdminServer to RSGroupAdminEndpoint Major .
HBASE-20591 nightly job doesn't respect maven options Critical test
HBASE-20560 Revisit the TestReplicationDroppedTables ut Major .
HBASE-20571 JMXJsonServlet generates invalid JSON if it has NaN in metrics Major UI
HBASE-20585 Need to clear peer map when clearing MasterProcedureScheduler Major proc-v2
HBASE-20457 Return immediately for a scan rpc call when we want to switch from pread to stream Major scan
HBASE-20447 Only fail cacheBlock if block collisions aren't related to next block metadata Major BlockCache, BucketCache
HBASE-20544 downstream HBaseTestingUtility fails with invalid port Blocker test
HBASE-20004 Client is not able to execute REST queries in a secure cluster Minor REST, security
HBASE-20475 Fix the flaky TestReplicationDroppedTables unit test. Major .
HBASE-20554 "WALs outstanding" message from CleanerChore is noisy Trivial .
HBASE-20204 Add locking to RefreshFileConnections in BucketCache Major BucketCache
HBASE-20485 Copy constructor of Scan doesn't copy the readType and replicaId Minor .
HBASE-20543 Fix the flaky TestThriftHttpServer Major .
HBASE-20521 TableOutputFormat.checkOutputSpecs conf checking sequence cause pig script run fail Major mapreduce
HBASE-20500 [rsgroup] should keep at least one server in default group Major rsgroup
HBASE-20517 Fix PerformanceEvaluation 'column' parameter Major test
HBASE-20524 Need to clear metrics when ReplicationSourceManager refresh replication sources Minor .
HBASE-20476 Open sequence number could go backwards in AssignProcedure Major Region Assignment
HBASE-20506 Add doc and test for unused RetryCounter, useful-looking utility Minor .
HBASE-20492 UnassignProcedure is stuck in retry loop on region stuck in OPENING state Critical amv2
HBASE-20497 The getRecoveredQueueStartPos always return 0 in RecoveredReplicationSourceShipper Major Replication
HBASE-18842 The hbase shell clone_snaphost command returns bad error message Minor shell
HBASE-20466 Consistently use override mechanism for exempt classes in CoprocessClassloader Major Coprocessors
HBASE-20006 TestRestoreSnapshotFromClientWithRegionReplicas is flakey Critical read replicas
HBASE-18059 The scanner order for memstore scanners are wrong Critical regionserver, scan, Scanners
HBASE-20404 Ugly cleanerchore complaint that dir is not empty Major master
HBASE-20419 Fix potential NPE in ZKUtil#listChildrenAndWatchForNewChildren callers Major .
HBASE-20364 nightly job gives old results or no results for stages that timeout on SCM Critical test
HBASE-20335 nightly jobs no longer contain machine information Critical test
HBASE-20338 WALProcedureStore#recoverLease() should have fixed sleeps for retrying rollWriter() Major .
HBASE-20356 make skipping protoc possible Critical dependencies, thirdparty
HBASE-15291 FileSystem not closed in secure bulkLoad Major .
HBASE-20068 Hadoopcheck project health check uses default maven repo instead of yetus managed ones Major community, test
HBASE-20361 Non-successive TableInputSplits may wrongly be merged by auto balancing feature Major mapreduce
HBASE-20260 Purge old content from the book for branch-2/master Critical documentation
HBASE-20058 improper quoting in presplitting command docs Minor documentation
HBASE-19923 Reset peer state and config when refresh replication source failed Major Replication
HBASE-19748 TestRegionReplicaFailover and TestRegionReplicaReplicationEndpoint UT hangs Major .

TESTS:

JIRA Summary Priority Component
HBASE-20667 Rename TestGlobalThrottler to TestReplicationGlobalThrottler Trivial .
HBASE-20646 TestWALProcedureStoreOnHDFS failing on branch-1 Trivial .
HBASE-20505 PE should support multi column family read and write cases Minor .
HBASE-20513 Collect and emit ScanMetrics in PerformanceEvaluation Minor test
HBASE-20414 TestLockProcedure#testMultipleLocks may fail on slow machine Major .

SUB-TASKS:

JIRA Summary Priority Component
HBASE-20831 Copy master doc into branch-2.1 and edit to make it suit 2.1.0 Blocker documentation
HBASE-20839 Fallback to FSHLog if we can not instantiated AsyncFSWAL when user does not specify AsyncFSWAL explicitly Blocker wal
HBASE-20244 NoSuchMethodException when retrieving private method decryptEncryptedDataEncryptionKey from DFSClient Blocker wal
HBASE-20193 Basic Replication Web UI - Regionserver Critical Replication, Usability
HBASE-20489 Update Reference Guide that CLUSTER_KEY value is present on the Master UI info page. Minor documentation
HBASE-19722 Meta query statistics metrics source Major .
HBASE-20781 Save recalculating families in a WALEdit batch of Cells Major Performance
HBASE-20194 Basic Replication WebUI - Master Critical Replication, Usability
HBASE-20780 ServerRpcConnection logging cleanup Major logging, Performance
HBASE-19764 Fix Checkstyle errors in hbase-endpoint Minor .
HBASE-20710 extra cloneFamily() in Mutation.add(Cell) Minor regionserver
HBASE-18569 Add prefetch support for async region locator Major asyncclient, Client
HBASE-20706 [hack] Don't add known not-OPEN regions in reopen phase of MTP Critical amv2
HBASE-20334 add a test that expressly uses both our shaded client and the one from hadoop 3 Major hadoop3, shading
HBASE-20615 emphasize use of shaded client jars when they're present in an install Major build, Client, Usability
HBASE-20333 break up shaded client into one with no Hadoop and one that's standalone Critical shading
HBASE-20332 shaded mapreduce module shouldn't include hadoop Critical mapreduce, shading
HBASE-20722 Make RegionServerTracker only depend on children changed event Major .
HBASE-20700 Move meta region when server crash can cause the procedure to be stuck Critical master, proc-v2, Region Assignment
HBASE-20628 SegmentScanner does over-comparing when one flushing Critical Performance
HBASE-19761 Fix Checkstyle errors in hbase-zookeeper Minor .
HBASE-19724 Fix Checkstyle errors in hbase-hadoop2-compat Minor .
HBASE-20518 Need to serialize the enabled field for UpdatePeerConfigProcedure Major Replication
HBASE-20481 Replicate entries from same region serially in ReplicationEndpoint for serial replication Major .
HBASE-20378 Provide a hbck option to cleanup replication barrier for a table Major .
HBASE-20128 Add new UTs which extends the old replication UTs but set replication scope to SERIAL Major .
HBASE-20417 Do not read wal entries when peer is disabled Major Replication
HBASE-20294 Also cleanup last pushed sequence id in ReplicationBarrierCleaner Major Replication
HBASE-20377 Deal with table in enabling and disabling state when modifying serial replication peer Major Replication
HBASE-20367 Write a replication barrier for regions when disabling a table Major Replication
HBASE-20296 Remove last pushed sequence ids when removing tables from a peer Major Replication
HBASE-20285 Delete all last pushed sequence ids when removing a peer or removing the serial flag for a peer Major Replication
HBASE-20138 Find a way to deal with the conflicts when updating replication position Major Replication
HBASE-20127 Add UT for serial replication after failover Major Replication, test
HBASE-20271 ReplicationSourceWALReader.switched should use the file name instead of the path object directly Major Replication
HBASE-20227 Add UT for ReplicationUtils.contains method Major Replication, test
HBASE-20147 Serial replication will be stuck if we create a table with serial replication but add it to a peer after there are region moves Major .
HBASE-20116 Optimize the region last pushed sequence id layout on zk Major Replication
HBASE-20242 The open sequence number will grow if we fail to open a region after writing the max sequence id file Major .
HBASE-20155 update branch-2 version to 2.1.0-SNAPSHOT Major build, community
HBASE-20206 WALEntryStream should not switch WAL file silently Major Replication
HBASE-20117 Cleanup the unused replication barriers in meta table Major master, Replication
HBASE-20165 Shell command to make a normal peer to be a serial replication peer Major .
HBASE-20167 Optimize the implementation of ReplicationSourceWALReader Major Replication
HBASE-20125 Add UT for serial replication after region split and merge Major Replication
HBASE-20129 Add UT for serial replication checker Major Replication
HBASE-20115 Reimplement serial replication based on the new replication storage layer Major Replication
HBASE-20050 Reimplement updateReplicationPositions logic in serial replication based on the newly introduced replication storage layer Major .
HBASE-20082 Fix findbugs errors only on master which are introduced by HBASE-19397 Major findbugs
HBASE-19936 Introduce a new base class for replication peer procedure Major .
HBASE-19719 Fix checkstyle issues Major proc-v2, Replication
HBASE-19711 TestReplicationAdmin.testConcurrentPeerOperations hangs Major proc-v2
HBASE-19707 Race in start and terminate of a replication source after we async start replicatione endpoint Major proc-v2, Replication
HBASE-19636 All rs should already start work with the new peer change when replication peer procedure is finished Major proc-v2, Replication
HBASE-19634 Add permission check for executeProcedures in AccessController Major proc-v2, Replication
HBASE-19697 Remove TestReplicationAdminUsingProcedure Major proc-v2, Replication
HBASE-19661 Replace ReplicationStateZKBase with ZKReplicationStorageBase Major proc-v2, Replication
HBASE-19687 Move the logic in ReplicationZKNodeCleaner to ReplicationChecker and remove ReplicationZKNodeCleanerChore Major proc-v2, Replication
HBASE-19544 Add UTs for testing concurrent modifications on replication peer Major proc-v2, Replication, test
HBASE-19686 Use KeyLocker instead of ReentrantLock in PeerProcedureHandlerImpl Major proc-v2, Replication
HBASE-19623 Create replication endpoint asynchronously when adding a replication source Major proc-v2, Replication
HBASE-19633 Clean up the replication queues in the postPeerModification stage when removing a peer Major proc-v2, Replication
HBASE-19622 Reimplement ReplicationPeers with the new replication storage interface Major proc-v2, Replication
HBASE-19635 Introduce a thread at RS side to call reportProcedureDone Major proc-v2
HBASE-19617 Remove ReplicationQueues, use ReplicationQueueStorage directly Major Replication
HBASE-19642 Fix locking for peer modification procedure Critical proc-v2, Replication
HBASE-19592 Add UTs to test retry on update zk failure Major proc-v2, Replication
HBASE-19630 Add peer cluster key check when add new replication peer Major proc-v2, Replication
HBASE-19573 Rewrite ReplicationPeer with the new replication storage interface Major proc-v2, Replication
HBASE-19579 Add peer lock test for shell command list_locks Major proc-v2, Replication
HBASE-19599 Remove ReplicationQueuesClient, use ReplicationQueueStorage directly Major Replication
HBASE-19543 Abstract a replication storage interface to extract the zk specific code Major proc-v2, Replication
HBASE-19525 RS side changes for moving peer modification from zk watcher to procedure Major proc-v2, Replication
HBASE-19580 Use slf4j instead of commons-logging in new, just-added Peer Procedure classes Major proc-v2, Replication
HBASE-19520 Add UTs for the new lock type PEER Major proc-v2
HBASE-19564 Procedure id is missing in the response of peer related operations Major proc-v2, Replication
HBASE-19536 Client side changes for moving peer modification from zk watcher to procedure Major Replication
HBASE-19524 Master side changes for moving peer modification from zk watcher to procedure Major proc-v2, Replication
HBASE-19216 Implement a general framework to execute remote procedure on RS Major proc-v2, Replication

OTHER:

JIRA Summary Priority Component
HBASE-19997 [rolling upgrade] 1.x => 2.x Blocker .
HBASE-20360 Further optimization for serial replication Major Replication
HBASE-20862 Address 2.1.0 Compatibility Report Issues Blocker compatibility
HBASE-20665 "Already cached block XXX" message should be DEBUG Minor BlockCache
HBASE-20677 Backport test of HBASE-20566 'Creating a system table after enabling rsgroup feature puts region into RIT' to branch-2 Major .
HBASE-19475 Extend backporting strategy in documentation Trivial documentation
HBASE-20595 Remove the concept of 'special tables' from rsgroups Major Region Assignment, rsgroup
HBASE-20415 branches-2 don't need maven-scala-plugin Major build
HBASE-20112 Include test results from nightly hadoop3 tests in jenkins test results Critical test
HBASE-17918 document serial replication Critical documentation, Replication
HBASE-19737 Manage a HBASE-19397-branch-2 branch and merge it to branch-2 Major proc-v2, Replication