89cb0c5370
* HBASE-25055 Add ReplicationSource for meta WALs; add enable/disable when hbase:meta assigned to RS Fill in gap left by HBASE-11183 'Timeline Consistent region replicas - Phase 2 design'. HBASE-11183 left off implementing 'async WAL Replication' on the hbase:meta Table; hbase:meta Table could only do Phase 1 Region Replicas reading the primary Regions' hfiles. Here we add 'async WAL Replication' to hbase:meta so Replicas can be more current with the primary's changes. Adds a 'special' ReplicationSource that reads hbase:meta WAL files and replicates all edits to the configured in-cluster endpoint (Defaults to the RegionReadReplicaEndpoint.class -- set hbase.region.replica.catalog.replication to target a different endpoint implementation). Set hbase.region.replica.replication.catalog.enabled to enable async WAL Replication for hbase:meta region replicas. Its off by default. The CatalogReplicationSource for async WAL Replication of hbase:meta does NOT need to keep up WAL offset or a queue of WALs-to-replicate in the replication queue store as is done in other ReplicationSource implementations; the CatalogReplicationSource is for Region Replicas only. General Replication does not replicate hbase:meta. hbase:meta Region Replicas reset on crash of the primary replica so there is no need to 'recover' replication that was running on the crashed server. Because it so different in operation, the CatalogReplicationSource is bolted on to the side of the ReplicationSourceManager. It is lazily instantiated to match the lazy instantiation of the hbase:meta WALProvider, created and started on the open of the first Region of an hbase:meta table. Thereafter it stays up till the process dies, even if all hbase:meta Regions have moved off the server, in case a hbase:meta Region is moved back (Doing this latter simplifies the implementation) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Read configuration to see if we need to wait on setting a Region read-enabled (if so, replicas will only flip to enable read after confirming a flush of the primary so they for sure are a replica of a known point) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/AssignRegionHandler.java If configured, on open of hbase:meta, ask the ReplicationSourceManager to add a ReplicationSource (if it hasn't already). hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/RegionReplicaFlushHandler.java Edit log message. hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/handler/UnassignRegionHandler.java If configured, on close of hbase:meta, update ReplicationSourceManager that a source Region has closed. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceFactory.java javadoc and make constructor private. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceInterface.java Add logPositionAndCleanOldLogs w/ default of the old behavior so CatalogReplicationSource can bypass updating store with WAL position, etc. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceManager.java Add creation and start of an CatalogReplicationSource. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSourceShipper.java Go via ReplicationSource when calling logPostionAndCleanOldLogs so new RS can intercept. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/WALFileLengthProvider.java Javadoc. hbase-server/src/main/java/org/apache/hadoop/hbase/util/ServerRegionReplicaUtil.java Add utility for reading configurations for hbase:meta region replicas. hbase-server/src/main/java/org/apache/hadoop/hbase/wal/WALFactory.java Javadoc. hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestRegionReplicaReplicationEndpoint.java Use define. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSource.java Specical version of ReplicationSource for Region Replicas on hbase:meta. hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/CatalogReplicationSourcePeer.java Needs a special peer too (peers are baked into replication though we don't use 'peers' here) hbase-server/src/test/java/org/apache/hadoop/hbase/replication/regionserver/TestMetaRegionReplicaReplicationEndpoint.java hbase-server/src/test/java/org/apache/hadoop/hbase/wal/TestWALProvider.java Tests. Signed-off-by: Duo Zhang <zhangduo@apache.org> Signed-off-by: Huaxiang Sun <huaxiangsun@apache.com>
Apache HBase [1] is an open-source, distributed, versioned, column-oriented store modeled after Google' Bigtable: A Distributed Storage System for Structured Data by Chang et al.[2] Just as Bigtable leverages the distributed data storage provided by the Google File System, HBase provides Bigtable-like capabilities on top of Apache Hadoop [3]. To get started using HBase, the full documentation for this release can be found under the doc/ directory that accompanies this README. Using a browser, open the docs/index.html to view the project home page (or browse to [1]). The hbase 'book' at http://hbase.apache.org/book.html has a 'quick start' section and is where you should being your exploration of the hbase project. The latest HBase can be downloaded from an Apache Mirror [4]. The source code can be found at [5] The HBase issue tracker is at [6] Apache HBase is made available under the Apache License, version 2.0 [7] The HBase mailing lists and archives are listed here [8]. The HBase distribution includes cryptographic software. See the export control notice here [9]. 1. http://hbase.apache.org 2. http://research.google.com/archive/bigtable.html 3. http://hadoop.apache.org 4. http://www.apache.org/dyn/closer.lua/hbase/ 5. https://hbase.apache.org/source-repository.html 6. https://hbase.apache.org/issue-tracking.html 7. http://hbase.apache.org/license.html 8. http://hbase.apache.org/mail-lists.html 9. https://hbase.apache.org/export_control.html