Overview ======== hbase-metrics and hbase-metrics-api are two modules that define and implement the "new" metric system used internally within HBase. These two modules (and some other code in hbase-hadoop2-compat) module are referred as "HBase metrics framework". HBase-metrics-api Module ======================== HBase Metrics API (hbase-metrics-api) contains the interface that HBase exposes internally and to third party code (including coprocessors). It is a thin abstraction over the actual implementation for backwards compatibility guarantees. The source / binary and other compatibility guarantees are for "LimitedPrivate API" (see [1] for an explanation). The metrics API in this hbase-metrics-api module is inspired by the Dropwizard metrics 3.1 API (See [2]). It is a subset of the API only containing metrics collection. However, the implementation is HBase-specific and provided in hbase-metrics module. All of the classes in this module is HBase-internal. See the latest documentation of Dropwizard metrics for examples of defining / using metrics. HBase-metrics Module ==================== hbase-metrics module contains implementation of the "HBase Metrics API", including MetricRegistry, Counter, Histogram, etc. These are highly concurrent implementations of the Metric interfaces. Metrics in HBase are grouped into different sets (like WAL, RPC, RegionServer, etc). Each group of metrics should be tracked via a MetricRegistry specific to that group. Metrics can be dynamically added or removed from the registry with a name. Each Registry is independent of the other registries and will have it's own JMX context and MetricRecord (when used with Metrics2). MetricRegistry's themselves are tracked via a global registry (of MetricRegistries) called MetricRegistries. MetricRegistries.global() can be used to obtain the global instance. MetricRegistry instances can also be dynamically registered and removed. However, unlike the MetricRegistry, MetricRegistries does reference counting of the MetricRegistry instances. Only Metrics in the MetricRegistry instances that are in the global MetricRegistry are exported to the metric sinks or JMX. Coprocessor Metrics =================== HBase allows custom coprocessors to track and export metrics using the new framework. Coprocessors and other third party code should only use the classes and interfaces from hbase-metrics-api module and only the classes that are marked with InterfaceAudience.LimitedPrivate annotation. There is no guarantee on the compatibility requirements for other classes. Coprocessors can obtain the MetricRegistry to register their custom metrics via corresponding CoprocessorEnvironment context. See ExampleRegionObserverWithMetrics and ExampleMasterObserverWithMetrics classes in hbase-examples module for usage. Developer Notes =============== Historically, HBase has been using Hadoop's Metrics2 framework [3] for collecting and reporting the metrics internally. However, due to the difficultly of dealing with the Metrics2 framework, HBase is moving away from Hadoop's metrics implementation to its custom implementation. The move will happen incrementally, and during the time, both Hadoop Metrics2-based metrics and hbase-metrics module based classes will be in the source code. All new implementations for metrics SHOULD use the new API and framework. Examples of the new framework can be found in MetricsCoprocessor and MetricsRegionServerSourceImpl classes. See HBASE-9774 [4] for more context. hbase-metrics module right now only deals with metrics tracking and collection. It does not do JMX reporting or reporting to console, ganglia, opentsdb, etc. We use Hadoop's Metrics2 for reporting metrics to different sinks or exporting via JMX. However, this is contained within the hbase-hadoop2-compat module completely, so that rest of the code does not know anything about the Metrics2 dependency. HBaseMetrics2HadoopMetricsAdapter is the adapter that can collect metrics in a MetricRegistry using the metric2 MetricsCollector / MetricRecordBuilder interfaces. GlobalMetricRegistriesSource is the global Metrics2 Source that collects all of the metrics in all of the metric registries in the MetricRegistries.global() instance. References 1. https://hbase.apache.org/book.html#hbase.versioning 2. http://metrics.dropwizard.io/ 3. https://hadoop.apache.org/docs/r2.7.2/api/org/apache/hadoop/metrics2/package-summary.html 4. https://issues.apache.org/jira/browse/HBASE-9774