diff --git a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md index 16f14c13ad2..35cbc580aa1 100644 --- a/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md +++ b/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/TimelineServiceV2.md @@ -73,8 +73,6 @@ The following diagram illustrates the design at a high level. ### Current Status and Future Plans -YARN Timeline Service v.2 is currently in alpha ("alpha 2"). It is a work in progress, and -many things can and will change rapidly. A complete end-to-end flow of writes and reads is functional, with Apache HBase as the backend. You should be able to start generating data. When enabled, all YARN-generic events are @@ -82,32 +80,31 @@ published as well as YARN system metrics such as CPU and memory. Furthermore, so including Distributed Shell and MapReduce can write per-framework data to YARN Timeline Service v.2. -The basic mode of accessing data is via REST. Currently there is no support for command line -access. The REST API comes with a good number of useful and flexible query patterns (see below for -more information). +The basic mode of accessing data is via REST. The REST API comes with a good number of useful and flexible query patterns (see below for +more information). YARN Client has been integrated with ATSv2. This enables fetching application/attempt/container +report from TimelineReader if details are not present in ResouceManager. The collectors (writers) are currently embedded in the node managers as auxiliary services. The resource manager also has its dedicated in-process collector. The reader is currently a single instance. Currently, it is not possible to write to Timeline Service outside the context of a YARN application (i.e. no off-cluster client). -Starting from alpha2, Timeline Service v.2 supports simple authorization in terms of a -configurable whitelist of users and groups who can read timeline data. Cluster admins are +Kerberos Authentication is supported end to end. All communication to HBase can be kerberized. Refer [Security Configuration](#Security_Configuration) for configs. +Support for simple authorization has been added in terms of a configurable whitelist of users and groups who can read timeline data. Cluster admins are allowed by default to read timeline data. + When YARN Timeline Service v.2 is disabled, one can expect no functional or performance impact on any other existing functionality. -The work to make it truly production-ready continues. Some key items include -* More robust storage fault tolerance +Road map includes + +* More robust storage fault tolerance. * Support for off-cluster clients -* Better support for long-running apps -* Support for ACLs +* Support for entity ACLs * Offline (time-based periodic) aggregation for flows, users, and queues for reporting and analysis -* Timeline collectors as separate instances from node managers -* Clustering of the readers * Migration and compatibility with v.1 @@ -143,7 +140,7 @@ New configuration parameters that are introduced with v.2 are marked bold. | **`yarn.timeline-service.hbase.coprocessor.app-final-value-retention-milliseconds`** | The setting that controls how long the final value of a metric of a completed app is retained before merging into the flow sum. Defaults to `259200000` (3 days). This should be set in the HBase cluster. | | **`yarn.rm.system-metrics-publisher.emit-container-events`** | The setting that controls whether yarn container metrics is published to the timeline server or not by RM. This configuration setting is for ATS V2. Defaults to `false`. | | **`yarn.nodemanager.emit-container-events`** | The setting that controls whether yarn container metrics is published to the timeline server or not by NM. This configuration setting is for ATS V2. Defaults to `true`. | -#### Security Configuration +#### Security Configuration Security can be enabled by setting `yarn.timeline-service.http-authentication.type` @@ -163,6 +160,7 @@ to `kerberos`, after which the following configuration options are available: | `yarn.timeline-service.delegation.token.max-lifetime` | Defaults to `604800000` (7 days). | | `yarn.timeline-service.read.authentication.enabled` | Enables or disables authorization checks for reading timeline service v2 data. Default is `false` which is disabled. | | `yarn.timeline-service.read.allowed.users` | Comma separated list of user, followed by space, then comma separated list of groups. It will allow this list of users and groups to read the data and reject everyone else. Default value is set to none. If authorization is enabled, then this configuration is mandatory. | +| `yarn.webapp.filter-entity-list-by-user` | Default is false. If set true and yarn.timeline-service.read.authentication.enabled is disabled, then listing of entities restricted to remote user entities only. It is YARN common configuration for listing APIs. Using this configuration TimelineReader authorize caller UGI with entity owner. If does not match, those entities will be removed from response.| #### Enabling CORS support To enable cross-origin support (CORS) for the Timeline Service v.2, please set the following configuration parameters: @@ -408,13 +406,13 @@ To write MapReduce framework data to Timeline Service v.2, enable the following ``` -### Upgrade from alpha1 to alpha2 +### Upgrade from alpha1 to GA If you are currently running Timeline Service v2 alpha1 version, we recommend the following: - Clear existing data in tables (truncate tables) since the row key for AppToFlow has changed. -- The coprocessor is now a dynamically loaded table level coprocessor in alpha2. We -recommend dropping the table, replacing the coprocessor jar on hdfs with the alpha2 one, +- The coprocessor is now a dynamically loaded table level coprocessor in GA. We +recommend dropping the table, replacing the coprocessor jar on hdfs with the GA one, restarting the Region servers and recreating the `flowrun` table. ### Publishing application specific data