From 525cbf5aab446f7e5ef2e640c64d19bddb54d85a Mon Sep 17 00:00:00 2001 From: Igal Levy Date: Thu, 27 Mar 2014 14:58:40 -0700 Subject: [PATCH] completed config info with rjection policy, segmentGranularity; some minor edits --- docs/content/Plumber.md | 16 +++++++++++----- 1 file changed, 11 insertions(+), 5 deletions(-) diff --git a/docs/content/Plumber.md b/docs/content/Plumber.md index 6a1a44b5045..774fbdb9a43 100644 --- a/docs/content/Plumber.md +++ b/docs/content/Plumber.md @@ -3,17 +3,23 @@ layout: doc_page --- # Druid Plumbers -The Plumber handles generated segments both while they are being generated and when they are "done". This is also technically a pluggable interface and there are multiple implementations. However, plumbers handle numerous complex details, and therefore an advanced understanding of Druid is recommended before implementing your own. +The plumber handles generated segments both while they are being generated and when they are "done". This is also technically a pluggable interface and there are multiple implementations. However, plumbers handle numerous complex details, and therefore an advanced understanding of Druid is recommended before implementing your own. |Field|Type|Description|Required| |-----|----|-----------|--------| -|type|String|Specifies the type of plumber. Each value will have its own configuration schema, plumbers packaged with Druid are described below.|yes| +|type|String|Specifies the type of plumber. Each value will have its own configuration schema. Plumbers packaged with Druid are described below.|yes| The following can be configured on the plumber: -* `windowPeriod` is the amount of lag time to allow events. This is configured with a 10 minute window, meaning that any event more than 10 minutes ago will be thrown away and not included in the segment generated by the realtime server. -* `basePersistDirectory` is the directory to put things that need persistence. The plumber is responsible for the actual intermediate persists and this tells it where to store those persists. -* `maxPendingPersists` is how many persists a plumber can do concurrently without starting to block. +* `windowPeriod` is the amount of lag time to allow events. This is configured with a 10 minute window, meaning that any event more than 10 minutes ago will be thrown away and not included in the segment generated by the realtime server. +* `basePersistDirectory` is the directory to put things that need persistence. The plumber is responsible for the actual intermediate persists and this tells it where to store those persists. +* `maxPendingPersists` is how many persists a plumber can do concurrently without starting to block. +* `segmentGranularity` specifies the granularity of the segment, or the amount of time a segment will represent. +* `rejectionPolicy` controls how data sets the data acceptance policy for creating and handing off segments. The following policies are available: + * `serverTime` – The default policy, it is optimal for current data that is generated and ingested in real time. Uses `windowPeriod` to accept only those events that are inside the window looking forward and back. + * `none` – Never hands off data unless shutdown() is called on the configured firehose. + * `test` – Useful for testing that handoff is working, *not useful in terms of data integrity*. It uses the sum of `segmentGranularity` plus `windowPeriod` as a window. + Available Plumbers