From b1c4a5de0de92a055e61fb05e401013e06c824d1 Mon Sep 17 00:00:00 2001 From: Jihoon Son Date: Wed, 13 Feb 2019 11:20:52 -0800 Subject: [PATCH] Fix and improve doc for partitioning of local index (#7064) --- docs/content/ingestion/index.md | 4 ++-- docs/content/ingestion/native_tasks.md | 2 +- 2 files changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/content/ingestion/index.md b/docs/content/ingestion/index.md index db1edfa0172..e4b6b70dc02 100644 --- a/docs/content/ingestion/index.md +++ b/docs/content/ingestion/index.md @@ -178,7 +178,7 @@ the best one for your situation. |Method|How it works|Can append and overwrite?|Can handle late data?|Exactly-once ingestion?|Real-time queries?| |------|------------|-------------------------|---------------------|-----------------------|------------------| |[Native batch](native_tasks.html)|Druid loads data directly from S3, HTTP, NFS, or other networked storage.|Append or overwrite|Yes|Yes|No| -|[Hadoop](hadoop.html)|Druid launches Hadoop Map/Reduce jobs to load data files.|Append or overwrite|Yes|Yes|No| +|[Hadoop](hadoop.html)|Druid launches Hadoop Map/Reduce jobs to load data files.|Overwrite|Yes|Yes|No| |[Kafka indexing service](../development/extensions-core/kafka-ingestion.html)|Druid reads directly from Kafka.|Append only|Yes|Yes|Yes| |[Tranquility](stream-push.html)|You use Tranquility, a client side library, to push individual records into Druid.|Append only|No - late data is dropped|No - may drop or duplicate data|Yes| @@ -191,7 +191,7 @@ a _time chunk_, and each time chunk contains one or more [segments](../design/se particular time chunk may be partitioned further using options that vary based on the ingestion method you have chosen. * With [Hadoop](hadoop.html) you can do hash- or range-based partitioning on one or more columns. - * With [Native batch](native_tasks.html) you can partition on a hash of all dimension columns. This is useful when + * With [Native batch](native_tasks.html) you can partition on a hash of dimension columns. This is useful when rollup is enabled, since it maximizes your space savings. * With [Kafka indexing](../development/extensions-core/kafka-ingestion.html), partitioning is based on Kafka partitions, and is not configurable through Druid. You can configure it on the Kafka side by using the partitioning diff --git a/docs/content/ingestion/native_tasks.md b/docs/content/ingestion/native_tasks.md index e5b2e7d2871..5f7298363a9 100644 --- a/docs/content/ingestion/native_tasks.md +++ b/docs/content/ingestion/native_tasks.md @@ -500,7 +500,7 @@ The tuningConfig is optional and default parameters will be used if no tuningCon |indexSpec|defines segment storage format options to be used at indexing time, see [IndexSpec](#indexspec)|null|no| |maxPendingPersists|Maximum number of persists that can be pending but not started. If this limit would be exceeded by a new intermediate persist, ingestion will block until the currently-running persist finishes. Maximum heap memory usage for indexing scales with maxRowsInMemory * (2 + maxPendingPersists).|0 (meaning one persist can be running concurrently with ingestion, and none can be queued up)|no| |forceExtendableShardSpecs|Forces use of extendable shardSpecs. Experimental feature intended for use with the [Kafka indexing service extension](../development/extensions-core/kafka-ingestion.html).|false|no| -|forceGuaranteedRollup|Forces guaranteeing the [perfect rollup](../ingestion/index.html#roll-up-modes). The perfect rollup optimizes the total size of generated segments and querying time while indexing time will be increased. This flag cannot be used with either `appendToExisting` of IOConfig or `forceExtendableShardSpecs`. For more details, see the below __Segment pushing modes__ section.|false|no| +|forceGuaranteedRollup|Forces guaranteeing the [perfect rollup](../ingestion/index.html#roll-up-modes). The perfect rollup optimizes the total size of generated segments and querying time while indexing time will be increased. If this is set to true, the index task will read the entire input data twice: one for finding the optimal number of partitions per time chunk and one for generating segments. Note that the result segments would be hash-partitioned. You can set `forceExtendableShardSpecs` if you plan to append more data to the same time range in the future. This flag cannot be used with `appendToExisting` of IOConfig. For more details, see the below __Segment pushing modes__ section.|false|no| |reportParseExceptions|DEPRECATED. If true, exceptions encountered during parsing will be thrown and will halt ingestion; if false, unparseable rows and fields will be skipped. Setting `reportParseExceptions` to true will override existing configurations for `maxParseExceptions` and `maxSavedParseExceptions`, setting `maxParseExceptions` to 0 and limiting `maxSavedParseExceptions` to no more than 1.|false|no| |pushTimeout|Milliseconds to wait for pushing segments. It must be >= 0, where 0 means to wait forever.|0|no| |segmentWriteOutMediumFactory|Segment write-out medium to use when creating segments. See [SegmentWriteOutMediumFactory](#segmentWriteOutMediumFactory).|Not specified, the value from `druid.peon.defaultSegmentWriteOutMediumFactory.type` is used|no|