Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Fixed: 

- Bad link for granularitySpec changed to real definition.
- All heads set to logical levels (consistent with all others).
- Fixed bad link in definition for spatialDimensions.
This commit is contained in:
Igal Levy 2014-03-18 10:28:33 -07:00
parent 71818288e2
commit 04040ef75f
1 changed files with 13 additions and 13 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

26
docs/content/Tasks.md
View File

@ -9,7 +9,7 @@ There are several different types of tasks.
Segment Creation Tasks Segment Creation Tasks
---------------------- ----------------------
#### Index Task ### Index Task
The Index Task is a simpler variation of the Index Hadoop task that is designed to be used for smaller data sets. The task executes within the indexing service and does not require an external Hadoop setup to use. The grammar of the index task is as follows: The Index Task is a simpler variation of the Index Hadoop task that is designed to be used for smaller data sets. The task executes within the indexing service and does not require an external Hadoop setup to use. The grammar of the index task is as follows:
@ -51,15 +51,15 @@ The Index Task is a simpler variation of the Index Hadoop task that is designed
|--------|-----------|---------| |--------|-----------|---------|
|type|The task type, this should always be "index".|yes| |type|The task type, this should always be "index".|yes|
|id|The task ID.|no| |id|The task ID.|no|
|granularitySpec|See [granularitySpec](Tasks.html)|yes| |granularitySpec|Specifies the segment chunks that the task will process. `type` is always "uniform"; `gran` sets the granularity of the chunks ("DAY" means all segments containing timestamps in the same day, while `intervals` sets the interval that the chunks will cover.|yes|
|spatialDimensions|Dimensions to build spatial indexes over. See [Spatial-Indexing](Spatial-Indexing.html)|no| |spatialDimensions|Dimensions to build spatial indexes over. See [Geographic Queries](GeographicQueries.html).|no|
|aggregators|The metrics to aggregate in the data set. For more info, see [Aggregations](Aggregations.html)|yes| |aggregators|The metrics to aggregate in the data set. For more info, see [Aggregations](Aggregations.html)|yes|
|indexGranularity|The rollup granularity for timestamps.|no| |indexGranularity|The rollup granularity for timestamps.|no|
|targetPartitionSize|Used in sharding. Determines how many rows are in each segment.|no| |targetPartitionSize|Used in sharding. Determines how many rows are in each segment.|no|
|firehose|The input source of data. For more info, see [Firehose](Firehose.html)|yes| |firehose|The input source of data. For more info, see [Firehose](Firehose.html)|yes|
|rowFlushBoundary|Used in determining when intermediate persist should occur to disk.|no| |rowFlushBoundary|Used in determining when intermediate persist should occur to disk.|no|
#### Index Hadoop Task ### Index Hadoop Task
The Hadoop Index Task is used to index larger data sets that require the parallelization and processing power of a Hadoop cluster. The Hadoop Index Task is used to index larger data sets that require the parallelization and processing power of a Hadoop cluster.
@ -79,11 +79,11 @@ The Hadoop Index Task is used to index larger data sets that require the paralle
The Hadoop Index Config submitted as part of an Hadoop Index Task is identical to the Hadoop Index Config used by the `HadoopBatchIndexer` except that three fields must be omitted: `segmentOutputPath`, `workingPath`, `updaterJobSpec`. The Indexing Service takes care of setting these fields internally. The Hadoop Index Config submitted as part of an Hadoop Index Task is identical to the Hadoop Index Config used by the `HadoopBatchIndexer` except that three fields must be omitted: `segmentOutputPath`, `workingPath`, `updaterJobSpec`. The Indexing Service takes care of setting these fields internally.
##### Using your own Hadoop distribution #### Using your own Hadoop distribution
Druid is compiled against Apache hadoop-core 1.0.3. However, if you happen to use a different flavor of hadoop that is API compatible with hadoop-core 1.0.3, you should only have to change the hadoopCoordinates property to point to the maven artifact used by your distribution. Druid is compiled against Apache hadoop-core 1.0.3. However, if you happen to use a different flavor of hadoop that is API compatible with hadoop-core 1.0.3, you should only have to change the hadoopCoordinates property to point to the maven artifact used by your distribution.
##### Resolving dependency conflicts running HadoopIndexTask #### Resolving dependency conflicts running HadoopIndexTask
Currently, the HadoopIndexTask creates a single classpath to run the HadoopDruidIndexerJob, which can lead to version conflicts between various dependencies of Druid, extension modules, and Hadoop's own dependencies. Currently, the HadoopIndexTask creates a single classpath to run the HadoopDruidIndexerJob, which can lead to version conflicts between various dependencies of Druid, extension modules, and Hadoop's own dependencies.
@ -91,7 +91,7 @@ The Hadoop index task will put Druid's dependencies first on the classpath, foll
If you are having trouble with any extensions in HadoopIndexTask, it may be the case that Druid, or one of its dependencies, depends on a different version of a library than what you are using as part of your extensions, but Druid's version overrides the one in your extension. In that case you probably want to build your own Druid version and override the offending library by adding an explicit dependency to the pom.xml of each druid sub-module that depends on it. If you are having trouble with any extensions in HadoopIndexTask, it may be the case that Druid, or one of its dependencies, depends on a different version of a library than what you are using as part of your extensions, but Druid's version overrides the one in your extension. In that case you probably want to build your own Druid version and override the offending library by adding an explicit dependency to the pom.xml of each druid sub-module that depends on it.
#### Realtime Index Task ### Realtime Index Task
The indexing service can also run real-time tasks. These tasks effectively transform a middle manager into a real-time node. We introduced real-time tasks as a way to programmatically add new real-time data sources without needing to manually add nodes. The grammar for the real-time task is as follows: The indexing service can also run real-time tasks. These tasks effectively transform a middle manager into a real-time node. We introduced real-time tasks as a way to programmatically add new real-time data sources without needing to manually add nodes. The grammar for the real-time task is as follows:
@ -172,7 +172,7 @@ For schema, fireDepartmentConfig, windowPeriod, segmentGranularity, and rejectio
Segment Merging Tasks Segment Merging Tasks
--------------------- ---------------------
#### Append Task ### Append Task
Append tasks append a list of segments together into a single segment (one after the other). The grammar is: Append tasks append a list of segments together into a single segment (one after the other). The grammar is:
@ -184,7 +184,7 @@ Append tasks append a list of segments together into a single segment (one after
} }
``` ```
#### Merge Task ### Merge Task
Merge tasks merge a list of segments together. Any common timestamps are merged. The grammar is: Merge tasks merge a list of segments together. Any common timestamps are merged. The grammar is:
@ -199,7 +199,7 @@ Merge tasks merge a list of segments together. Any common timestamps are merged.
Segment Destroying Tasks Segment Destroying Tasks
------------------------ ------------------------
#### Delete Task ### Delete Task
Delete tasks create empty segments with no data. The grammar is: Delete tasks create empty segments with no data. The grammar is:
@ -211,7 +211,7 @@ Delete tasks create empty segments with no data. The grammar is:
} }
``` ```
#### Kill Task ### Kill Task
Kill tasks delete all information about a segment and removes it from deep storage. Killable segments must be disabled (used==0) in the Druid segment table. The available grammar is: Kill tasks delete all information about a segment and removes it from deep storage. Killable segments must be disabled (used==0) in the Druid segment table. The available grammar is:
@ -226,7 +226,7 @@ Kill tasks delete all information about a segment and removes it from deep stora
Misc. Tasks Misc. Tasks
----------- -----------
#### Version Converter Task ### Version Converter Task
These tasks convert segments from an existing older index version to the latest index version. The available grammar is: These tasks convert segments from an existing older index version to the latest index version. The available grammar is:
@ -240,7 +240,7 @@ These tasks convert segments from an existing older index version to the latest
} }
``` ```
#### Noop Task ### Noop Task
These tasks start, sleep for a time and are used only for testing. The available grammar is: These tasks start, sleep for a time and are used only for testing. The available grammar is: