diff --git a/docs/content/Indexing-Service.md b/docs/content/Indexing-Service.md index 4829dcdd28c..aed200a7c8f 100644 --- a/docs/content/Indexing-Service.md +++ b/docs/content/Indexing-Service.md @@ -3,12 +3,35 @@ layout: doc_page --- The indexing service is a highly-available, distributed service that runs indexing related tasks. Indexing service [tasks](Tasks.html) create (and sometimes destroy) Druid [segments](Segments.html). The indexing service has a master/slave like architecture. -The indexing service is composed of three main components: a peon component that can run a single task, a middle manager component that manages peons, and an overlord component that manages task distribution to middle managers. -Overlords and middle managers may run on the same node or across multiple nodes while middle managers and peons always run on the same node. +The indexing service is composed of three main components: a peon component that can run a single task, a [Middle Manager](Middlemanager.html) component that manages peons, and an overlord component that manages task distribution to middle managers. +Overlords and middle managers may run on the same node or across multiple nodes while middle managers and [Peons](Peons.html) always run on the same node. -Quick Start ----------------------------------------- -Run: +Indexing Service Overview +------------------------- + +![Indexing Service](../img/indexing_service.png "Indexing Service") + + + + +Overlord Node +----------------- + +The overlord node is responsible for accepting tasks, coordinating task distribution, creating locks around tasks, and returning statuses to callers. Overlord can be configured to run in one of two modes - local or remote (local being default). +In local mode overlord is also responsible for creating peons for executing tasks. When running the overlord in local mode, all middle manager and peon configurations must be provided as well. +Local mode is typically used for simple workflows. In remote mode, the overlord and middle manager are run in separate processes and you can run each on a different server. +This mode is recommended if you intend to use the indexing service as the single endpoint for all Druid indexing. + + +Run Overlord +---- ``` io.druid.cli.Main server overlord @@ -18,7 +41,7 @@ With the following JVM configuration: ``` -server --Xmx2g +-Xmx256m -Duser.timezone=UTC -Dfile.encoding=UTF-8 @@ -40,32 +63,16 @@ With the following JVM configuration: You can now submit simple indexing tasks to the indexing service. - - -Indexing Service Overview -------------------------- - -![Indexing Service](../img/indexing_service.png "Indexing Service") - -Overlord Node -------------- - -The overlord node is responsible for accepting tasks, coordinating task distribution, creating locks around tasks, and returning statuses to callers. - -#### Usage +#### Submitting Tasks and Querying Task Status Tasks are submitted to the overlord node in the form of JSON objects. Tasks can be submitted via POST requests to: ``` http://:/druid/indexer/v1/task ``` +this will return you the taskId of the submitted task. Tasks can cancelled via POST requests to: @@ -87,12 +94,12 @@ Task segments can be retrieved via GET requests to: http://:/druid/indexer/v1/task/{taskId}/segments ``` -#### Console +#### Overlord Console The overlord console can be used to view pending tasks, running tasks, available workers, and recent worker creation and termination. The console can be accessed at: ``` -http://:8080/console.html +http://:/console.html ``` #### Autoscaling @@ -180,83 +187,17 @@ Issuing a GET request at the same URL will return the current worker setup spec |`nodeData`|A JSON object that contains metadata about new nodes to create.|none| |`userData`|A JSON object that contains metadata about how the node should register itself on startup. This data is sent with node creation requests.|none| -#### Running -``` -io.druid.cli.Main server overlord -``` -Note: When running the overlord in local mode, all middle manager and peon configurations must be provided as well. +Middle Managers +----- -MiddleManager Node ------------------- - -The middle manager node is a worker node that executes submitted tasks. Middle Managers forward tasks to peons that run in separate JVMs. Each peon is capable of running only one task at a time, however, a middle manager may have multiple peons. - -#### JVM Configuration - -Middle managers pass their configurations down to their child peons. The middle manager module requires the following configs: - -|Property|Description|Default| -|--------|-----------|-------| -|`druid.worker.ip`|The IP of the worker.|localhost| -|`druid.worker.version`|Version identifier for the middle manager.|0| -|`druid.worker.capacity`|Maximum number of tasks the middle manager can accept.|Number of available processors - 1| -|`druid.indexer.runner.compressZnodes`|Indicates whether or not the middle managers should compress Znodes.|false| -|`druid.indexer.runner.maxZnodeBytes`|The maximum size Znode in bytes that can be created in Zookeeper.|524288| -|`druid.indexer.runner.taskDir`|Temporary intermediate directory used during task execution.|/tmp/persistent| -|`druid.indexer.runner.javaCommand`|Command required to execute java.|java| -|`druid.indexer.runner.javaOpts`|-X Java options to run the peon in its own JVM.|""| -|`druid.indexer.runner.classpath`|Java classpath for the peon.|System.getProperty("java.class.path")| -|`druid.indexer.runner.startPort`|The port that peons begin running on.|8080| -|`druid.indexer.runner.allowedPrefixes`|Whitelist of prefixes for configs that can be passed down to child peons.|"com.metamx", "druid", "io.druid", "user.timezone","file.encoding"| - -#### Running - -``` -io.druid.cli.Main server middleManager -``` +See [Middle Manager](Middlemanager.html). Peons ----- -Peons run a single task in a single JVM. Peons are a part of middle managers and should rarely (if ever) be run on their own. -#### JVM Configuration -Although peons inherit the configurations of their parent middle managers, explicit child peon configs can be set by prefixing them with: - -``` -druid.indexer.fork.property -``` - -Additional peon configs include: - -|Property|Description|Default| -|--------|-----------|-------| -|`druid.peon.mode`|Choices are "local" and "remote". Setting this to local means you intend to run the peon as a standalone node (Not recommended).|remote| -|`druid.indexer.task.baseDir`|Base temporary working directory.|/tmp| -|`druid.indexer.task.baseTaskDir`|Base temporary working directory for tasks.|/tmp/persistent/tasks| -|`druid.indexer.task.hadoopWorkingPath`|Temporary working directory for Hadoop tasks.|/tmp/druid-indexing| -|`druid.indexer.task.defaultRowFlushBoundary`|Highest row count before persisting to disk. Used for indexing generating tasks.|50000| -|`druid.indexer.task.chathandler.type`|Choices are "noop" and "announce". Certain tasks will use service discovery to announce an HTTP endpoint that events can be posted to.|noop| - -If the peon is running in remote mode, there must be an overlord up and running. Running peons in remote mode require the following configurations: - -|Property|Description|Default| -|--------|-----------|-------| -|`druid.peon.taskActionClient.retry.minWait`|The minimum retry time to communicate with overlord.|PT1M| -|`druid.peon.taskActionClient.retry.maxWait`|The maximum retry time to communicate with overlord.|PT10M| -|`druid.peon.taskActionClient.retry.maxRetryCount`|The maximum number of retries to communicate with overlord.|10| - -#### Running - -The peon should very rarely ever be run independent of the middle manager. - -``` -io.druid.cli.Main internal peon -``` - -The task file contains the task JSON object. -The status file indicates where the task status will be output. +See [Peon](Peons.html). Tasks ----- diff --git a/docs/content/Middlemanager.md b/docs/content/Middlemanager.md new file mode 100644 index 00000000000..c143bb91987 --- /dev/null +++ b/docs/content/Middlemanager.md @@ -0,0 +1,57 @@ +--- +layout: doc_page +--- + +Middle Manager Node +------------------ + +The middle manager node is a worker node that executes submitted tasks. Middle Managers forward tasks to peons that run in separate JVMs. +The reason we have separate JVMs for tasks is for log isolation. Each [Peon](Peons.html) is capable of running only one task at a time, however, a middle manager may have multiple peons. + +Quick Start +------------------ + +#### Running + +``` +io.druid.cli.Main server middleManager +``` + +With the following JVM configuration: + +``` +-Duser.timezone=UTC +-Dfile.encoding=UTF-8 + +-Ddruid.host=localhost +-Ddruid.port=8091 +-Ddruid.service=middleManager + +-Ddruid.zk.service.host=localhost + +-Ddruid.db.connector.connectURI=jdbc:mysql://localhost:3306/druid +-Ddruid.db.connector.user=druid +-Ddruid.db.connector.password=diurd +-Ddruid.selectors.indexing.serviceName=overlord +-Ddruid.indexer.runner.startPort=8092 +-Ddruid.indexer.fork.property.druid.computation.buffer.size=268435456 +``` + +#### JVM Configuration + +Middle managers pass their configurations down to their child peons. The middle manager module requires the following configs: + +|Property|Description|Default| +|--------|-----------|-------| +|`druid.worker.ip`|The IP of the worker.|localhost| +|`druid.worker.version`|Version identifier for the middle manager.|0| +|`druid.worker.capacity`|Maximum number of tasks the middle manager can accept.|Number of available processors - 1| +|`druid.indexer.runner.compressZnodes`|Indicates whether or not the middle managers should compress Znodes.|false| +|`druid.indexer.runner.maxZnodeBytes`|The maximum size Znode in bytes that can be created in Zookeeper.|524288| +|`druid.indexer.runner.taskDir`|Temporary intermediate directory used during task execution.|/tmp/persistent| +|`druid.indexer.runner.javaCommand`|Command required to execute java.|java| +|`druid.indexer.runner.javaOpts`|-X Java options to run the peon in its own JVM.|""| +|`druid.indexer.runner.classpath`|Java classpath for the peon.|System.getProperty("java.class.path")| +|`druid.indexer.runner.startPort`|The port that peons begin running on.|8080| +|`druid.indexer.runner.allowedPrefixes`|Whitelist of prefixes for configs that can be passed down to child peons.|"com.metamx", "druid", "io.druid", "user.timezone","file.encoding"| + diff --git a/docs/content/Peons.md b/docs/content/Peons.md new file mode 100644 index 00000000000..b5d16e1df2a --- /dev/null +++ b/docs/content/Peons.md @@ -0,0 +1,44 @@ +--- +layout: doc_page +--- + +Peons +----- +Peons run a single task in a single JVM. MiddleManager is responsible for creating Peons for running tasks. +Peons should rarely (if ever for testing purposes) be run on their own. + +#### JVM Configuration +Although peons inherit the configurations of their parent middle managers, explicit child peon configs in middlemanager can be set by prefixing them with: + +``` +druid.indexer.fork.property +``` +Additional peon configs include: + +|Property|Description|Default| +|--------|-----------|-------| +|`druid.peon.mode`|Choices are "local" and "remote". Setting this to local means you intend to run the peon as a standalone node (Not recommended).|remote| +|`druid.indexer.task.baseDir`|Base temporary working directory.|/tmp| +|`druid.indexer.task.baseTaskDir`|Base temporary working directory for tasks.|/tmp/persistent/tasks| +|`druid.indexer.task.hadoopWorkingPath`|Temporary working directory for Hadoop tasks.|/tmp/druid-indexing| +|`druid.indexer.task.defaultRowFlushBoundary`|Highest row count before persisting to disk. Used for indexing generating tasks.|50000| +|`druid.indexer.task.chathandler.type`|Choices are "noop" and "announce". Certain tasks will use service discovery to announce an HTTP endpoint that events can be posted to.|noop| + +If the peon is running in remote mode, there must be an overlord up and running. Running peons in remote mode require the following configurations: + +|Property|Description|Default| +|--------|-----------|-------| +|`druid.peon.taskActionClient.retry.minWait`|The minimum retry time to communicate with overlord.|PT1M| +|`druid.peon.taskActionClient.retry.maxWait`|The maximum retry time to communicate with overlord.|PT10M| +|`druid.peon.taskActionClient.retry.maxRetryCount`|The maximum number of retries to communicate with overlord.|10| + +#### Running + +The peon should very rarely ever be run independent of the middle manager. + +``` +io.druid.cli.Main internal peon +``` + +The task file contains the task JSON object. +The status file indicates where the task status will be output. \ No newline at end of file diff --git a/docs/content/toc.textile b/docs/content/toc.textile index 9f212df47e6..84fde036997 100644 --- a/docs/content/toc.textile +++ b/docs/content/toc.textile @@ -59,6 +59,8 @@ h2. Architecture *** "Firehose":./Firehose.html *** "Plumber":./Plumber.html ** "Indexing Service":./Indexing-Service.html +*** "Middle Manager":./Middlemanager.html +*** "Peon":./Peons.html * External Dependencies ** "Deep Storage":./Deep-Storage.html ** "MySQL":./MySQL.html diff --git a/indexing-service/src/main/java/io/druid/indexing/overlord/RemoteTaskRunner.java b/indexing-service/src/main/java/io/druid/indexing/overlord/RemoteTaskRunner.java index 0d9f6e6d60d..be87cef7255 100644 --- a/indexing-service/src/main/java/io/druid/indexing/overlord/RemoteTaskRunner.java +++ b/indexing-service/src/main/java/io/druid/indexing/overlord/RemoteTaskRunner.java @@ -498,7 +498,7 @@ public class RemoteTaskRunner implements TaskRunner, TaskLogStreamer } } catch (Exception e) { - log.makeAlert("Exception while trying to run task") + log.makeAlert(e, "Exception while trying to run task") .addData("taskId", taskRunnerWorkItem.getTask().getId()) .emit(); } @@ -748,8 +748,8 @@ public class RemoteTaskRunner implements TaskRunner, TaskLogStreamer } ); sortedWorkers.addAll(zkWorkers.values()); - final String configMinWorkerVer = workerSetupData.get().getMinVersion(); - final String minWorkerVer = configMinWorkerVer == null ? config.getMinWorkerVersion() : configMinWorkerVer; + final String workerSetupDataMinVer = workerSetupData.get() == null ? null :workerSetupData.get().getMinVersion(); + final String minWorkerVer = workerSetupDataMinVer == null ? config.getMinWorkerVersion() : workerSetupDataMinVer; for (ZkWorker zkWorker : sortedWorkers) { if (zkWorker.canRunTask(task) && zkWorker.isValidVersion(minWorkerVer)) { diff --git a/indexing-service/src/main/java/io/druid/indexing/overlord/TaskQueue.java b/indexing-service/src/main/java/io/druid/indexing/overlord/TaskQueue.java index e033ff60ce9..580dfd02a0d 100644 --- a/indexing-service/src/main/java/io/druid/indexing/overlord/TaskQueue.java +++ b/indexing-service/src/main/java/io/druid/indexing/overlord/TaskQueue.java @@ -373,11 +373,11 @@ public class TaskQueue if(didPersistStatus) { log.info("Task done: %s", task); taskLockbox.unlock(task); - workMayBeAvailable.signalAll(); } else { log.warn("Status could not be persisted! Reinserting task: %s", task.getId()); queue.add(task); } + workMayBeAvailable.signalAll(); } } finally { diff --git a/indexing-service/src/main/java/io/druid/indexing/overlord/config/RemoteTaskRunnerConfig.java b/indexing-service/src/main/java/io/druid/indexing/overlord/config/RemoteTaskRunnerConfig.java index dddb1671c51..fb149391a06 100644 --- a/indexing-service/src/main/java/io/druid/indexing/overlord/config/RemoteTaskRunnerConfig.java +++ b/indexing-service/src/main/java/io/druid/indexing/overlord/config/RemoteTaskRunnerConfig.java @@ -37,7 +37,7 @@ public class RemoteTaskRunnerConfig private boolean compressZnodes = false; @JsonProperty - private String minWorkerVersion = null; + private String minWorkerVersion = "0"; @JsonProperty @Min(10 * 1024)