Apache/druid
1
0
Fork 0
mirror of https://github.com/apache/druid.git synced 2025-02-10 12:05:00 +00:00
Code Issues Packages Projects Releases Wiki Activity
435 Commits 116 Branches 507 Tags
Commit Graph

150 Commits

Author SHA1 Message Date
Gian Merlino
de4fc98128 TaskMasterLifecycle: Stop leaderLifecycle if it dies while starting 2013-02-14 13:36:43 -08:00
Gian Merlino
18daa19fc9 Merge branch 'master' into task-stuff 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskHolder.java
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/task/AbstractTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexGeneratorTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskContext.java
	merger/src/main/java/com/metamx/druid/merger/worker/TaskMonitor.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/TaskQueueTest.java
	server/src/main/java/com/metamx/druid/master/DruidMaster.java
 2013-02-14 13:10:29 -08:00
Gian Merlino
ff325b1f65 Merger: Changes to support rolling locks and a task-action API 
Generally, TaskStatus is no longer used as the sole way for tasks to
communicate status and metadata update requests back to the
coordinator. It is still used to communicate success or failure, but a
TaskActionClient is used for all other communication. In remote mode
this uses an http api.

TaskContext no longer exists, as its purpose is now served by
the TaskActionClient.

The TaskQueue and TaskConsumer are no longer in charge of committing
stuff for a task. Instead, it happens when an action is sent in. Tasks
are expected to ensure that this is OK by making their behavior
idempotent (since they may be retried after partially completing).

One new kind of request afforded to tasks is the ability to release
and acquire arbitrary locks. This will be useful for long-running tasks.
Short-lived tasks can still be given a lock automatically upon startup
by providing a "fixed interval".

New tables when using db storage:
- tasklocks, to support persistence of arbitrary locks.
- tasklogs, to log actions taken by tasks
 2013-02-14 12:46:40 -08:00
Eric Tschetter
a0b159fed7 1) Move up to Jackson 2.1 
2) Some bugs were fixed, but they ended up getting mixed in with the Jackson upgrade :(
 2013-02-12 20:58:17 -06:00
Fangjin Yang
e519f82e87 [maven-release-plugin] prepare for next development iteration 2013-02-11 10:58:51 -08:00
Fangjin Yang
73f7468a23 [maven-release-plugin] prepare release druid-0.2.7 2013-02-11 10:58:43 -08:00
Fangjin Yang
ec2b906fad [maven-release-plugin] prepare for next development iteration 2013-02-06 18:43:42 -08:00
Fangjin Yang
22d9607169 [maven-release-plugin] prepare release druid-0.2.6 2013-02-06 18:43:34 -08:00
Fangjin Yang
e09c658352 [maven-release-plugin] prepare for next development iteration 2013-02-06 17:42:39 -08:00
Fangjin Yang
668353e895 [maven-release-plugin] prepare release druid-0.2.5 2013-02-06 17:42:33 -08:00
Fangjin Yang
d023d219ae [maven-release-plugin] prepare for next development iteration 2013-02-06 16:54:21 -08:00
Fangjin Yang
d0fdf4ccf0 [maven-release-plugin] prepare release druid-0.2.4 2013-02-06 16:54:16 -08:00
Fangjin Yang
ea64eaabce Merge branch 'task-stuff' of github.com:metamx/druid into task-stuff 2013-02-01 17:16:39 -08:00
Fangjin Yang
d9f2e68f7f better logs 2013-02-01 17:16:24 -08:00
Eric Tschetter
34b4383934 1) Adjust DataSegment to have a binaryVersion field that indicates the expected binary version of the segment files 
2) Eliminate race condition from RemoteTaskRunnerTest.testAlreadyExecutedTask()
 2013-02-01 18:59:33 -06:00
Gian Merlino
4ec67f612e Merger: Emit indexer/segment/bytes once per segment. No longer emit indexer/segment/count. 2013-02-01 16:29:58 -08:00
xvrl
7ec3563e7b [maven-release-plugin] prepare for next development iteration 2013-02-01 16:25:42 -08:00
xvrl
a239f62158 [maven-release-plugin] prepare release druid-0.2.3 2013-02-01 16:25:35 -08:00
Fangjin Yang
d252059e4a Merge branch 'task-stuff' of github.com:metamx/druid into task-stuff 2013-02-01 12:59:04 -08:00
Fangjin Yang
58bc305e26 changes to remote task runner to kill nodes better 2013-02-01 12:58:38 -08:00
Gian Merlino
393bec0539 TaskQueue: Behavior tweaks, simplification 
- Lock tasks on add if possible
- Detect "already added" using exception test instead of bookkeeping map
- Update task status after commitRunnable instead of before commitRunnable
 2013-02-01 12:16:06 -08:00
Gian Merlino
455bae2c76 Merge branch 'master' into task-stuff 2013-01-31 11:28:23 -08:00
Fangjin Yang
f50c214514 Merge branch 'master' into task-stuff 2013-01-31 09:59:52 -08:00
Gian Merlino
f946fc3ee6 TaskConsumer: Commit to segment db in one transaction 2013-01-31 08:22:10 -08:00
Gian Merlino
1e35e6ad46 TaskQueue: Tweak to implementation of findTaskGroupForTask 2013-01-31 08:22:10 -08:00
Gian Merlino
34e76f868f TaskQueue: Clarify "Naptime!" log message 2013-01-31 08:22:10 -08:00
Gian Merlino
86ec3a372e TaskGroup: Add javadocs 2013-01-31 08:22:10 -08:00
Gian Merlino
65576cbade TaskStorageQueryAdapter: Rename some methods and tweak javadocs for clarity 2013-01-31 08:22:10 -08:00
Gian Merlino
779c54d8f2 TaskToolbox: Replace IndexerCoordinatorConfig with TaskConfig 2013-01-31 08:22:09 -08:00
Gian Merlino
fe38ed2547 Introduce TaskExistsException, thrown by TaskStorage.insert when appropriate 2013-01-31 08:22:09 -08:00
Gian Merlino
0e469c6f4c TaskStatus: Fix javadocs 2013-01-31 08:09:22 -08:00
Fangjin Yang
00e6ac1e10 [maven-release-plugin] prepare for next development iteration 2013-01-30 17:09:23 -08:00
Fangjin Yang
ef19de8270 [maven-release-plugin] prepare release druid-0.2.2 2013-01-30 17:09:18 -08:00
Fangjin Yang
26ed96f05d bug fix for autoscaling termination 2013-01-30 17:06:02 -08:00
Fangjin Yang
fa733565e8 [maven-release-plugin] prepare for next development iteration 2013-01-30 16:28:28 -08:00
Fangjin Yang
0b116a8fe4 [maven-release-plugin] prepare release druid-0.2.1 2013-01-30 16:28:22 -08:00
Fangjin Yang
2a0de2090b [maven-release-plugin] prepare for next development iteration 2013-01-30 13:47:15 -08:00
Fangjin Yang
049ca2bad4 [maven-release-plugin] prepare release druid-0.2.0 2013-01-30 13:47:08 -08:00
Fangjin Yang
74057600f9 fix worker node still using db connection 2013-01-27 09:57:37 -08:00
Gian Merlino
1cff766e2a RemoteTaskRunner: Only cleanup paths during bootstrapping if task is complete 2013-01-25 16:01:17 -08:00
Fangjin Yang
efdff7b9f1 fix unit test bug 2013-01-25 14:33:05 -08:00
Gian Merlino
0f4746450b RemoteTaskRunner: null check on workerData 2013-01-25 13:40:20 -08:00
Gian Merlino
bb68091cef Index service: TaskConsumer tests 2013-01-25 13:15:55 -08:00
Gian Merlino
82d77a8b72 Index service: Fix kill task 2013-01-25 13:15:49 -08:00
Gian Merlino
e6a618ca76 Merge branch 'killsegments' into task-stuff 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexGeneratorTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/MergeTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/LocalTaskRunner.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskQueue.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/exec/TaskConsumer.java
	merger/src/main/java/com/metamx/druid/merger/worker/TaskMonitor.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/TaskQueueTest.java
 2013-01-25 11:30:10 -08:00
Gian Merlino
a14200d779 Index service support for early returns and choice of commit semantics. 
Task:
- Add TaskCallback to run method (for early returns)

TaskStatus:
- Remove CONTINUED status
- Add segmentsNuked (placeholder for future deletion support)
- Add more builder methods
- Add validations to constructor

TaskStorage:
- Add TaskStorageQueryAdapter, a concrete class that wraps TaskStorages and
  provides various read-only convenience methods
- Add getTask method for benefit of TaskStorageQueryAdapter

TaskQueue:
- Rename "done" to "notify"
- notify is responsible for deciding if we should commit
- Add optional commitRunnable to "notify", which gets called when it's time to commit
- Allow nextTasks and commits to run early (statusCode RUNNING)
- Move getStatus, collapseStatus functionality to TaskStorageQueryAdapter
 2013-01-25 11:05:34 -08:00
Fangjin Yang
272d737517 cleaning up some interactions with RTR and workers 2013-01-22 16:21:38 -08:00
Fangjin Yang
bab9ee8827 Merge branch 'master' into killsegments 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
 2013-01-21 14:47:49 -08:00
cheddar
07131c51ed Merge pull request #53 from metamx/autoscaling 
Use a database to store configurations for indexer workers
 2013-01-18 17:08:52 -08:00
Gian Merlino
7166534666 YeOldePlumberSchool: Tweak for IndexIO changes 2013-01-17 16:03:18 -08:00