Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
936 Commits 111 Branches 503 Tags 721 MiB
Commit Graph

393 Commits

Author SHA1 Message Date
Fangjin Yang 9d9efadf72 working indexer console 2013-03-06 10:38:40 -08:00
Eric Tschetter 6e95dd49b3 1) Remove Task parameter from other methods off of TaskToolbox that required it 
2) Move getTaskDir() off of TaskConfig and onto TaskToolbox
3) Rename SegmentKiller interface to DataSegmentKiller
4) Change signature of "DataSegmentKiller.kill(Collection<DataSegment>) throws ServiceException" to just kill(DataSegment) throws SegmentLoadingException
5) Add various log messages
6) Update the version of the segment that has been converted
 2013-03-05 13:30:50 -06:00
Fangjin Yang b410f5e805 [maven-release-plugin] prepare for next development iteration 2013-03-05 13:30:49 -06:00
Fangjin Yang c0ab1a503d [maven-release-plugin] prepare release druid-0.3.18 2013-03-05 13:30:49 -06:00
Fangjin Yang e51a01f85f fix broken UT 2013-03-05 13:30:49 -06:00
Fangjin Yang 665f1909c6 fixing a bug with indexing service not correctly killing worker nodes 2013-03-05 13:30:49 -06:00
Fangjin Yang 75cefa05b5 [maven-release-plugin] prepare for next development iteration 2013-03-05 13:30:49 -06:00
Fangjin Yang 90637ab7b7 [maven-release-plugin] prepare release druid-0.3.16 2013-03-05 13:30:49 -06:00
Fangjin Yang 848d49ca87 fix bugs in balancing logic and removing auto-scaled node 2013-03-05 13:30:49 -06:00
Eric Tschetter a9f7094ef3 1) Remove the need for TaskActions to require a Task as a constructor parameter 2013-03-04 18:10:32 -06:00
Fangjin Yang 10ec2288ee Merge branch 'master' into indexing_console 2013-03-04 15:23:34 -08:00
Fangjin Yang 552b365194 [maven-release-plugin] prepare for next development iteration 2013-03-04 15:11:37 -08:00
Fangjin Yang fdb3911794 [maven-release-plugin] prepare release druid-0.3.18 2013-03-04 15:11:28 -08:00
Fangjin Yang 91f54a72ee fix broken UT 2013-03-04 15:09:55 -08:00
Fangjin Yang 571375a33e fixing a bug with indexing service not correctly killing worker nodes 2013-03-04 14:40:02 -08:00
Fangjin Yang d18c9f2b0a Merge branch 'master' into indexing_console 2013-03-04 13:43:37 -08:00
Fangjin Yang 6977fe03bd [maven-release-plugin] prepare for next development iteration 2013-03-04 13:08:26 -08:00
Fangjin Yang d4f4e41320 [maven-release-plugin] prepare release druid-0.3.16 2013-03-04 13:08:09 -08:00
Fangjin Yang 88339fe99b Merge branch 'master' of github.com:metamx/druid 2013-03-04 13:03:27 -08:00
Fangjin Yang 92ab51e70c fix bugs in balancing logic and removing auto-scaled node 2013-03-04 13:03:15 -08:00
Eric Tschetter d7379db6fe 1) Initial commit: Converter Task 2013-03-04 13:44:42 -06:00
Gian Merlino 4e2107fbee Merger: Rename Task.getFixedInterval -> getImplicitLockInterval. Add more Task docs. 2013-03-04 11:23:36 -08:00
Fangjin Yang ee38d096c7 [maven-release-plugin] prepare for next development iteration 2013-03-02 09:29:04 -08:00
Fangjin Yang a90d204ba2 [maven-release-plugin] prepare release druid-0.3.15 2013-03-02 09:28:51 -08:00
Fangjin Yang aa63e0b25b account for duplicate events 2013-03-02 09:25:44 -08:00
Fangjin Yang e5cc6d241d cleaning up some code for RTR 2013-03-02 08:58:22 -08:00
Fangjin Yang 879b2475d4 [maven-release-plugin] prepare for next development iteration 2013-03-01 19:53:13 -08:00
Fangjin Yang e3ba6efb80 [maven-release-plugin] prepare release druid-0.3.14 2013-03-01 19:53:07 -08:00
Fangjin Yang bb83ddb564 fix bug with worker disappearance 2013-03-01 19:51:33 -08:00
Fangjin Yang 7f6747194d [maven-release-plugin] prepare for next development iteration 2013-03-01 19:45:59 -08:00
Fangjin Yang 06a2a3a793 [maven-release-plugin] prepare release druid-0.3.13 2013-03-01 19:45:50 -08:00
Fangjin Yang 1995039fb9 additional logs 2013-03-01 19:44:14 -08:00
Fangjin Yang e0700bb161 [maven-release-plugin] prepare for next development iteration 2013-03-01 19:28:34 -08:00
Fangjin Yang 38c8ffb552 [maven-release-plugin] prepare release druid-0.3.12 2013-03-01 19:28:28 -08:00
Fangjin Yang cbe0555d00 better logging 2013-03-01 19:25:25 -08:00
Fangjin Yang 49e61d00fd [maven-release-plugin] prepare for next development iteration 2013-03-01 19:05:46 -08:00
Fangjin Yang 7e316e3508 [maven-release-plugin] prepare release druid-0.3.11 2013-03-01 19:05:39 -08:00
Fangjin Yang e21978603a Merge branch 'master' of github.com:metamx/druid 2013-03-01 19:03:29 -08:00
Fangjin Yang df532f021a more logging 2013-03-01 19:03:16 -08:00
Gian Merlino 665d1723c0 Add TaskLockPosse#toString 2013-03-01 18:48:34 -08:00
Fangjin Yang 0b054cdd3d better logging for indexing service 2013-03-01 18:43:40 -08:00
Fangjin Yang e2b4163e31 [maven-release-plugin] prepare for next development iteration 2013-03-01 18:24:09 -08:00
Fangjin Yang 023199702e [maven-release-plugin] prepare release druid-0.3.10 2013-03-01 18:24:03 -08:00
Fangjin Yang a574da55ed fix broken UT 2013-03-01 18:22:30 -08:00
Fangjin Yang 8a9655a19e Merge branch 'master' of github.com:metamx/druid 2013-03-01 18:00:12 -08:00
Fangjin Yang 375e342d5f remove deadlock problem in pending tasks 2013-03-01 18:00:02 -08:00
Gian Merlino 0092ad7efb [maven-release-plugin] prepare for next development iteration 2013-03-01 16:57:47 -08:00
Gian Merlino 03ff72b7e2 [maven-release-plugin] prepare release druid-0.3.8 2013-03-01 16:57:38 -08:00
Fangjin Yang c9cf091998 Merge branch 'master' of github.com:metamx/druid 2013-03-01 16:54:00 -08:00
Fangjin Yang 53021c2e35 more logging 2013-03-01 16:53:49 -08:00
Gian Merlino 998ed0a8de [maven-release-plugin] prepare for next development iteration 2013-03-01 14:43:20 -08:00
Gian Merlino 89d5ddf2f7 [maven-release-plugin] prepare release druid-0.3.7 2013-03-01 14:43:13 -08:00
Gian Merlino 6b4f6bffce DbTaskStorage: Fix getAuditLogs 2013-03-01 14:40:38 -08:00
Fangjin Yang 360c6a1df1 [maven-release-plugin] prepare for next development iteration 2013-03-01 13:55:07 -08:00
Fangjin Yang a43dc44ebb [maven-release-plugin] prepare release druid-0.3.6 2013-03-01 13:55:01 -08:00
Fangjin Yang 3264a689f1 [maven-release-plugin] prepare for next development iteration 2013-03-01 11:43:23 -08:00
Fangjin Yang 334fffdef5 [maven-release-plugin] prepare release druid-0.3.5 2013-03-01 11:43:16 -08:00
Gian Merlino e40fba4de2 HadoopIndexTask: Jackson fixes and general overriding of storage-specific stuff 2013-02-28 07:53:28 -08:00
Gian Merlino f862d9205d Merge branch 'master' into hadoop-index-task 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
 2013-02-27 06:53:06 -08:00
Fangjin Yang 2e1e1753d5 [maven-release-plugin] prepare for next development iteration 2013-02-26 15:15:54 -08:00
Fangjin Yang ab456cf3b1 [maven-release-plugin] prepare release druid-0.3.4 2013-02-26 15:15:36 -08:00
Gian Merlino d32a6284ce Merge branch 'task-stuff' into indexing_refactor 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/worker/config/WorkerConfig.java
 2013-02-26 13:03:56 -08:00
Gian Merlino d8fbddb9d4 Merger: Service discovery for worker -> master communication 2013-02-26 11:36:07 -08:00
Gian Merlino 2427e81874 Merger: Feedback from code review 2013-02-25 17:27:02 -08:00
Gian Merlino 3129e5451b Merge branch 'master' into task-stuff 2013-02-25 12:33:45 -08:00
Gian Merlino 9964e7dfe4 [maven-release-plugin] prepare for next development iteration 2013-02-25 12:24:24 -08:00
Gian Merlino 99c4f9446c [maven-release-plugin] prepare release druid-0.3.3 2013-02-25 12:24:15 -08:00
Gian Merlino 49700dffe9 Merge branch 'master' into task-stuff 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskToolbox.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexGeneratorTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/MergeTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
	merger/src/main/java/com/metamx/druid/merger/worker/http/WorkerNode.java
 2013-02-25 12:18:31 -08:00
Eric Tschetter e0f6df1a5c [maven-release-plugin] prepare for next development iteration 2013-02-21 20:39:50 -06:00
Eric Tschetter 846bc0e4f0 [maven-release-plugin] prepare release druid-0.3.2 2013-02-21 20:39:45 -06:00
Fangjin Yang 9dbb8841a1 initial commit 2013-02-21 18:16:14 -08:00
Eric Tschetter 699aadd2b1 [maven-release-plugin] prepare for next development iteration 2013-02-21 15:50:01 -06:00
Eric Tschetter 606a2e4b01 [maven-release-plugin] prepare release druid-0.3.1 2013-02-21 15:49:55 -06:00
Eric Tschetter f8c54a72c2 1) Changes to allow for local storage 2013-02-21 15:47:01 -06:00
Eric Tschetter dc3459d3f9 1) Initial commit of refactorings on top of housejester's to simplify the zipping and simplify the creation of other methods of loading data 2013-02-21 15:47:01 -06:00
James Estes d1626576c0 Working toward making it easier to add new SegmentPullers. 
1) Move the local cacheFile logic out of the S3 pullers into the SingleSegmentLoader
2) Make the S3SegmentPuller just pull down the file
3) Make the Loader do the unzip, ungzip, or rename
4) 2 and 3 make S3ZippedSegmentPuller not necessary (still there, just deprecated and empty)
4) Tweak the TaskToolbox so that the Pullers returned by getSegmentGetters behave the same as they did before
 2013-02-21 15:47:01 -06:00
Deep Ganguli 60da9ffddb Merge branch 'master' into hadoop-index-task 
Conflicts:
	common/src/main/java/com/metamx/druid/utils/JodaUtils.java
	indexer/src/main/java/com/metamx/druid/indexer/DbUpdaterJob.java
	indexer/src/main/java/com/metamx/druid/indexer/HadoopDruidIndexerConfig.java
	indexer/src/main/java/com/metamx/druid/indexer/IndexGeneratorJob.java
 2013-02-15 13:22:31 -08:00
Gian Merlino 824e3c0eb2 [maven-release-plugin] prepare for next development iteration 2013-02-15 13:03:39 -08:00
Gian Merlino 6bbc992101 [maven-release-plugin] prepare release druid-0.3.0 2013-02-15 13:03:32 -08:00
Deep Ganguli e042c29173 Fixed typos in comments, changed prefix on s3Paths to s3n from s3://, cleaned 
up umbrellaIntervals code in JodaUtils, modified the DbUpdater to push segment
meta-data to the database in batches.
 2013-02-15 11:39:49 -08:00
Fangjin Yang bf0d9c601d Merge branch 'task-stuff' into indexing_refactor 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/TaskToolbox.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/KillTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
	merger/src/main/java/com/metamx/druid/merger/common/task/V8toV9UpgradeTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/RemoteTaskRunner.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskMasterLifecycle.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskQueue.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskRunner.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskRunnerWorkQueue.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorServletModule.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
 2013-02-14 15:42:07 -08:00
Fangjin Yang ef768feeb1 new resource 2013-02-14 14:20:20 -08:00
Gian Merlino de4fc98128 TaskMasterLifecycle: Stop leaderLifecycle if it dies while starting 2013-02-14 13:36:43 -08:00
Gian Merlino 18daa19fc9 Merge branch 'master' into task-stuff 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskHolder.java
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/task/AbstractTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexGeneratorTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskContext.java
	merger/src/main/java/com/metamx/druid/merger/worker/TaskMonitor.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/TaskQueueTest.java
	server/src/main/java/com/metamx/druid/master/DruidMaster.java
 2013-02-14 13:10:29 -08:00
Gian Merlino ff325b1f65 Merger: Changes to support rolling locks and a task-action API 
Generally, TaskStatus is no longer used as the sole way for tasks to
communicate status and metadata update requests back to the
coordinator. It is still used to communicate success or failure, but a
TaskActionClient is used for all other communication. In remote mode
this uses an http api.

TaskContext no longer exists, as its purpose is now served by
the TaskActionClient.

The TaskQueue and TaskConsumer are no longer in charge of committing
stuff for a task. Instead, it happens when an action is sent in. Tasks
are expected to ensure that this is OK by making their behavior
idempotent (since they may be retried after partially completing).

One new kind of request afforded to tasks is the ability to release
and acquire arbitrary locks. This will be useful for long-running tasks.
Short-lived tasks can still be given a lock automatically upon startup
by providing a "fixed interval".

New tables when using db storage:
- tasklocks, to support persistence of arbitrary locks.
- tasklogs, to log actions taken by tasks
 2013-02-14 12:46:40 -08:00
Eric Tschetter c5b78e75cf 1) Move up to Jackson 2.1 
2) Some bugs were fixed, but they ended up getting mixed in with the Jackson upgrade :(
 2013-02-13 18:51:32 -08:00
Fangjin Yang 82f02a4fe4 Merge branch 'master' into indexing_refactor 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
	merger/src/main/java/com/metamx/druid/merger/common/task/V8toV9UpgradeTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/RemoteTaskRunner.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
 2013-02-13 16:34:14 -08:00
Fangjin Yang 1963bfe83f cleanup 2013-02-13 16:04:43 -08:00
Fangjin Yang c7b4973700 bug fixes for RTR 2013-02-13 14:19:11 -08:00
Eric Tschetter a0b159fed7 1) Move up to Jackson 2.1 
2) Some bugs were fixed, but they ended up getting mixed in with the Jackson upgrade :(
 2013-02-12 20:58:17 -06:00
xvrl 07b0b4611b bump up to 0.3.0 2013-02-12 09:52:21 -08:00
Fangjin Yang b27a459f71 more refactoring and UT passing at this point 2013-02-11 18:13:01 -08:00
Fangjin Yang a9176e738a Merge branch 'master' into indexing_refactor 2013-02-11 12:36:49 -08:00
Fangjin Yang e519f82e87 [maven-release-plugin] prepare for next development iteration 2013-02-11 10:58:51 -08:00
Fangjin Yang 73f7468a23 [maven-release-plugin] prepare release druid-0.2.7 2013-02-11 10:58:43 -08:00
Fangjin Yang 111df9a90e Merge branch 'master' into indexing_refactor 
Conflicts:
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
	server/src/main/java/com/metamx/druid/master/DruidMaster.java
 2013-02-07 10:35:32 -08:00
Fangjin Yang ec2b906fad [maven-release-plugin] prepare for next development iteration 2013-02-06 18:43:42 -08:00
Fangjin Yang 22d9607169 [maven-release-plugin] prepare release druid-0.2.6 2013-02-06 18:43:34 -08:00
Fangjin Yang e09c658352 [maven-release-plugin] prepare for next development iteration 2013-02-06 17:42:39 -08:00
Fangjin Yang 668353e895 [maven-release-plugin] prepare release druid-0.2.5 2013-02-06 17:42:33 -08:00
Fangjin Yang d023d219ae [maven-release-plugin] prepare for next development iteration 2013-02-06 16:54:21 -08:00
Fangjin Yang d0fdf4ccf0 [maven-release-plugin] prepare release druid-0.2.4 2013-02-06 16:54:16 -08:00
Fangjin Yang efd0dc0062 refactor scaling stuff 2013-02-06 11:09:49 -08:00
Fangjin Yang ea64eaabce Merge branch 'task-stuff' of github.com:metamx/druid into task-stuff 2013-02-01 17:16:39 -08:00
Fangjin Yang d9f2e68f7f better logs 2013-02-01 17:16:24 -08:00
Eric Tschetter 34b4383934 1) Adjust DataSegment to have a binaryVersion field that indicates the expected binary version of the segment files 
2) Eliminate race condition from RemoteTaskRunnerTest.testAlreadyExecutedTask()
 2013-02-01 18:59:33 -06:00
Gian Merlino 4ec67f612e Merger: Emit indexer/segment/bytes once per segment. No longer emit indexer/segment/count. 2013-02-01 16:29:58 -08:00
xvrl 7ec3563e7b [maven-release-plugin] prepare for next development iteration 2013-02-01 16:25:42 -08:00
xvrl a239f62158 [maven-release-plugin] prepare release druid-0.2.3 2013-02-01 16:25:35 -08:00
Fangjin Yang d252059e4a Merge branch 'task-stuff' of github.com:metamx/druid into task-stuff 2013-02-01 12:59:04 -08:00
Fangjin Yang 58bc305e26 changes to remote task runner to kill nodes better 2013-02-01 12:58:38 -08:00
Gian Merlino 393bec0539 TaskQueue: Behavior tweaks, simplification 
- Lock tasks on add if possible
- Detect "already added" using exception test instead of bookkeeping map
- Update task status after commitRunnable instead of before commitRunnable
 2013-02-01 12:16:06 -08:00
Gian Merlino 455bae2c76 Merge branch 'master' into task-stuff 2013-01-31 11:28:23 -08:00
Fangjin Yang f50c214514 Merge branch 'master' into task-stuff 2013-01-31 09:59:52 -08:00
Gian Merlino f946fc3ee6 TaskConsumer: Commit to segment db in one transaction 2013-01-31 08:22:10 -08:00
Gian Merlino 1e35e6ad46 TaskQueue: Tweak to implementation of findTaskGroupForTask 2013-01-31 08:22:10 -08:00
Gian Merlino 34e76f868f TaskQueue: Clarify "Naptime!" log message 2013-01-31 08:22:10 -08:00
Gian Merlino 86ec3a372e TaskGroup: Add javadocs 2013-01-31 08:22:10 -08:00
Gian Merlino 65576cbade TaskStorageQueryAdapter: Rename some methods and tweak javadocs for clarity 2013-01-31 08:22:10 -08:00
Gian Merlino 779c54d8f2 TaskToolbox: Replace IndexerCoordinatorConfig with TaskConfig 2013-01-31 08:22:09 -08:00
Gian Merlino fe38ed2547 Introduce TaskExistsException, thrown by TaskStorage.insert when appropriate 2013-01-31 08:22:09 -08:00
Gian Merlino 0e469c6f4c TaskStatus: Fix javadocs 2013-01-31 08:09:22 -08:00
Fangjin Yang 00e6ac1e10 [maven-release-plugin] prepare for next development iteration 2013-01-30 17:09:23 -08:00
Fangjin Yang ef19de8270 [maven-release-plugin] prepare release druid-0.2.2 2013-01-30 17:09:18 -08:00
Fangjin Yang 26ed96f05d bug fix for autoscaling termination 2013-01-30 17:06:02 -08:00
Fangjin Yang fa733565e8 [maven-release-plugin] prepare for next development iteration 2013-01-30 16:28:28 -08:00
Fangjin Yang 0b116a8fe4 [maven-release-plugin] prepare release druid-0.2.1 2013-01-30 16:28:22 -08:00
Fangjin Yang 2a0de2090b [maven-release-plugin] prepare for next development iteration 2013-01-30 13:47:15 -08:00
Fangjin Yang 049ca2bad4 [maven-release-plugin] prepare release druid-0.2.0 2013-01-30 13:47:08 -08:00
Fangjin Yang 74057600f9 fix worker node still using db connection 2013-01-27 09:57:37 -08:00
Gian Merlino 1cff766e2a RemoteTaskRunner: Only cleanup paths during bootstrapping if task is complete 2013-01-25 16:01:17 -08:00
Fangjin Yang efdff7b9f1 fix unit test bug 2013-01-25 14:33:05 -08:00
Gian Merlino 0f4746450b RemoteTaskRunner: null check on workerData 2013-01-25 13:40:20 -08:00
Gian Merlino bb68091cef Index service: TaskConsumer tests 2013-01-25 13:15:55 -08:00
Gian Merlino 82d77a8b72 Index service: Fix kill task 2013-01-25 13:15:49 -08:00
Gian Merlino e6a618ca76 Merge branch 'killsegments' into task-stuff 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexGeneratorTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/MergeTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/LocalTaskRunner.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskQueue.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/exec/TaskConsumer.java
	merger/src/main/java/com/metamx/druid/merger/worker/TaskMonitor.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/TaskQueueTest.java
 2013-01-25 11:30:10 -08:00
Gian Merlino a14200d779 Index service support for early returns and choice of commit semantics. 
Task:
- Add TaskCallback to run method (for early returns)

TaskStatus:
- Remove CONTINUED status
- Add segmentsNuked (placeholder for future deletion support)
- Add more builder methods
- Add validations to constructor

TaskStorage:
- Add TaskStorageQueryAdapter, a concrete class that wraps TaskStorages and
  provides various read-only convenience methods
- Add getTask method for benefit of TaskStorageQueryAdapter

TaskQueue:
- Rename "done" to "notify"
- notify is responsible for deciding if we should commit
- Add optional commitRunnable to "notify", which gets called when it's time to commit
- Allow nextTasks and commits to run early (statusCode RUNNING)
- Move getStatus, collapseStatus functionality to TaskStorageQueryAdapter
 2013-01-25 11:05:34 -08:00
Deep Ganguli cb845e6f09 Addresed Gian's code review 2013-01-24 17:54:06 -08:00
Deep Ganguli 017d4779d6 Implemented Hadoop Index Task which takes as input a HadoopDruidIndexConfig and 
generates index segments.

The HadoopIndexTask run method wraps a HadoopDruidIndexerJob run method. The
key modifications to the HadoopDruidIndexerJob are as follows:

- The UpDaterJobSpec field of the config that is used to set up the indexer job
  is set to null. This ensures that the job does not push a list of published
segments to the database, in order to allow the indexing service to handle this
later.
- Set the version field of the config file based on the TaskContext. Also
  changed config.setVersion method to take a string (as opposed to a Date) as
input, and propogated this change where necessary.
- Set the SegmentOutputDir field of the config file based on the TaskToolbox,
  to allow the indexing service to handle where to write the segments too.
- Added a method to IndexGeneratorJob called getPublishedSegments, that simply
  returns a list of published segments without publishing this list to the
database.
 2013-01-23 19:27:14 -08:00
Fangjin Yang 272d737517 cleaning up some interactions with RTR and workers 2013-01-22 16:21:38 -08:00
Fangjin Yang bab9ee8827 Merge branch 'master' into killsegments 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
 2013-01-21 14:47:49 -08:00
cheddar 07131c51ed Merge pull request #53 from metamx/autoscaling 
Use a database to store configurations for indexer workers
 2013-01-18 17:08:52 -08:00
Gian Merlino 7166534666 YeOldePlumberSchool: Tweak for IndexIO changes 2013-01-17 16:03:18 -08:00
Gian Merlino 5ce53eb2ac Merge branch 'master' into yops-descriptor-dimensions 2013-01-17 16:02:32 -08:00
Fangjin Yang 38b2041ad9 key/value config table 2013-01-17 14:56:48 -08:00
Fangjin Yang 21613bc73b initial commit to hard delete segments 2013-01-16 17:31:01 -08:00
Eric Tschetter c8cb96b006 1) Remove vast majority of usages of IndexIO.mapDir() and deprecated it. IndexIO.loadIndex() is the new IndexIO.mapDir() 
2) Fix bug with IndexMerger and null columns
3) Add QueryableIndexIndexableAdapter so that QueryableIndexes can be merged
4) Adjust twitter example to have multiple values for each hash tag
5) Adjusted GroupByQueryEngine to just drop dimensions that don't exist instead of throwing an NPE
 2013-01-16 17:10:33 -06:00
Gian Merlino 6fc350bfba YeOldePlumberSchool: Populate dimension metadata in segment descriptor 2013-01-16 11:30:24 -08:00
Fangjin Yang 3c0880fe28 Merge branch 'master' into autoscaling 2013-01-15 13:36:39 -08:00
Fangjin Yang 7e074e8158 fix pom breakage 2013-01-15 12:04:12 -08:00
Fangjin Yang d1f4317af7 updates to autoscaling config based on code review comments 2013-01-14 14:55:04 -08:00
Fangjin Yang 4c2da93389 bug fixes with ec2 auto scaling 2013-01-09 14:51:35 -08:00
Fangjin Yang 3685839376 Merge branch 'master' into autoscaling 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
 2013-01-07 18:04:14 -08:00
Fangjin Yang 79e9ff94a6 add missing headers 2013-01-07 14:07:47 -08:00
Fangjin Yang aff4a05ca3 db based configs for indexer workers 2013-01-07 13:41:26 -08:00
Eric Tschetter b184c46ae0 1) Introduce idea of value types and generic columns to the serialization format 
2) Create SegmentLoader interface for the loading of segments!
3) Setup serialization of new format and conversion function inside IndexIO
4) Make some adjustments to the twitter demo to make it make a bit more sense based on some email feedback from someone kicking the tires.
 2013-01-04 18:12:49 -06:00
xvrl 9627fdcd1c [maven-release-plugin] prepare for next development iteration 2013-01-03 10:09:52 -08:00
xvrl f2c2d3f4cf [maven-release-plugin] prepare release druid-0.1.25 2013-01-03 10:09:43 -08:00
xvrl 2a188996cd [maven-release-plugin] prepare for next development iteration 2012-12-31 14:56:10 -08:00
xvrl 6cdcc2becc [maven-release-plugin] prepare release druid-0.1.24 2012-12-31 14:56:03 -08:00
cheddar 438b85d2dd Merge pull request #46 from metamx/short-alerts 
Merger: De-parameterize alert messages
 2012-12-27 14:08:02 -08:00
Fangjin Yang 5137af0750 [maven-release-plugin] prepare for next development iteration 2012-12-18 16:17:44 -08:00
Fangjin Yang c579ba5a42 [maven-release-plugin] prepare release druid-0.1.23 2012-12-18 16:17:38 -08:00
Fangjin Yang 090a5a105d [maven-release-plugin] prepare for next development iteration 2012-12-18 14:19:09 -08:00
Fangjin Yang fb8f4f1f41 [maven-release-plugin] prepare release druid-0.1.22 2012-12-18 14:19:02 -08:00
Fangjin Yang 75d465aed0 [maven-release-plugin] prepare for next development iteration 2012-12-18 10:37:16 -08:00
Fangjin Yang decc4e4c0e [maven-release-plugin] prepare release druid-0.1.21 2012-12-18 10:37:08 -08:00
Fangjin Yang e34944b1c5 [maven-release-plugin] prepare for next development iteration 2012-12-15 00:52:01 -08:00
Fangjin Yang 24f0207fd8 [maven-release-plugin] prepare release druid-0.1.20 2012-12-15 00:51:54 -08:00
Fangjin Yang f9386c3623 [maven-release-plugin] prepare for next development iteration 2012-12-15 00:44:58 -08:00
Fangjin Yang 4bcc8d27fa [maven-release-plugin] prepare release druid-0.1.19 2012-12-15 00:44:50 -08:00
Fangjin Yang 5fff07a8b0 [maven-release-plugin] prepare for next development iteration 2012-12-14 22:56:49 -08:00
Fangjin Yang 2b2818dbaf [maven-release-plugin] prepare release druid-0.1.18 2012-12-14 22:56:43 -08:00
Fangjin Yang b0e1f90e35 [maven-release-plugin] prepare for next development iteration 2012-12-14 22:07:57 -08:00
Fangjin Yang ce8e02b5a3 [maven-release-plugin] prepare release druid-0.1.17 2012-12-14 22:07:50 -08:00
Fangjin Yang 0821dce8c1 [maven-release-plugin] prepare for next development iteration 2012-12-14 21:29:20 -08:00
Fangjin Yang a0d22a6646 [maven-release-plugin] prepare release druid-0.1.16 2012-12-14 21:29:14 -08:00
Gian Merlino 61afd0a46a Merger: De-parameterize alert messages 2012-12-14 18:14:44 -08:00
Fangjin Yang e37e24cc25 [maven-release-plugin] prepare for next development iteration 2012-12-14 17:19:59 -08:00
Fangjin Yang 4e78d8e734 [maven-release-plugin] prepare release druid-0.1.15 2012-12-14 17:19:53 -08:00
Eric Tschetter b4a1bb4a00 [maven-release-plugin] prepare for next development iteration 2012-12-14 11:37:17 -06:00
Eric Tschetter cb9b18c7c2 [maven-release-plugin] prepare release druid-0.1.14 2012-12-14 11:37:12 -06:00
Eric Tschetter 29a64f50e2 [maven-release-plugin] prepare for next development iteration 2012-12-13 17:57:46 -06:00
Eric Tschetter c82007973e [maven-release-plugin] prepare release druid-0.1.13 2012-12-13 17:57:42 -06:00
Fangjin Yang 6729f166d9 [maven-release-plugin] prepare for next development iteration 2012-12-12 15:38:58 -08:00
Fangjin Yang 578cf94bc0 [maven-release-plugin] prepare release druid-0.1.12 2012-12-12 15:38:37 -08:00
Gian Merlino 091ed069f5 RemoteTaskRunner: More robust task status listener 
- Handle CHILD_ADDED and CHILD_UPDATED together, to cover the case where a worker
  reconnects to zk in the middle of a task (which makes the ephemeral status node
  disappear and come back as an addition)
- There's no need to retry on most exceptions that can occur here, so don't
 2012-12-12 12:51:04 -08:00
Fangjin Yang 0105233fc3 [maven-release-plugin] prepare for next development iteration 2012-12-07 17:50:47 -08:00
Fangjin Yang 943fe5e6bf [maven-release-plugin] prepare release druid-0.1.11 2012-12-07 17:50:41 -08:00
Fangjin Yang da4b31c28f [maven-release-plugin] prepare for next development iteration 2012-12-07 15:18:19 -08:00
Fangjin Yang 6df32ac0b0 [maven-release-plugin] prepare release druid-0.1.10 2012-12-07 15:18:11 -08:00
Fangjin Yang 9eb28254f8 [maven-release-plugin] prepare for next development iteration 2012-12-07 14:49:13 -08:00
Fangjin Yang 9ce536e040 [maven-release-plugin] prepare release druid-0.1.9 2012-12-07 14:49:04 -08:00
Fangjin Yang 35362b3e6b [maven-release-plugin] prepare for next development iteration 2012-12-07 14:37:25 -08:00
Fangjin Yang bd3c05dddb [maven-release-plugin] prepare release druid-0.1.8 2012-12-07 14:37:18 -08:00
Fangjin Yang 50a28f827e [maven-release-plugin] prepare for next development iteration 2012-12-07 10:59:52 -08:00
Fangjin Yang b59361065e [maven-release-plugin] prepare release druid-0.1.7 2012-12-07 10:59:34 -08:00
Richard Benkovsky e950c6333d RemoteTaskRunner#run should fail on task with taskId from tasks.keys set 2012-11-29 09:42:28 +01:00
Gian Merlino cc8a14d5a1 IndexGeneratorTask: Log metrics when complete 2012-11-27 15:46:03 -08:00
Eric Tschetter 0390e7fa5c [maven-release-plugin] prepare for next development iteration 2012-11-21 22:13:15 -06:00