458 Commits

Author SHA1 Message Date
Fangjin Yang
49e61d00fd [maven-release-plugin] prepare for next development iteration 2013-03-01 19:05:46 -08:00
Fangjin Yang
7e316e3508 [maven-release-plugin] prepare release druid-0.3.11 2013-03-01 19:05:39 -08:00
Fangjin Yang
e21978603a Merge branch 'master' of github.com:metamx/druid 2013-03-01 19:03:29 -08:00
Fangjin Yang
df532f021a more logging 2013-03-01 19:03:16 -08:00
Gian Merlino
665d1723c0 Add TaskLockPosse#toString 2013-03-01 18:48:34 -08:00
Fangjin Yang
0b054cdd3d better logging for indexing service 2013-03-01 18:43:40 -08:00
Fangjin Yang
e2b4163e31 [maven-release-plugin] prepare for next development iteration 2013-03-01 18:24:09 -08:00
Fangjin Yang
023199702e [maven-release-plugin] prepare release druid-0.3.10 2013-03-01 18:24:03 -08:00
Fangjin Yang
a574da55ed fix broken UT 2013-03-01 18:22:30 -08:00
Fangjin Yang
8a9655a19e Merge branch 'master' of github.com:metamx/druid 2013-03-01 18:00:12 -08:00
Fangjin Yang
375e342d5f remove deadlock problem in pending tasks 2013-03-01 18:00:02 -08:00
Gian Merlino
0092ad7efb [maven-release-plugin] prepare for next development iteration 2013-03-01 16:57:47 -08:00
Gian Merlino
03ff72b7e2 [maven-release-plugin] prepare release druid-0.3.8 2013-03-01 16:57:38 -08:00
Fangjin Yang
c9cf091998 Merge branch 'master' of github.com:metamx/druid 2013-03-01 16:54:00 -08:00
Fangjin Yang
53021c2e35 more logging 2013-03-01 16:53:49 -08:00
Gian Merlino
998ed0a8de [maven-release-plugin] prepare for next development iteration 2013-03-01 14:43:20 -08:00
Gian Merlino
89d5ddf2f7 [maven-release-plugin] prepare release druid-0.3.7 2013-03-01 14:43:13 -08:00
Gian Merlino
6b4f6bffce DbTaskStorage: Fix getAuditLogs 2013-03-01 14:40:38 -08:00
Fangjin Yang
360c6a1df1 [maven-release-plugin] prepare for next development iteration 2013-03-01 13:55:07 -08:00
Fangjin Yang
a43dc44ebb [maven-release-plugin] prepare release druid-0.3.6 2013-03-01 13:55:01 -08:00
Fangjin Yang
3264a689f1 [maven-release-plugin] prepare for next development iteration 2013-03-01 11:43:23 -08:00
Fangjin Yang
334fffdef5 [maven-release-plugin] prepare release druid-0.3.5 2013-03-01 11:43:16 -08:00
Gian Merlino
e40fba4de2 HadoopIndexTask: Jackson fixes and general overriding of storage-specific stuff 2013-02-28 07:53:28 -08:00
Gian Merlino
f862d9205d Merge branch 'master' into hadoop-index-task
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
2013-02-27 06:53:06 -08:00
Fangjin Yang
2e1e1753d5 [maven-release-plugin] prepare for next development iteration 2013-02-26 15:15:54 -08:00
Fangjin Yang
ab456cf3b1 [maven-release-plugin] prepare release druid-0.3.4 2013-02-26 15:15:36 -08:00
Gian Merlino
d32a6284ce Merge branch 'task-stuff' into indexing_refactor
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/worker/config/WorkerConfig.java
2013-02-26 13:03:56 -08:00
Gian Merlino
d8fbddb9d4 Merger: Service discovery for worker -> master communication 2013-02-26 11:36:07 -08:00
Gian Merlino
2427e81874 Merger: Feedback from code review 2013-02-25 17:27:02 -08:00
Gian Merlino
3129e5451b Merge branch 'master' into task-stuff 2013-02-25 12:33:45 -08:00
Gian Merlino
9964e7dfe4 [maven-release-plugin] prepare for next development iteration 2013-02-25 12:24:24 -08:00
Gian Merlino
99c4f9446c [maven-release-plugin] prepare release druid-0.3.3 2013-02-25 12:24:15 -08:00
Gian Merlino
49700dffe9 Merge branch 'master' into task-stuff
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskToolbox.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexGeneratorTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/MergeTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
	merger/src/main/java/com/metamx/druid/merger/worker/http/WorkerNode.java
2013-02-25 12:18:31 -08:00
Eric Tschetter
e0f6df1a5c [maven-release-plugin] prepare for next development iteration 2013-02-21 20:39:50 -06:00
Eric Tschetter
846bc0e4f0 [maven-release-plugin] prepare release druid-0.3.2 2013-02-21 20:39:45 -06:00
Fangjin Yang
9dbb8841a1 initial commit 2013-02-21 18:16:14 -08:00
Eric Tschetter
699aadd2b1 [maven-release-plugin] prepare for next development iteration 2013-02-21 15:50:01 -06:00
Eric Tschetter
606a2e4b01 [maven-release-plugin] prepare release druid-0.3.1 2013-02-21 15:49:55 -06:00
Eric Tschetter
f8c54a72c2 1) Changes to allow for local storage 2013-02-21 15:47:01 -06:00
Eric Tschetter
dc3459d3f9 1) Initial commit of refactorings on top of housejester's to simplify the zipping and simplify the creation of other methods of loading data 2013-02-21 15:47:01 -06:00
James Estes
d1626576c0 Working toward making it easier to add new SegmentPullers.
1) Move the local cacheFile logic out of the S3 pullers into the SingleSegmentLoader
2) Make the S3SegmentPuller just pull down the file
3) Make the Loader do the unzip, ungzip, or rename
4) 2 and 3 make S3ZippedSegmentPuller not necessary (still there, just deprecated and empty)
4) Tweak the TaskToolbox so that the Pullers returned by getSegmentGetters behave the same as they did before
2013-02-21 15:47:01 -06:00
Deep Ganguli
60da9ffddb Merge branch 'master' into hadoop-index-task
Conflicts:
	common/src/main/java/com/metamx/druid/utils/JodaUtils.java
	indexer/src/main/java/com/metamx/druid/indexer/DbUpdaterJob.java
	indexer/src/main/java/com/metamx/druid/indexer/HadoopDruidIndexerConfig.java
	indexer/src/main/java/com/metamx/druid/indexer/IndexGeneratorJob.java
2013-02-15 13:22:31 -08:00
Gian Merlino
824e3c0eb2 [maven-release-plugin] prepare for next development iteration 2013-02-15 13:03:39 -08:00
Gian Merlino
6bbc992101 [maven-release-plugin] prepare release druid-0.3.0 2013-02-15 13:03:32 -08:00
Deep Ganguli
e042c29173 Fixed typos in comments, changed prefix on s3Paths to s3n from s3://, cleaned
up umbrellaIntervals code in JodaUtils, modified the DbUpdater to push segment
meta-data to the database in batches.
2013-02-15 11:39:49 -08:00
Fangjin Yang
bf0d9c601d Merge branch 'task-stuff' into indexing_refactor
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/TaskToolbox.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/KillTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
	merger/src/main/java/com/metamx/druid/merger/common/task/V8toV9UpgradeTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/RemoteTaskRunner.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskMasterLifecycle.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskQueue.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskRunner.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskRunnerWorkQueue.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorServletModule.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
2013-02-14 15:42:07 -08:00
Fangjin Yang
ef768feeb1 new resource 2013-02-14 14:20:20 -08:00
Gian Merlino
de4fc98128 TaskMasterLifecycle: Stop leaderLifecycle if it dies while starting 2013-02-14 13:36:43 -08:00
Gian Merlino
18daa19fc9 Merge branch 'master' into task-stuff
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskHolder.java
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/task/AbstractTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexGeneratorTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskContext.java
	merger/src/main/java/com/metamx/druid/merger/worker/TaskMonitor.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/TaskQueueTest.java
	server/src/main/java/com/metamx/druid/master/DruidMaster.java
2013-02-14 13:10:29 -08:00
Gian Merlino
ff325b1f65 Merger: Changes to support rolling locks and a task-action API
Generally, TaskStatus is no longer used as the sole way for tasks to
communicate status and metadata update requests back to the
coordinator. It is still used to communicate success or failure, but a
TaskActionClient is used for all other communication. In remote mode
this uses an http api.

TaskContext no longer exists, as its purpose is now served by
the TaskActionClient.

The TaskQueue and TaskConsumer are no longer in charge of committing
stuff for a task. Instead, it happens when an action is sent in. Tasks
are expected to ensure that this is OK by making their behavior
idempotent (since they may be retried after partially completing).

One new kind of request afforded to tasks is the ability to release
and acquire arbitrary locks. This will be useful for long-running tasks.
Short-lived tasks can still be given a lock automatically upon startup
by providing a "fixed interval".

New tables when using db storage:
- tasklocks, to support persistence of arbitrary locks.
- tasklogs, to log actions taken by tasks
2013-02-14 12:46:40 -08:00