Apache/druid
1
0
Fork 0
mirror of https://github.com/apache/druid.git synced 2025-02-10 12:05:00 +00:00
Code Issues Packages Projects Releases Wiki Activity
561 Commits 116 Branches 507 Tags
Commit Graph

195 Commits

Author SHA1 Message Date
Fangjin Yang
375e342d5f remove deadlock problem in pending tasks 2013-03-01 18:00:02 -08:00
Fangjin Yang
c9cf091998 Merge branch 'master' of github.com:metamx/druid 2013-03-01 16:54:00 -08:00
Fangjin Yang
53021c2e35 more logging 2013-03-01 16:53:49 -08:00
Gian Merlino
998ed0a8de [maven-release-plugin] prepare for next development iteration 2013-03-01 14:43:20 -08:00
Gian Merlino
89d5ddf2f7 [maven-release-plugin] prepare release druid-0.3.7 2013-03-01 14:43:13 -08:00
Gian Merlino
6b4f6bffce DbTaskStorage: Fix getAuditLogs 2013-03-01 14:40:38 -08:00
Fangjin Yang
360c6a1df1 [maven-release-plugin] prepare for next development iteration 2013-03-01 13:55:07 -08:00
Fangjin Yang
a43dc44ebb [maven-release-plugin] prepare release druid-0.3.6 2013-03-01 13:55:01 -08:00
Fangjin Yang
3264a689f1 [maven-release-plugin] prepare for next development iteration 2013-03-01 11:43:23 -08:00
Fangjin Yang
334fffdef5 [maven-release-plugin] prepare release druid-0.3.5 2013-03-01 11:43:16 -08:00
Gian Merlino
e40fba4de2 HadoopIndexTask: Jackson fixes and general overriding of storage-specific stuff 2013-02-28 07:53:28 -08:00
Gian Merlino
f862d9205d Merge branch 'master' into hadoop-index-task 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
 2013-02-27 06:53:06 -08:00
Fangjin Yang
2e1e1753d5 [maven-release-plugin] prepare for next development iteration 2013-02-26 15:15:54 -08:00
Fangjin Yang
ab456cf3b1 [maven-release-plugin] prepare release druid-0.3.4 2013-02-26 15:15:36 -08:00
Gian Merlino
d32a6284ce Merge branch 'task-stuff' into indexing_refactor 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/worker/config/WorkerConfig.java
 2013-02-26 13:03:56 -08:00
Gian Merlino
d8fbddb9d4 Merger: Service discovery for worker -> master communication 2013-02-26 11:36:07 -08:00
Gian Merlino
2427e81874 Merger: Feedback from code review 2013-02-25 17:27:02 -08:00
Gian Merlino
3129e5451b Merge branch 'master' into task-stuff 2013-02-25 12:33:45 -08:00
Gian Merlino
9964e7dfe4 [maven-release-plugin] prepare for next development iteration 2013-02-25 12:24:24 -08:00
Gian Merlino
99c4f9446c [maven-release-plugin] prepare release druid-0.3.3 2013-02-25 12:24:15 -08:00
Gian Merlino
49700dffe9 Merge branch 'master' into task-stuff 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskToolbox.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexGeneratorTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/MergeTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
	merger/src/main/java/com/metamx/druid/merger/worker/http/WorkerNode.java
 2013-02-25 12:18:31 -08:00
Eric Tschetter
e0f6df1a5c [maven-release-plugin] prepare for next development iteration 2013-02-21 20:39:50 -06:00
Eric Tschetter
846bc0e4f0 [maven-release-plugin] prepare release druid-0.3.2 2013-02-21 20:39:45 -06:00
Eric Tschetter
699aadd2b1 [maven-release-plugin] prepare for next development iteration 2013-02-21 15:50:01 -06:00
Eric Tschetter
606a2e4b01 [maven-release-plugin] prepare release druid-0.3.1 2013-02-21 15:49:55 -06:00
Eric Tschetter
f8c54a72c2 1) Changes to allow for local storage 2013-02-21 15:47:01 -06:00
Eric Tschetter
dc3459d3f9 1) Initial commit of refactorings on top of housejester's to simplify the zipping and simplify the creation of other methods of loading data 2013-02-21 15:47:01 -06:00
James Estes
d1626576c0 Working toward making it easier to add new SegmentPullers. 
1) Move the local cacheFile logic out of the S3 pullers into the SingleSegmentLoader
2) Make the S3SegmentPuller just pull down the file
3) Make the Loader do the unzip, ungzip, or rename
4) 2 and 3 make S3ZippedSegmentPuller not necessary (still there, just deprecated and empty)
4) Tweak the TaskToolbox so that the Pullers returned by getSegmentGetters behave the same as they did before
 2013-02-21 15:47:01 -06:00
Deep Ganguli
60da9ffddb Merge branch 'master' into hadoop-index-task 
Conflicts:
	common/src/main/java/com/metamx/druid/utils/JodaUtils.java
	indexer/src/main/java/com/metamx/druid/indexer/DbUpdaterJob.java
	indexer/src/main/java/com/metamx/druid/indexer/HadoopDruidIndexerConfig.java
	indexer/src/main/java/com/metamx/druid/indexer/IndexGeneratorJob.java
 2013-02-15 13:22:31 -08:00
Gian Merlino
824e3c0eb2 [maven-release-plugin] prepare for next development iteration 2013-02-15 13:03:39 -08:00
Gian Merlino
6bbc992101 [maven-release-plugin] prepare release druid-0.3.0 2013-02-15 13:03:32 -08:00
Deep Ganguli
e042c29173 Fixed typos in comments, changed prefix on s3Paths to s3n from s3://, cleaned 
up umbrellaIntervals code in JodaUtils, modified the DbUpdater to push segment
meta-data to the database in batches.
 2013-02-15 11:39:49 -08:00
Fangjin Yang
bf0d9c601d Merge branch 'task-stuff' into indexing_refactor 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/TaskToolbox.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/KillTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
	merger/src/main/java/com/metamx/druid/merger/common/task/V8toV9UpgradeTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/RemoteTaskRunner.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskMasterLifecycle.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskQueue.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskRunner.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskRunnerWorkQueue.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorServletModule.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
 2013-02-14 15:42:07 -08:00
Fangjin Yang
ef768feeb1 new resource 2013-02-14 14:20:20 -08:00
Gian Merlino
de4fc98128 TaskMasterLifecycle: Stop leaderLifecycle if it dies while starting 2013-02-14 13:36:43 -08:00
Gian Merlino
18daa19fc9 Merge branch 'master' into task-stuff 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskHolder.java
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/task/AbstractTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexGeneratorTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskContext.java
	merger/src/main/java/com/metamx/druid/merger/worker/TaskMonitor.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/TaskQueueTest.java
	server/src/main/java/com/metamx/druid/master/DruidMaster.java
 2013-02-14 13:10:29 -08:00
Gian Merlino
ff325b1f65 Merger: Changes to support rolling locks and a task-action API 
Generally, TaskStatus is no longer used as the sole way for tasks to
communicate status and metadata update requests back to the
coordinator. It is still used to communicate success or failure, but a
TaskActionClient is used for all other communication. In remote mode
this uses an http api.

TaskContext no longer exists, as its purpose is now served by
the TaskActionClient.

The TaskQueue and TaskConsumer are no longer in charge of committing
stuff for a task. Instead, it happens when an action is sent in. Tasks
are expected to ensure that this is OK by making their behavior
idempotent (since they may be retried after partially completing).

One new kind of request afforded to tasks is the ability to release
and acquire arbitrary locks. This will be useful for long-running tasks.
Short-lived tasks can still be given a lock automatically upon startup
by providing a "fixed interval".

New tables when using db storage:
- tasklocks, to support persistence of arbitrary locks.
- tasklogs, to log actions taken by tasks
 2013-02-14 12:46:40 -08:00
Eric Tschetter
c5b78e75cf 1) Move up to Jackson 2.1 
2) Some bugs were fixed, but they ended up getting mixed in with the Jackson upgrade :(
 2013-02-13 18:51:32 -08:00
Fangjin Yang
82f02a4fe4 Merge branch 'master' into indexing_refactor 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
	merger/src/main/java/com/metamx/druid/merger/common/task/V8toV9UpgradeTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/RemoteTaskRunner.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
 2013-02-13 16:34:14 -08:00
Fangjin Yang
1963bfe83f cleanup 2013-02-13 16:04:43 -08:00
Fangjin Yang
c7b4973700 bug fixes for RTR 2013-02-13 14:19:11 -08:00
Eric Tschetter
a0b159fed7 1) Move up to Jackson 2.1 
2) Some bugs were fixed, but they ended up getting mixed in with the Jackson upgrade :(
 2013-02-12 20:58:17 -06:00
xvrl
07b0b4611b bump up to 0.3.0 2013-02-12 09:52:21 -08:00
Fangjin Yang
b27a459f71 more refactoring and UT passing at this point 2013-02-11 18:13:01 -08:00
Fangjin Yang
a9176e738a Merge branch 'master' into indexing_refactor 2013-02-11 12:36:49 -08:00
Fangjin Yang
e519f82e87 [maven-release-plugin] prepare for next development iteration 2013-02-11 10:58:51 -08:00
Fangjin Yang
73f7468a23 [maven-release-plugin] prepare release druid-0.2.7 2013-02-11 10:58:43 -08:00
Fangjin Yang
111df9a90e Merge branch 'master' into indexing_refactor 
Conflicts:
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
	server/src/main/java/com/metamx/druid/master/DruidMaster.java
 2013-02-07 10:35:32 -08:00
Fangjin Yang
ec2b906fad [maven-release-plugin] prepare for next development iteration 2013-02-06 18:43:42 -08:00
Fangjin Yang
22d9607169 [maven-release-plugin] prepare release druid-0.2.6 2013-02-06 18:43:34 -08:00