Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,719 Commits 113 Branches 506 Tags 732 MiB
Commit Graph

2719 Commits

Author SHA1 Message Date
Gian Merlino 370e2f855a TaskSerdeTest: Fix IndexTask test by including an actual firehoseFactory 2013-12-12 13:58:44 -08:00
Gian Merlino 169f149cf9 TaskLifecycleTest: Fix broken setUp and broken assumptions. 2013-12-12 13:51:13 -08:00
Gian Merlino ba757b1e5a IndexTask: Actually make and publish segments for the correct intervals. 2013-12-12 13:50:53 -08:00
Gian Merlino be25d51a2c RemoteTaskRunner: Fix issues leading to failing tests 2013-12-12 13:49:49 -08:00
Gian Merlino 81ccb37019 Indexing-Service.md: New overlord-related configs. 2013-12-12 11:46:56 -08:00
Gian Merlino 58d8f0046a Configuration.md: Update TaskLogs docs 2013-12-12 11:46:30 -08:00
Gian Merlino c60158a21a RemoteTaskRunner: Remove task from pendingTaskPayloads on shutdown if needed 2013-12-12 10:59:16 -08:00
fjy 7f8c7470e7 Merge pull request #324 from liquidm/kafka_eight_release 
kafka-eight: bump kafka dependency to 0.8 release
 2013-12-12 10:51:15 -08:00
Gian Merlino 0129ea99cf RemoteTaskRunner changes to make bootstrapping actually work. 
- Workers are not added to zkWorkers until caches have been initialized.
- Worker status we haven't heard about will be added to runningTasks or
  completeTasks as appropriate. 
- TaskRunnerWorkItem now only needs a taskId, not the entire Task. This makes
  it possible to create them from TaskStatus objects, if that's all we have.
- Also remove some dead code.
 2013-12-12 10:44:46 -08:00
Gian Merlino d92b88718c OverlordResource: Fix comment 2013-12-12 08:46:24 -08:00
Gian Merlino b6a52610bc IndexTask: Call plumber.startJob() 2013-12-12 08:46:10 -08:00
Gian Merlino db9b515e71 IndexTask: Remove unnecessary args to determinePartitions. 2013-12-12 08:46:00 -08:00
Gian Merlino f4a09d4ee3 TaskAction: Add JsonSubType for LockTryAcquireAction 2013-12-12 08:45:23 -08:00
Hagen Rother cd8d2fd5c3 kafka-eight: bump kafka dependency to 0.8 release 2013-12-12 15:24:16 +01:00
Gian Merlino b17dc6f744 Task interval, isReady hygiene 2013-12-11 22:42:20 -08:00
Gian Merlino 05e24bd85c RemoteTaskRunner: Fix typo 2013-12-11 22:38:04 -08:00
Gian Merlino bed263efa5 VersionConverterTask: Less goofy import for Preconditions 2013-12-11 22:37:55 -08:00
Gian Merlino 53d90efe30 TaskQueueConfig: Copyright header 2013-12-11 22:37:40 -08:00
Gian Merlino 0adda97776 AbstractFixedIntervalTask: Copyright header 2013-12-11 22:37:28 -08:00
Gian Merlino c4b8c8bc6f Rework indexing service internals to hopefully be more reliable. 
The TaskQueue directly manages the TaskRunner. The main management loop runs
periodically and checks that the runner is doing reasonable things. If not, it
attempts to adjust the runner. The management loop also runs on-demand when a
task is added to keep task assignment relatively low latency. The TaskConsumer
is no longer necessary and so it no longer exists.

Task interval locks are handled differently. Instead of some tasks acquiring
locks at runtime and some tasks having implicit fixed lock intervals, all tasks
ask for locks explicitly. This occurs either in "isReady" (which runs on the
overlord) or in "run" (which runs on the peon).

Other changes:
- The TaskQueue is attached to the leader lifecycle, instead of global
- The TaskLockbox is able to sync itself from storage and is no longer
  bootstrapped by the TaskQueue.
- RemoteTaskRunner does not clean up zk paths until asked to. This will
  prevent deletion of statuses that have not yet been committed.
- Added retries on DbTaskStorage operations.
- Removed SpawnTasksAction (no more subtasks)
- Removed obsolete EventReceiverFirehose configs
- Removed obsolete OldOverlordResource
- Removed TaskStorageQueryAdapter methods related to subtasks
 2013-12-11 15:05:16 -08:00
Gian Merlino b2ea216bdf HttpClientModule: Default http timeout to PT5M 2013-12-11 15:05:16 -08:00
Gian Merlino 1ec6d5b0ef S3Utils: Use RetryUtils from java-util 2013-12-11 15:05:16 -08:00
Igal Levy 476ce8677d Merge branch 'master' into igalDruid 2013-12-11 09:17:30 -08:00
Igal Levy 60577a6e24 adjusted size of first section head to match others 2013-12-11 09:16:49 -08:00
fjy 6c80de53b0 Merge pull request #322 from metamx/filetasklogs 
Filetasklogs
 2013-12-10 17:51:44 -08:00
fjy 96f679f31c clean up for merge 2013-12-10 17:51:13 -08:00
Gian Merlino f3cfd1d781 Introduce FileTaskLogs, and move TaskLogs module from server to indexing-service 2013-12-10 17:39:43 -08:00
Gian Merlino 47c1c8cab2 TaskStorage: Rename getRunningTasks -> getActiveTasks 2013-12-10 17:39:42 -08:00
fjy 64e75becba Merge pull request #321 from metamx/js-dimextraction-null-fix 
fix JS dim extraction null handling + test case
 2013-12-10 17:14:56 -08:00
fjy a7cc559623 [maven-release-plugin] prepare for next development iteration 2013-12-09 20:07:38 -08:00
fjy 54755b8864 [maven-release-plugin] prepare release druid-0.6.30 2013-12-09 20:07:34 -08:00
fjy 6cc0860657 fix s3 module problems and prepare for next release 2013-12-09 20:05:55 -08:00
fjy a262d8bf11 fix curator shutdown problem in curator inv manager test 2013-12-09 18:56:30 -08:00
fjy 7a134272b3 [maven-release-plugin] prepare for next development iteration 2013-12-09 18:45:27 -08:00
fjy 829658f171 [maven-release-plugin] prepare release druid-0.6.29 2013-12-09 18:45:23 -08:00
fjy ab1f02b2bc fix version brokenness and prepare for next release 2013-12-09 18:38:22 -08:00
Xavier Léauté bb69f8adcd fix JS dim extraction null handling + test case 2013-12-09 18:36:14 -08:00
fjy 98a0f40227 [maven-release-plugin] prepare for next development iteration 2013-12-09 18:35:22 -08:00
fjy 939d2313c0 [maven-release-plugin] prepare release druid-0.6.28 2013-12-09 18:35:18 -08:00
fjy c03d573e84 [maven-release-plugin] prepare for next development iteration 2013-12-09 18:30:15 -08:00
fjy 71fe8a43da [maven-release-plugin] prepare release druid-0.6.27 2013-12-09 18:30:11 -08:00
fjy 976affb3af prepare for next release 2013-12-09 18:28:31 -08:00
fjy 303f6ff334 fix worker config setup problems 2013-12-09 18:25:29 -08:00
fjy 8783007eb1 Merge branch 'master' of github.com:metamx/druid 2013-12-09 18:01:21 -08:00
fjy 8e0336ff83 fix formatting on paper 2013-12-09 18:01:11 -08:00
fjy dc798768a0 update pdf 2013-12-09 17:58:17 -08:00
fjy 9d4f18d12b fix figures 2013-12-09 17:57:29 -08:00
fjy 867347658f some spell check fixes for paper 2013-12-09 17:19:17 -08:00
fjy 8aba4ad3d2 Merge pull request #311 from metamx/versions 
Add versions to static page
 2013-12-09 17:18:16 -08:00
Igal Levy fcc8801b13 added title 2013-12-09 16:11:06 -08:00