Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,172 Commits 111 Branches 503 Tags 728 MiB
Commit Graph

250 Commits

Author SHA1 Message Date
Gian Merlino 2c53af4d66 ForkingTaskRunner: Upload task logs even when job fails 2014-01-08 14:46:18 -08:00
Gian Merlino 7f430d9fde RealtimeIndexTask: If a Throwable was thrown it is not a normalExit 2014-01-08 14:45:35 -08:00
Gian Merlino 83b4641e31 ForkingTaskRunnerConfig: Add java.io.tmpdir to allowedPrefixes 2014-01-07 16:12:24 -08:00
Gian Merlino bf158102c4 IndexTask: Print metrics even if finishJob fails 2014-01-07 07:17:19 -08:00
Gian Merlino 26991b5a2a Indexing service: Fix termination related log message 2013-12-20 12:05:42 -08:00
Gian Merlino 4d83837e88 RealtimeIndexTask: Clean up imports and comments 2013-12-20 11:37:16 -08:00
Gian Merlino 17ad4ee2f0 Fix RemoteTaskRunnerTest 2013-12-20 11:23:28 -08:00
Gian Merlino e5b8546d19 Autoscaling fixes. 
- Initial targetWorkerCount must be subject to pool size limits
- Use consistent workerSetupData for the entire autoscaling run
- Don't call terminate() when we have nothing to terminate
- Terminate obsolete workers even faster
 2013-12-20 11:17:01 -08:00
fjy 3ec2766cd3 Merge pull request #339 from metamx/autoscaling 
Autoscaling: Move target count independent of actual count.
 2013-12-20 10:04:26 -08:00
Gian Merlino 6224577ed1 Autoscaling: Terminate obsolete workers faster 2013-12-20 10:01:32 -08:00
Gian Merlino 4a722c0a6d Autoscaling changes from code review. 
- Log and return immediately when workerSetupData is null
- Allow provisioning more nodes while other nodes are still provisioning
- Add tests for bumping up the minimum version
 2013-12-20 08:59:35 -08:00
Gian Merlino 0ee6136ea3 NoopTask: Fix things that should be static. Add simple factory method. 2013-12-20 08:56:17 -08:00
Gian Merlino 3dd9a25546 Fix import 2013-12-19 16:18:16 -08:00
Gian Merlino 0ff7f0e8e0 TaskActionToolbox: Combine adjacent ifs 2013-12-19 16:16:34 -08:00
Gian Merlino f86342f7dc DbTaskStorage: Protect against invalid lock_payload 2013-12-19 16:16:20 -08:00
Gian Merlino 1f4b99634f Autoscaling: Move target count independent of actual count. 
This should let us grow and shrink the worker pool in chunks when necessary
(like when a bunch of them go offline, or when there is a worker version
change).
 2013-12-19 16:11:30 -08:00
Gian Merlino 846c3da4ab Empty task intervals, and empty lock intervals, aren't useful. 
So prevent them from being created, through checks in AbstractFixedIntervalTask
and TaskLockbox.tryLock.
 2013-12-19 13:21:41 -08:00
Gian Merlino 566a3a6112 Indexing service: Break up segment actions 
Each one now one operates on at most a collection of segments that comprise
a single partition. The main purpose of this change is to prevent audit log
payload sizes from getting out of control.
 2013-12-19 13:10:40 -08:00
Gian Merlino 6fbe67eeea IndexerDBCoordinator: Work around SELECT -> INSERT races when adding segments 2013-12-19 13:10:40 -08:00
Gian Merlino 1ff855d744 Fix MoveTask serde and ArchiveTask id creation 2013-12-18 15:17:12 -08:00
Gian Merlino 58d1262edf Indexing console: Clarify "Complete" with "recently completed" 2013-12-17 08:16:49 -08:00
Xavier Léauté e333776aca rename SegmentMoveAction to SegmentMetadataUpdateAction 2013-12-16 14:00:56 -08:00
Xavier Léauté ac2ca0e46c separate move and archive tasks 2013-12-16 14:00:55 -08:00
Xavier Léauté 123bddd615 update for new interfaces 2013-12-16 13:59:16 -08:00
Xavier Léauté 4a291fdf30 better naming 2013-12-16 13:59:16 -08:00
Xavier Léauté a417cd5df2 add archive task 2013-12-16 13:59:15 -08:00
fjy 87b83bceb1 fix task storage config serde and prepare for next release 2013-12-13 16:55:22 -08:00
fjy 01f9c1df31 fix broken task storage config and prepare for next release 2013-12-13 16:45:32 -08:00
Gian Merlino 600dc7546f Configurability of recency threshold 2013-12-13 16:02:54 -08:00
fjy 4a8140be81 better messaging to console again 2013-12-13 15:04:25 -08:00
fjy 52cdb20f10 add better messaging and error handling 2013-12-13 15:01:07 -08:00
Gian Merlino e63c69dd57 TaskStorage: Return recently complete tasks in reverse chronological order 2013-12-13 12:27:45 -08:00
Gian Merlino 6c993d87bf Indexing service API and GUI improvements! 
- New APIs: waitingTasks, completeTasks, task payload
- GUI for the above, and for task logs + status
 2013-12-13 11:38:18 -08:00
Gian Merlino f36a5b677c TaskLifecycleTest: Add test for noop task 2013-12-13 07:48:28 -08:00
Gian Merlino 3b053a66ff TaskLifecycleTest: Add test for never-ready task 2013-12-13 07:48:27 -08:00
Gian Merlino 863012c384 TaskQueue: Exception during isReady does not warrant an alert. 2013-12-13 07:48:27 -08:00
Gian Merlino 6227963af9 TaskQueue: Copy task list before management loop. 2013-12-13 07:48:27 -08:00
Gian Merlino 70c153592f CliPeon: Fix local mode 2013-12-12 14:22:57 -08:00
Gian Merlino 370e2f855a TaskSerdeTest: Fix IndexTask test by including an actual firehoseFactory 2013-12-12 13:58:44 -08:00
Gian Merlino 169f149cf9 TaskLifecycleTest: Fix broken setUp and broken assumptions. 2013-12-12 13:51:13 -08:00
Gian Merlino ba757b1e5a IndexTask: Actually make and publish segments for the correct intervals. 2013-12-12 13:50:53 -08:00
Gian Merlino be25d51a2c RemoteTaskRunner: Fix issues leading to failing tests 2013-12-12 13:49:49 -08:00
Gian Merlino c60158a21a RemoteTaskRunner: Remove task from pendingTaskPayloads on shutdown if needed 2013-12-12 10:59:16 -08:00
Gian Merlino 0129ea99cf RemoteTaskRunner changes to make bootstrapping actually work. 
- Workers are not added to zkWorkers until caches have been initialized.
- Worker status we haven't heard about will be added to runningTasks or
  completeTasks as appropriate. 
- TaskRunnerWorkItem now only needs a taskId, not the entire Task. This makes
  it possible to create them from TaskStatus objects, if that's all we have.
- Also remove some dead code.
 2013-12-12 10:44:46 -08:00
Gian Merlino d92b88718c OverlordResource: Fix comment 2013-12-12 08:46:24 -08:00
Gian Merlino b6a52610bc IndexTask: Call plumber.startJob() 2013-12-12 08:46:10 -08:00
Gian Merlino db9b515e71 IndexTask: Remove unnecessary args to determinePartitions. 2013-12-12 08:46:00 -08:00
Gian Merlino f4a09d4ee3 TaskAction: Add JsonSubType for LockTryAcquireAction 2013-12-12 08:45:23 -08:00
Gian Merlino b17dc6f744 Task interval, isReady hygiene 2013-12-11 22:42:20 -08:00
Gian Merlino 05e24bd85c RemoteTaskRunner: Fix typo 2013-12-11 22:38:04 -08:00
Gian Merlino bed263efa5 VersionConverterTask: Less goofy import for Preconditions 2013-12-11 22:37:55 -08:00
Gian Merlino 53d90efe30 TaskQueueConfig: Copyright header 2013-12-11 22:37:40 -08:00
Gian Merlino 0adda97776 AbstractFixedIntervalTask: Copyright header 2013-12-11 22:37:28 -08:00
Gian Merlino c4b8c8bc6f Rework indexing service internals to hopefully be more reliable. 
The TaskQueue directly manages the TaskRunner. The main management loop runs
periodically and checks that the runner is doing reasonable things. If not, it
attempts to adjust the runner. The management loop also runs on-demand when a
task is added to keep task assignment relatively low latency. The TaskConsumer
is no longer necessary and so it no longer exists.

Task interval locks are handled differently. Instead of some tasks acquiring
locks at runtime and some tasks having implicit fixed lock intervals, all tasks
ask for locks explicitly. This occurs either in "isReady" (which runs on the
overlord) or in "run" (which runs on the peon).

Other changes:
- The TaskQueue is attached to the leader lifecycle, instead of global
- The TaskLockbox is able to sync itself from storage and is no longer
  bootstrapped by the TaskQueue.
- RemoteTaskRunner does not clean up zk paths until asked to. This will
  prevent deletion of statuses that have not yet been committed.
- Added retries on DbTaskStorage operations.
- Removed SpawnTasksAction (no more subtasks)
- Removed obsolete EventReceiverFirehose configs
- Removed obsolete OldOverlordResource
- Removed TaskStorageQueryAdapter methods related to subtasks
 2013-12-11 15:05:16 -08:00
fjy 96f679f31c clean up for merge 2013-12-10 17:51:13 -08:00
Gian Merlino f3cfd1d781 Introduce FileTaskLogs, and move TaskLogs module from server to indexing-service 2013-12-10 17:39:43 -08:00
Gian Merlino 47c1c8cab2 TaskStorage: Rename getRunningTasks -> getActiveTasks 2013-12-10 17:39:42 -08:00
fjy 303f6ff334 fix worker config setup problems 2013-12-09 18:25:29 -08:00
nishantmonu51 2186bd6cb7 Minor fixes and documentation changes 2013-12-09 19:07:48 +05:30
fjy 5d7173ac98 fix autoscaling termination duration bug 2013-11-15 19:21:25 -08:00
fjy 346cf0e04c fix out of order in urls for hadoop classpath 2013-11-14 18:18:39 -08:00
fjy 6b41681424 fix ordering of urls, ugh, need to write more tests 2013-11-14 15:06:07 -08:00
fjy 23ebca6d32 more hadoop dependency hell, getting the right urls to hadoop is hard 2013-11-14 14:29:42 -08:00
fjy c51eed060f actually use a class loader pull out dependencies from extension modules 2013-11-14 11:52:21 -08:00
fjy fcf0c6ce06 change the classpath ordering for batch processing and prepare for next release 2013-11-14 10:52:51 -08:00
fjy 64b93bf448 fix broken autoscaling and prepare for next release 2013-11-08 11:29:42 -08:00
fjy a049b42674 fix an issue with task tables not getting created automatically and prepare for next release 2013-11-07 18:01:35 -08:00
fjy 7f85a126ac fix broken event receiving firehose 2013-11-07 16:59:01 -08:00
fjy 621133d6f2 removing dead code 2013-11-07 16:53:34 -08:00
fjy 084c90aa19 cleanup to prepare for next release 2013-11-07 15:55:51 -08:00
fjy bad1a7e9f8 fix according to code review 2013-11-07 15:52:34 -08:00
fjy aeb411a3a3 fix according to code review and fix broken examples 2013-11-07 15:42:48 -08:00
fjy e2e10fae1f clean up code 2013-11-07 15:03:14 -08:00
fjy 913ff3a082 clean up code 2013-11-07 15:00:11 -08:00
fjy 6b573c76f1 more fixes 2013-11-07 14:40:45 -08:00
Gian Merlino 8660db93fc RemoteTaskRunner: Run taskComplete after a task times out 2013-11-04 10:56:26 -08:00
Gian Merlino 186bbd1cb6 WorkerTaskMonitor: Add log message 2013-11-04 10:25:16 -08:00
Gian Merlino 781673a8f8 ForkingTaskRunner: Fix pass-down of nodeType 2013-10-23 14:13:05 -07:00
Gian Merlino b68b3526e8 IndexingServiceFirehoseModule: Add header 2013-10-18 13:27:40 -07:00
Gian Merlino 2e9c46867f Fixes for indexing service. 
- Create IndexingServiceFirehoseModule so firehoses can be loaded by all mains
- Fix implicit lock acquisition in AbstractTask
 2013-10-18 11:14:33 -07:00
fjy 4862852b43 more docs about how to use different versions of hadoop in druid 2013-10-16 17:54:49 -07:00
fjy 6192602893 fix extensions config not getting picked up in hadoop index task 2013-10-16 16:52:23 -07:00
fjy a1c09df17f make the hadoop index task work again 2013-10-16 09:45:17 -07:00
fjy 9796a40b92 port docs over to 0.6 and a bunch of misc fixes 2013-10-11 18:38:53 -07:00
fjy 4e509d1d09 Merge branch 'master' into is-docs 2013-10-09 14:05:10 -07:00
cheddar c47fe202c7 Fix HadoopDruidIndexer to work with the new way of things 
There are multiple and sundry changes in here.

First, "HadoopDruidIndexer" has been split into two pieces, (1) CliHadoop which pulls the hadoop version and builds up the right classpath with the proper hadoop version to run the indexer and (2) CliInternalHadoopIndexer which actually runs the indexer.

In order to work around a bunch of jets3t version conflicts with Hadoop and Druid, I needed to extract the S3 deep storage stuff into its own module.  I then also moved the HDFS stuff into its own module so that I could eliminate the dependency on Hadoop for druid-server.

In doing these changes, I wanted to make the extensions buildable with only the druid-api jar, so a few other things had to move out of Druid and into druid-api.  They are all API-level things, however, so they really belong in druid-api instead.

Lastly, I removed the druid-realtime module and put it all in druid-server.
 2013-10-09 15:15:44 -05:00
fjy 4ec4b8e024 rewrite indexing service docs 2013-10-08 16:34:58 -07:00
fjy 703b674800 add availability zone info to autoscaling 2013-10-07 12:16:50 -07:00
fjy ac330f72bb first set of changes to standarize the naming convention we use in druid 2013-10-03 16:36:48 -07:00
fjy 17874eeb67 make the CliPeon actually able to run on its own 2013-10-02 15:55:10 -07:00
fjy bc8db7daa5 1) make chat handler resource work again 
2) add more default configs
3) make examples work again
 2013-10-02 14:22:39 -07:00
Gian Merlino 384dcda7e4 Chat handlers still don't work, but, they're closer maybe. 2013-10-01 17:45:53 -07:00
fjy 59f2d0711d Merge branch 'master' into local-index 2013-10-01 13:21:43 -07:00
fjy 30df53671e remove line iterator factory because it is not needed 2013-10-01 13:21:20 -07:00
Gian Merlino 62eda5020c ShardSpec: Remove isInChunk(Map<String, String>) 2013-10-01 12:50:08 -07:00
fjy 5d0d71250b fix chat handler resources not correctly registering themselves 2013-10-01 11:25:39 -07:00
fjy 53698a135a add interface to new firehose as per code review comments 2013-09-30 18:00:59 -07:00
fjy f55a5199b1 add a firehose module to remove so much copy and pasted code 2013-09-30 16:29:20 -07:00
fjy ed9e0cf9f6 add a local firehose for indexing local files 2013-09-30 16:03:26 -07:00
Gian Merlino dc5dab8747 Fixes for property conversion, firehose registration, and the indexing service 2013-09-27 17:09:59 -07:00