Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,256 Commits 111 Branches 503 Tags 721 MiB
Commit Graph

623 Commits

Author SHA1 Message Date
fjy 2b46c1d292 [maven-release-plugin] prepare for next development iteration 2014-01-02 19:53:41 -08:00
fjy 7a0f634b02 [maven-release-plugin] prepare release druid-0.6.47 2014-01-02 19:53:35 -08:00
Xavier Léauté ef05312c97 cleanup dependencies 2013-12-30 16:31:12 -08:00
Gian Merlino 668cf009a0 [maven-release-plugin] prepare for next development iteration 2013-12-20 12:28:54 -08:00
Gian Merlino bd58def933 [maven-release-plugin] prepare release druid-0.6.46 2013-12-20 12:28:49 -08:00
Gian Merlino 26991b5a2a Indexing service: Fix termination related log message 2013-12-20 12:05:42 -08:00
Gian Merlino 4d83837e88 RealtimeIndexTask: Clean up imports and comments 2013-12-20 11:37:16 -08:00
Gian Merlino 9043c211b2 [maven-release-plugin] prepare for next development iteration 2013-12-20 11:25:36 -08:00
Gian Merlino 29c0fa1df9 [maven-release-plugin] prepare release druid-0.6.45 2013-12-20 11:25:31 -08:00
Gian Merlino 17ad4ee2f0 Fix RemoteTaskRunnerTest 2013-12-20 11:23:28 -08:00
Gian Merlino 5711ac4aff [maven-release-plugin] prepare for next development iteration 2013-12-20 11:17:01 -08:00
Gian Merlino 3b15f6a834 [maven-release-plugin] prepare release druid-0.6.44 2013-12-20 11:17:01 -08:00
Gian Merlino e5b8546d19 Autoscaling fixes. 
- Initial targetWorkerCount must be subject to pool size limits
- Use consistent workerSetupData for the entire autoscaling run
- Don't call terminate() when we have nothing to terminate
- Terminate obsolete workers even faster
 2013-12-20 11:17:01 -08:00
Gian Merlino c503f5e9c5 [maven-release-plugin] prepare for next development iteration 2013-12-20 11:17:01 -08:00
Gian Merlino 97095ee3db [maven-release-plugin] prepare release druid-0.6.43 2013-12-20 11:17:00 -08:00
fjy 3ec2766cd3 Merge pull request #339 from metamx/autoscaling 
Autoscaling: Move target count independent of actual count.
 2013-12-20 10:04:26 -08:00
Gian Merlino 6224577ed1 Autoscaling: Terminate obsolete workers faster 2013-12-20 10:01:32 -08:00
Gian Merlino 4a722c0a6d Autoscaling changes from code review. 
- Log and return immediately when workerSetupData is null
- Allow provisioning more nodes while other nodes are still provisioning
- Add tests for bumping up the minimum version
 2013-12-20 08:59:35 -08:00
Gian Merlino 0ee6136ea3 NoopTask: Fix things that should be static. Add simple factory method. 2013-12-20 08:56:17 -08:00
Gian Merlino 3dd9a25546 Fix import 2013-12-19 16:18:16 -08:00
Gian Merlino 0ff7f0e8e0 TaskActionToolbox: Combine adjacent ifs 2013-12-19 16:16:34 -08:00
Gian Merlino f86342f7dc DbTaskStorage: Protect against invalid lock_payload 2013-12-19 16:16:20 -08:00
Gian Merlino 1f4b99634f Autoscaling: Move target count independent of actual count. 
This should let us grow and shrink the worker pool in chunks when necessary
(like when a bunch of them go offline, or when there is a worker version
change).
 2013-12-19 16:11:30 -08:00
Gian Merlino 846c3da4ab Empty task intervals, and empty lock intervals, aren't useful. 
So prevent them from being created, through checks in AbstractFixedIntervalTask
and TaskLockbox.tryLock.
 2013-12-19 13:21:41 -08:00
Gian Merlino 566a3a6112 Indexing service: Break up segment actions 
Each one now one operates on at most a collection of segments that comprise
a single partition. The main purpose of this change is to prevent audit log
payload sizes from getting out of control.
 2013-12-19 13:10:40 -08:00
Gian Merlino 6fbe67eeea IndexerDBCoordinator: Work around SELECT -> INSERT races when adding segments 2013-12-19 13:10:40 -08:00
Gian Merlino 25c3f28907 [maven-release-plugin] prepare for next development iteration 2013-12-18 18:38:27 -08:00
Gian Merlino 6a740768d6 [maven-release-plugin] prepare release druid-0.6.42 2013-12-18 18:38:22 -08:00
Gian Merlino 8d8ec51c9c [maven-release-plugin] prepare for next development iteration 2013-12-18 16:05:33 -08:00
Gian Merlino 00d651bf6f [maven-release-plugin] prepare release druid-0.6.41 2013-12-18 16:05:27 -08:00
Gian Merlino d0fd58bbae [maven-release-plugin] prepare for next development iteration 2013-12-18 15:35:55 -08:00
Gian Merlino 231d7a6d1f [maven-release-plugin] prepare release druid-0.6.40 2013-12-18 15:35:50 -08:00
Gian Merlino 1ff855d744 Fix MoveTask serde and ArchiveTask id creation 2013-12-18 15:17:12 -08:00
fjy 494ec53050 [maven-release-plugin] prepare for next development iteration 2013-12-17 17:48:46 -08:00
fjy 2a98a4d3e0 [maven-release-plugin] prepare release druid-0.6.39 2013-12-17 17:48:42 -08:00
fjy 5932150d24 [maven-release-plugin] prepare for next development iteration 2013-12-17 16:29:53 -08:00
fjy ed9f4b4cf2 [maven-release-plugin] prepare release druid-0.6.38 2013-12-17 16:29:49 -08:00
Gian Merlino 58d1262edf Indexing console: Clarify "Complete" with "recently completed" 2013-12-17 08:16:49 -08:00
fjy 48d677d135 [maven-release-plugin] prepare for next development iteration 2013-12-16 16:17:45 -08:00
fjy 7e0b4d4fd2 [maven-release-plugin] prepare release druid-0.6.37 2013-12-16 16:17:41 -08:00
Xavier Léauté e333776aca rename SegmentMoveAction to SegmentMetadataUpdateAction 2013-12-16 14:00:56 -08:00
Xavier Léauté ac2ca0e46c separate move and archive tasks 2013-12-16 14:00:55 -08:00
Xavier Léauté 123bddd615 update for new interfaces 2013-12-16 13:59:16 -08:00
Xavier Léauté 4a291fdf30 better naming 2013-12-16 13:59:16 -08:00
Xavier Léauté a417cd5df2 add archive task 2013-12-16 13:59:15 -08:00
fjy 3812bf7215 [maven-release-plugin] prepare for next development iteration 2013-12-13 16:57:07 -08:00
fjy a7ff58f877 [maven-release-plugin] prepare release druid-0.6.36 2013-12-13 16:57:02 -08:00
fjy 87b83bceb1 fix task storage config serde and prepare for next release 2013-12-13 16:55:22 -08:00
fjy 41d64f4aa6 [maven-release-plugin] prepare for next development iteration 2013-12-13 16:47:21 -08:00
fjy 1c050a02f8 [maven-release-plugin] prepare release druid-0.6.35 2013-12-13 16:47:17 -08:00
fjy 01f9c1df31 fix broken task storage config and prepare for next release 2013-12-13 16:45:32 -08:00
fjy 7891e0a2e2 [maven-release-plugin] prepare for next development iteration 2013-12-13 16:20:23 -08:00
fjy 1d75738f81 [maven-release-plugin] prepare release druid-0.6.34 2013-12-13 16:20:18 -08:00
Gian Merlino 600dc7546f Configurability of recency threshold 2013-12-13 16:02:54 -08:00
fjy 4a8140be81 better messaging to console again 2013-12-13 15:04:25 -08:00
fjy 52cdb20f10 add better messaging and error handling 2013-12-13 15:01:07 -08:00
Gian Merlino e63c69dd57 TaskStorage: Return recently complete tasks in reverse chronological order 2013-12-13 12:27:45 -08:00
Gian Merlino 6c993d87bf Indexing service API and GUI improvements! 
- New APIs: waitingTasks, completeTasks, task payload
- GUI for the above, and for task logs + status
 2013-12-13 11:38:18 -08:00
Gian Merlino f36a5b677c TaskLifecycleTest: Add test for noop task 2013-12-13 07:48:28 -08:00
Gian Merlino 3b053a66ff TaskLifecycleTest: Add test for never-ready task 2013-12-13 07:48:27 -08:00
Gian Merlino 863012c384 TaskQueue: Exception during isReady does not warrant an alert. 2013-12-13 07:48:27 -08:00
Gian Merlino 6227963af9 TaskQueue: Copy task list before management loop. 2013-12-13 07:48:27 -08:00
fjy aef7726c33 [maven-release-plugin] prepare for next development iteration 2013-12-12 18:53:18 -08:00
fjy 1d1baf5976 [maven-release-plugin] prepare release druid-0.6.33 2013-12-12 18:53:14 -08:00
fjy f920dd6fb2 [maven-release-plugin] prepare for next development iteration 2013-12-12 17:28:38 -08:00
fjy 857d24a4ca [maven-release-plugin] prepare release druid-0.6.32 2013-12-12 17:28:34 -08:00
fjy 7c744ffda0 [maven-release-plugin] prepare for next development iteration 2013-12-12 14:53:01 -08:00
fjy 365da245c2 [maven-release-plugin] prepare release druid-0.6.31 2013-12-12 14:52:57 -08:00
Gian Merlino 70c153592f CliPeon: Fix local mode 2013-12-12 14:22:57 -08:00
Gian Merlino 370e2f855a TaskSerdeTest: Fix IndexTask test by including an actual firehoseFactory 2013-12-12 13:58:44 -08:00
Gian Merlino 169f149cf9 TaskLifecycleTest: Fix broken setUp and broken assumptions. 2013-12-12 13:51:13 -08:00
Gian Merlino ba757b1e5a IndexTask: Actually make and publish segments for the correct intervals. 2013-12-12 13:50:53 -08:00
Gian Merlino be25d51a2c RemoteTaskRunner: Fix issues leading to failing tests 2013-12-12 13:49:49 -08:00
Gian Merlino c60158a21a RemoteTaskRunner: Remove task from pendingTaskPayloads on shutdown if needed 2013-12-12 10:59:16 -08:00
Gian Merlino 0129ea99cf RemoteTaskRunner changes to make bootstrapping actually work. 
- Workers are not added to zkWorkers until caches have been initialized.
- Worker status we haven't heard about will be added to runningTasks or
  completeTasks as appropriate. 
- TaskRunnerWorkItem now only needs a taskId, not the entire Task. This makes
  it possible to create them from TaskStatus objects, if that's all we have.
- Also remove some dead code.
 2013-12-12 10:44:46 -08:00
Gian Merlino d92b88718c OverlordResource: Fix comment 2013-12-12 08:46:24 -08:00
Gian Merlino b6a52610bc IndexTask: Call plumber.startJob() 2013-12-12 08:46:10 -08:00
Gian Merlino db9b515e71 IndexTask: Remove unnecessary args to determinePartitions. 2013-12-12 08:46:00 -08:00
Gian Merlino f4a09d4ee3 TaskAction: Add JsonSubType for LockTryAcquireAction 2013-12-12 08:45:23 -08:00
Gian Merlino b17dc6f744 Task interval, isReady hygiene 2013-12-11 22:42:20 -08:00
Gian Merlino 05e24bd85c RemoteTaskRunner: Fix typo 2013-12-11 22:38:04 -08:00
Gian Merlino bed263efa5 VersionConverterTask: Less goofy import for Preconditions 2013-12-11 22:37:55 -08:00
Gian Merlino 53d90efe30 TaskQueueConfig: Copyright header 2013-12-11 22:37:40 -08:00
Gian Merlino 0adda97776 AbstractFixedIntervalTask: Copyright header 2013-12-11 22:37:28 -08:00
Gian Merlino c4b8c8bc6f Rework indexing service internals to hopefully be more reliable. 
The TaskQueue directly manages the TaskRunner. The main management loop runs
periodically and checks that the runner is doing reasonable things. If not, it
attempts to adjust the runner. The management loop also runs on-demand when a
task is added to keep task assignment relatively low latency. The TaskConsumer
is no longer necessary and so it no longer exists.

Task interval locks are handled differently. Instead of some tasks acquiring
locks at runtime and some tasks having implicit fixed lock intervals, all tasks
ask for locks explicitly. This occurs either in "isReady" (which runs on the
overlord) or in "run" (which runs on the peon).

Other changes:
- The TaskQueue is attached to the leader lifecycle, instead of global
- The TaskLockbox is able to sync itself from storage and is no longer
  bootstrapped by the TaskQueue.
- RemoteTaskRunner does not clean up zk paths until asked to. This will
  prevent deletion of statuses that have not yet been committed.
- Added retries on DbTaskStorage operations.
- Removed SpawnTasksAction (no more subtasks)
- Removed obsolete EventReceiverFirehose configs
- Removed obsolete OldOverlordResource
- Removed TaskStorageQueryAdapter methods related to subtasks
 2013-12-11 15:05:16 -08:00
fjy 96f679f31c clean up for merge 2013-12-10 17:51:13 -08:00
Gian Merlino f3cfd1d781 Introduce FileTaskLogs, and move TaskLogs module from server to indexing-service 2013-12-10 17:39:43 -08:00
Gian Merlino 47c1c8cab2 TaskStorage: Rename getRunningTasks -> getActiveTasks 2013-12-10 17:39:42 -08:00
fjy a7cc559623 [maven-release-plugin] prepare for next development iteration 2013-12-09 20:07:38 -08:00
fjy 54755b8864 [maven-release-plugin] prepare release druid-0.6.30 2013-12-09 20:07:34 -08:00
fjy 7a134272b3 [maven-release-plugin] prepare for next development iteration 2013-12-09 18:45:27 -08:00
fjy 829658f171 [maven-release-plugin] prepare release druid-0.6.29 2013-12-09 18:45:23 -08:00
fjy 98a0f40227 [maven-release-plugin] prepare for next development iteration 2013-12-09 18:35:22 -08:00
fjy 939d2313c0 [maven-release-plugin] prepare release druid-0.6.28 2013-12-09 18:35:18 -08:00
fjy c03d573e84 [maven-release-plugin] prepare for next development iteration 2013-12-09 18:30:15 -08:00
fjy 71fe8a43da [maven-release-plugin] prepare release druid-0.6.27 2013-12-09 18:30:11 -08:00
fjy 303f6ff334 fix worker config setup problems 2013-12-09 18:25:29 -08:00
fjy 8aba4ad3d2 Merge pull request #311 from metamx/versions 
Add versions to static page
 2013-12-09 17:18:16 -08:00
nishantmonu51 2186bd6cb7 Minor fixes and documentation changes 2013-12-09 19:07:48 +05:30
fjy 7c7cfdf819 [maven-release-plugin] prepare for next development iteration 2013-12-04 18:57:30 -08:00
fjy 6bfb3b8d41 [maven-release-plugin] prepare release druid-0.6.26 2013-12-04 18:57:25 -08:00
Himadri Singh 47278ad109 Merge relevant changes from 'druid-version-info' 2013-12-04 10:21:19 +05:30
fjy 1a3a1221c2 [maven-release-plugin] prepare for next development iteration 2013-12-02 14:22:54 -08:00
fjy b18b65108a [maven-release-plugin] prepare release druid-0.6.25 2013-12-02 14:22:50 -08:00
fjy 730e3a2303 [maven-release-plugin] prepare for next development iteration 2013-11-25 17:55:01 -08:00
fjy c7171e3311 [maven-release-plugin] prepare release druid-0.6.24 2013-11-25 17:54:57 -08:00
fjy 5c6e647148 [maven-release-plugin] prepare for next development iteration 2013-11-20 12:00:15 -08:00
fjy 5b94895a36 [maven-release-plugin] prepare release druid-0.6.23 2013-11-20 12:00:11 -08:00
fjy ea95566cd9 [maven-release-plugin] prepare for next development iteration 2013-11-20 10:45:21 -08:00
fjy fe7122b717 [maven-release-plugin] prepare release druid-0.6.22 2013-11-20 10:45:17 -08:00
fjy 88378eb036 [maven-release-plugin] prepare for next development iteration 2013-11-20 10:28:57 -08:00
fjy 2e6056f537 [maven-release-plugin] prepare release druid-0.6.21 2013-11-20 10:28:52 -08:00
fjy 7fec5e6ff7 [maven-release-plugin] prepare for next development iteration 2013-11-19 09:59:48 -08:00
fjy 265ef23cd5 [maven-release-plugin] prepare release druid-0.6.20 2013-11-19 09:59:44 -08:00
fjy 5d7173ac98 fix autoscaling termination duration bug 2013-11-15 19:21:25 -08:00
fjy 48d275e5cb [maven-release-plugin] prepare for next development iteration 2013-11-14 18:20:29 -08:00
fjy c31c5ce1ac [maven-release-plugin] prepare release druid-0.6.19 2013-11-14 18:20:25 -08:00
fjy 346cf0e04c fix out of order in urls for hadoop classpath 2013-11-14 18:18:39 -08:00
fjy d935e1b47b [maven-release-plugin] prepare for next development iteration 2013-11-14 15:08:12 -08:00
fjy 32dbbe4f63 [maven-release-plugin] prepare release druid-0.6.18 2013-11-14 15:08:09 -08:00
fjy 6b41681424 fix ordering of urls, ugh, need to write more tests 2013-11-14 15:06:07 -08:00
fjy 373251a476 [maven-release-plugin] prepare for next development iteration 2013-11-14 14:31:26 -08:00
fjy 7120db20a4 [maven-release-plugin] prepare release druid-0.6.17 2013-11-14 14:31:22 -08:00
fjy 23ebca6d32 more hadoop dependency hell, getting the right urls to hadoop is hard 2013-11-14 14:29:42 -08:00
fjy 89e6ce3550 [maven-release-plugin] prepare for next development iteration 2013-11-14 13:18:24 -08:00
fjy 97293b3f04 [maven-release-plugin] prepare release druid-0.6.16 2013-11-14 13:18:20 -08:00
fjy c51eed060f actually use a class loader pull out dependencies from extension modules 2013-11-14 11:52:21 -08:00
fjy e7118cd9be [maven-release-plugin] prepare for next development iteration 2013-11-14 10:54:37 -08:00
fjy d3f46c826a [maven-release-plugin] prepare release druid-0.6.15 2013-11-14 10:54:32 -08:00
fjy fcf0c6ce06 change the classpath ordering for batch processing and prepare for next release 2013-11-14 10:52:51 -08:00
fjy 6e7e2428e8 [maven-release-plugin] prepare for next development iteration 2013-11-13 13:27:47 -08:00
fjy 9c4cecc015 [maven-release-plugin] prepare release druid-0.6.14 2013-11-13 13:27:43 -08:00
fjy 6c41cd8682 [maven-release-plugin] prepare for next development iteration 2013-11-13 11:38:44 -08:00
fjy b93550194a [maven-release-plugin] prepare release druid-0.6.13 2013-11-13 11:38:39 -08:00
fjy 9bbaade8f9 [maven-release-plugin] prepare for next development iteration 2013-11-13 10:37:12 -08:00
fjy 4dffef698e [maven-release-plugin] prepare release druid-0.6.12 2013-11-13 10:37:08 -08:00
fjy 12a188d8d6 [maven-release-plugin] prepare for next development iteration 2013-11-12 15:20:54 -08:00
fjy 63d9f7362a [maven-release-plugin] prepare release druid-0.6.11 2013-11-12 15:20:47 -08:00
fjy 0681fb614d [maven-release-plugin] prepare for next development iteration 2013-11-08 15:56:10 -08:00
fjy de5984ccea [maven-release-plugin] prepare release druid-0.6.10 2013-11-08 15:56:06 -08:00
fjy d212b3fc43 [maven-release-plugin] prepare for next development iteration 2013-11-08 15:14:46 -08:00
fjy 4cd4578539 [maven-release-plugin] prepare release druid-0.6.9 2013-11-08 15:14:41 -08:00
fjy fd4fb78e1f [maven-release-plugin] prepare for next development iteration 2013-11-08 11:34:12 -08:00
fjy f8d5d127a4 [maven-release-plugin] prepare release druid-0.6.8 2013-11-08 11:34:09 -08:00
fjy 64b93bf448 fix broken autoscaling and prepare for next release 2013-11-08 11:29:42 -08:00
fjy 90240e6f1e [maven-release-plugin] prepare for next development iteration 2013-11-07 23:29:02 -08:00
fjy c3d6233bc4 [maven-release-plugin] prepare release druid-0.6.7 2013-11-07 23:28:58 -08:00
fjy d70bcc5657 repair broken pom 2013-11-07 23:27:06 -08:00
fjy fafd9c8d27 [maven-release-plugin] prepare for next development iteration 2013-11-07 18:20:19 -08:00
fjy f945416538 [maven-release-plugin] prepare release druid-0.6.5 2013-11-07 18:20:16 -08:00
fjy ed328aca6e [maven-release-plugin] prepare for next development iteration 2013-11-07 18:03:39 -08:00
fjy e34f1a829b [maven-release-plugin] prepare release druid-0.6.4 2013-11-07 18:03:36 -08:00
fjy a049b42674 fix an issue with task tables not getting created automatically and prepare for next release 2013-11-07 18:01:35 -08:00
fjy 009646ed56 [maven-release-plugin] prepare for next development iteration 2013-11-07 17:40:12 -08:00
fjy 4cde47c9d1 [maven-release-plugin] prepare release druid-0.6.3 2013-11-07 17:40:06 -08:00
fjy e6a83c0339 [maven-release-plugin] prepare for next development iteration 2013-11-07 17:05:52 -08:00
fjy 2ac09b798d [maven-release-plugin] prepare release druid-0.6.2 2013-11-07 17:05:48 -08:00
fjy 7f85a126ac fix broken event receiving firehose 2013-11-07 16:59:01 -08:00
fjy 621133d6f2 removing dead code 2013-11-07 16:53:34 -08:00
fjy 084c90aa19 cleanup to prepare for next release 2013-11-07 15:55:51 -08:00
fjy bad1a7e9f8 fix according to code review 2013-11-07 15:52:34 -08:00
fjy aeb411a3a3 fix according to code review and fix broken examples 2013-11-07 15:42:48 -08:00
fjy e2e10fae1f clean up code 2013-11-07 15:03:14 -08:00
fjy 913ff3a082 clean up code 2013-11-07 15:00:11 -08:00
fjy 6b573c76f1 more fixes 2013-11-07 14:40:45 -08:00
fjy ee15fd9f85 [maven-release-plugin] prepare for next development iteration 2013-11-05 15:42:37 -08:00
fjy 713015199a [maven-release-plugin] prepare release druid-0.6.1 2013-11-05 15:42:33 -08:00
Gian Merlino 8660db93fc RemoteTaskRunner: Run taskComplete after a task times out 2013-11-04 10:56:26 -08:00
Gian Merlino 186bbd1cb6 WorkerTaskMonitor: Add log message 2013-11-04 10:25:16 -08:00
Gian Merlino 781673a8f8 ForkingTaskRunner: Fix pass-down of nodeType 2013-10-23 14:13:05 -07:00
fjy a684885839 [maven-release-plugin] prepare for next development iteration 2013-10-18 15:48:16 -07:00
fjy 88dcfe9a94 [maven-release-plugin] prepare release druid-0.6.0 2013-10-18 15:48:12 -07:00
Gian Merlino b68b3526e8 IndexingServiceFirehoseModule: Add header 2013-10-18 13:27:40 -07:00
Gian Merlino 2e9c46867f Fixes for indexing service. 
- Create IndexingServiceFirehoseModule so firehoses can be loaded by all mains
- Fix implicit lock acquisition in AbstractTask
 2013-10-18 11:14:33 -07:00
fjy 4862852b43 more docs about how to use different versions of hadoop in druid 2013-10-16 17:54:49 -07:00
fjy 6192602893 fix extensions config not getting picked up in hadoop index task 2013-10-16 16:52:23 -07:00
fjy a1c09df17f make the hadoop index task work again 2013-10-16 09:45:17 -07:00
fjy 9796a40b92 port docs over to 0.6 and a bunch of misc fixes 2013-10-11 18:38:53 -07:00
fjy a9a723bd11 clean up poms, add a new loading your own data tutorial, add new validation, clean up logs 2013-10-09 15:42:39 -07:00
fjy 4e509d1d09 Merge branch 'master' into is-docs 2013-10-09 14:05:10 -07:00
cheddar c47fe202c7 Fix HadoopDruidIndexer to work with the new way of things 
There are multiple and sundry changes in here.

First, "HadoopDruidIndexer" has been split into two pieces, (1) CliHadoop which pulls the hadoop version and builds up the right classpath with the proper hadoop version to run the indexer and (2) CliInternalHadoopIndexer which actually runs the indexer.

In order to work around a bunch of jets3t version conflicts with Hadoop and Druid, I needed to extract the S3 deep storage stuff into its own module.  I then also moved the HDFS stuff into its own module so that I could eliminate the dependency on Hadoop for druid-server.

In doing these changes, I wanted to make the extensions buildable with only the druid-api jar, so a few other things had to move out of Druid and into druid-api.  They are all API-level things, however, so they really belong in druid-api instead.

Lastly, I removed the druid-realtime module and put it all in druid-server.
 2013-10-09 15:15:44 -05:00
fjy 4ec4b8e024 rewrite indexing service docs 2013-10-08 16:34:58 -07:00
fjy 703b674800 add availability zone info to autoscaling 2013-10-07 12:16:50 -07:00
fjy ac330f72bb first set of changes to standarize the naming convention we use in druid 2013-10-03 16:36:48 -07:00
fjy 17874eeb67 make the CliPeon actually able to run on its own 2013-10-02 15:55:10 -07:00
fjy bc8db7daa5 1) make chat handler resource work again 
2) add more default configs
3) make examples work again
 2013-10-02 14:22:39 -07:00
Gian Merlino 384dcda7e4 Chat handlers still don't work, but, they're closer maybe. 2013-10-01 17:45:53 -07:00
fjy 59f2d0711d Merge branch 'master' into local-index 2013-10-01 13:21:43 -07:00
fjy 30df53671e remove line iterator factory because it is not needed 2013-10-01 13:21:20 -07:00
Gian Merlino 62eda5020c ShardSpec: Remove isInChunk(Map<String, String>) 2013-10-01 12:50:08 -07:00
fjy 5d0d71250b fix chat handler resources not correctly registering themselves 2013-10-01 11:25:39 -07:00
fjy 53698a135a add interface to new firehose as per code review comments 2013-09-30 18:00:59 -07:00
fjy f55a5199b1 add a firehose module to remove so much copy and pasted code 2013-09-30 16:29:20 -07:00
fjy ed9e0cf9f6 add a local firehose for indexing local files 2013-09-30 16:03:26 -07:00
Gian Merlino dc5dab8747 Fixes for property conversion, firehose registration, and the indexing service 2013-09-27 17:09:59 -07:00
fjy 0b04325ee8 fix things up according to code review comments 2013-09-27 10:17:45 -07:00
fjy e404295c1f make indexing service work 2013-09-26 17:44:21 -07:00
fjy 8bc56daa66 fix things up according to code review comments 2013-09-26 11:35:45 -07:00
fjy 87259321b6 port hadoop druid indexer to new guice framework 2013-09-26 11:04:42 -07:00
fjy 15843c3978 refactor how server service discovery is done 2013-09-24 10:36:26 -07:00
fjy dc8a119787 fix broken unit tests are a result of the last merge 2013-09-23 12:56:01 -07:00
cheddar 5712b29c8c Fix issues with bindings and handling extensions 
The way the Guice bindings were setup previously, each process only had bindings
for the things it cared about.  This became problematic when adding extension modules
that bound everything that they could possibly need expecting that the processes would
only instantiate what they actually do need.  Guice tries to fail-fast and verifies that all
 bindings exist before it does anything, which is a problem because the extension bind
 some objects that don't necessarily have all of their dependencies bound in all processes.

The fix for this is to build a single Injector with all bindings in it and let each of the
 processes only load the things that they care about.  This also requires the use of
 Module overrides and other such interesting things, which are node done.

 In doing the fix, I also swapped out the way that the DataSegmentPusher/Puller stuff is bound, as well as made the Cassandra stuff fail if its settings are not provided.  This all of a sudden made all of the things require Cassandra's settings, so I migrated the Cassandra deep storage stuff into its own module.

 In doing these changes, I also discovered that some properties weren't properly converting for the ConvertProperties command (specifically, the properties related to data segment loading and pushing), so I fixed that.
 2013-09-20 17:45:01 -05:00
cheddar dc728b5721 Update versions of poms and make IncrementalIndexStorageAdapterTest compile 2013-09-18 17:01:03 -05:00
cheddar 6bfe2310bb Merge branch 'master' into guice 
Conflicts:
	client/pom.xml
	client/src/main/java/com/metamx/druid/query/segment/LegacySegmentSpec.java
	indexing-common/pom.xml
	indexing-service/src/main/java/com/metamx/druid/indexing/common/TaskToolbox.java
	indexing-service/src/main/java/com/metamx/druid/indexing/common/TaskToolboxFactory.java
	indexing-service/src/main/java/com/metamx/druid/indexing/common/task/RealtimeIndexTask.java
	indexing-service/src/main/java/com/metamx/druid/indexing/worker/executor/ExecutorNode.java
	indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/RemoteTaskRunnerTest.java
	indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/TaskLifecycleTest.java
	indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/TaskQueueTest.java
	realtime/src/main/java/com/metamx/druid/realtime/RealtimeNode.java
	realtime/src/main/java/com/metamx/druid/realtime/firehose/IrcFirehoseFactory.java
	realtime/src/main/java/com/metamx/druid/realtime/firehose/WikipediaIrcDecoder.java
	realtime/src/main/java/com/metamx/druid/realtime/plumber/RealtimePlumberSchool.java
	server/src/main/java/com/metamx/druid/BaseServerNode.java
	server/src/main/java/com/metamx/druid/http/ComputeNode.java
	server/src/main/java/com/metamx/druid/index/v1/IncrementalIndexStorageAdapter.java
	server/src/main/java/com/metamx/druid/query/group/GroupByQueryEngine.java
 2013-09-18 16:46:54 -05:00
fjy cabae7993d port over multi threaded realtime and also fix broken realtime nodes that can't start up 2013-09-16 16:03:47 -07:00
Gian Merlino 163fa14e62 [maven-release-plugin] prepare for next development iteration 2013-09-13 19:02:46 -07:00
Gian Merlino a20f3fc6dd [maven-release-plugin] prepare release druid-0.5.58 2013-09-13 19:02:40 -07:00
fjy d0b0adbb68 [maven-release-plugin] prepare for next development iteration 2013-09-13 18:57:25 -07:00
fjy 23c5fdb107 [maven-release-plugin] prepare release druid-0.5.57 2013-09-13 18:57:20 -07:00
fjy b0a09ba1f1 [maven-release-plugin] prepare for next development iteration 2013-09-13 18:45:20 -07:00
fjy b17f5c403b [maven-release-plugin] prepare release druid-0.5.56 2013-09-13 18:45:14 -07:00
Gian Merlino 22a99c7589 [maven-release-plugin] prepare for next development iteration 2013-09-13 17:59:03 -07:00
Gian Merlino 542d564453 [maven-release-plugin] prepare release druid-0.5.55 2013-09-13 17:58:56 -07:00
Gian Merlino 24df6c482d Use shared query executor in indexing-service 2013-09-13 14:33:13 -07:00
fjy f7c10e3594 rework tests in indexing service to be more unit testy 2013-09-12 16:37:58 -07:00
cheddar a2dcc45a8e 1) Remove SingleSegmentLoader and replace with OmniSegmentLoader 2013-09-12 11:47:03 -05:00
cheddar 6c9a107356 1) remove duplicate package initialization.initialization 2013-09-09 17:02:57 -05:00
fjy e03c623467 [maven-release-plugin] prepare for next development iteration 2013-09-05 13:48:21 -07:00
fjy 552d0c061a [maven-release-plugin] prepare release druid-0.5.54 2013-09-05 13:48:11 -07:00
fjy 5a64e2a64a [maven-release-plugin] prepare for next development iteration 2013-09-04 14:24:04 -07:00
fjy 523355d00b [maven-release-plugin] prepare release druid-0.5.53 2013-09-04 14:23:56 -07:00
fjy 567168330b [maven-release-plugin] prepare for next development iteration 2013-09-04 14:06:33 -07:00
fjy 39bda32b81 [maven-release-plugin] prepare release druid-0.5.52 2013-09-04 14:06:25 -07:00
fjy f08b8287a4 [maven-release-plugin] prepare for next development iteration 2013-09-04 13:22:19 -07:00
fjy cf9038abb4 [maven-release-plugin] prepare release druid-0.5.51 2013-09-04 13:22:11 -07:00
cheddar 3c39f90c89 1) Move Firehose interface and dependencies to druid-api 
2) Move DataSegment* interfaces and dependencies to druid-api
 2013-08-31 16:43:28 -05:00
cheddar 5ab671050e No more com.metamx.druid, it is now all io.druid! 2013-08-30 19:42:12 -05:00
cheddar bd0756e360 More stuff moved, things still compiling and tests still passing. Yay! 2013-08-30 18:58:35 -05:00
cheddar 56e2b956d0 OMG!!! A lot of stuff has been moved. Modules have been created and destroyed, but everything is compiling and unit tests are passing, OMFG this is awesome.! 2013-08-30 18:21:04 -05:00
cheddar 386cc79420 [maven-release-plugin] prepare for next development iteration 2013-08-29 17:20:10 -05:00
cheddar 83e5e5434e [maven-release-plugin] prepare release druid-0.5.50 2013-08-29 17:20:05 -05:00
cheddar cb90ed05b0 Revert the previous commit. After going down this path, I realized that extracting things enough to allow Queries to be extended without depending on Druid proper was going to lead down a very nasty path. So, I've decided against that. Extending queries will require a tight dependency on Druid proper. 2013-08-29 16:45:03 -05:00
cheddar 2a46086e20 1) Didn't remove the io.druid files from client. Remove those and make sure things compile 
2) Switch DefaultObjectMapper to CommonObjectMapper
3) Create new DefaultObjectMapper in client that has Query stuff registered on it by default
 2013-08-29 15:25:36 -05:00
cheddar 5ad023cd3f Merge branch 'master' into guice 2013-08-29 13:59:59 -05:00
cheddar d9a8340dc7 [maven-release-plugin] prepare for next development iteration 2013-08-29 13:29:54 -05:00
cheddar 5118e1da12 [maven-release-plugin] prepare release druid-0.5.49 2013-08-29 13:29:49 -05:00
cheddar 9c30ced5ea 1) Move various "api" classes to io.druid packages and make sure things compile and stuff 2013-08-28 15:51:02 -05:00
cheddar ee1e73cfa1 1) Make it compile again after the merge 2013-08-27 14:36:01 -05:00
cheddar 8097450d8c Some things that didn't get committed with the merge for some reason!? 2013-08-27 14:29:03 -05:00
cheddar 5fa944dd26 Merge branch 'master' into guice 
Conflicts:
	client/src/main/java/com/metamx/druid/coordination/BatchDataSegmentAnnouncer.java
	client/src/main/java/com/metamx/druid/curator/announcement/Announcer.java
	client/src/main/java/com/metamx/druid/query/filter/SelectorDimFilter.java
	client/src/main/java/com/metamx/druid/query/search/SearchQueryQueryToolChest.java
	indexing-service/src/main/java/com/metamx/druid/indexing/common/tasklogs/S3TaskLogs.java
	indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/ForkingTaskRunner.java
	indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/RemoteTaskRunner.java
	indexing-service/src/main/java/com/metamx/druid/indexing/worker/WorkerCuratorCoordinator.java
	indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/RemoteTaskRunnerTest.java
	pom.xml
	server/src/main/java/com/metamx/druid/http/MasterMain.java
	server/src/main/java/com/metamx/druid/http/MasterServletModule.java
	server/src/main/java/com/metamx/druid/master/DruidMasterConfig.java
	server/src/test/java/com/metamx/druid/master/DruidMasterTest.java
	server/src/test/java/com/metamx/druid/query/group/GroupByQueryRunnerTest.java
 2013-08-27 14:27:32 -05:00
cheddar 3617ac17fc 1) Eliminate ExecutorMain and have it run using the new Main! 2013-08-27 14:11:05 -05:00
cheddar 269997dc94 1) ExecutorNode is working, except for the running of the task. Need to adjust it to be able to run a task and then everything will be wonderful 2013-08-26 18:08:41 -05:00
cheddar 6636ef1ea8 Remove unused files again 2013-08-23 18:00:56 -05:00
cheddar 55dbda2046 1) Worker appears to be running! It's also now known as the MiddleManager 2013-08-23 17:59:48 -05:00
cheddar 613ebd54b5 1) Delete unused things 2013-08-23 14:32:14 -05:00
cheddar b897c2cb22 1) IndexCoordinator appears to work as the CliOverlord now, yay! 2013-08-23 14:11:34 -05:00
cheddar 344ec1ec2f [maven-release-plugin] prepare for next development iteration 2013-08-23 13:20:09 -05:00
cheddar 5b0de6eb5c [maven-release-plugin] prepare release druid-0.5.48 2013-08-23 13:20:02 -05:00
fjy 4102dbd1e0 [maven-release-plugin] prepare for next development iteration 2013-08-21 21:50:55 -07:00
fjy 0daa2a95fe [maven-release-plugin] prepare release druid-0.5.47 2013-08-21 21:50:46 -07:00