Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
3,256 Commits 111 Branches 503 Tags 721 MiB
Commit Graph

623 Commits

Author SHA1 Message Date
fjy 01f9c1df31 fix broken task storage config and prepare for next release 2013-12-13 16:45:32 -08:00
fjy 7891e0a2e2 [maven-release-plugin] prepare for next development iteration 2013-12-13 16:20:23 -08:00
fjy 1d75738f81 [maven-release-plugin] prepare release druid-0.6.34 2013-12-13 16:20:18 -08:00
Gian Merlino 600dc7546f Configurability of recency threshold 2013-12-13 16:02:54 -08:00
fjy 4a8140be81 better messaging to console again 2013-12-13 15:04:25 -08:00
fjy 52cdb20f10 add better messaging and error handling 2013-12-13 15:01:07 -08:00
Gian Merlino e63c69dd57 TaskStorage: Return recently complete tasks in reverse chronological order 2013-12-13 12:27:45 -08:00
Gian Merlino 6c993d87bf Indexing service API and GUI improvements! 
- New APIs: waitingTasks, completeTasks, task payload
- GUI for the above, and for task logs + status
 2013-12-13 11:38:18 -08:00
Gian Merlino f36a5b677c TaskLifecycleTest: Add test for noop task 2013-12-13 07:48:28 -08:00
Gian Merlino 3b053a66ff TaskLifecycleTest: Add test for never-ready task 2013-12-13 07:48:27 -08:00
Gian Merlino 863012c384 TaskQueue: Exception during isReady does not warrant an alert. 2013-12-13 07:48:27 -08:00
Gian Merlino 6227963af9 TaskQueue: Copy task list before management loop. 2013-12-13 07:48:27 -08:00
fjy aef7726c33 [maven-release-plugin] prepare for next development iteration 2013-12-12 18:53:18 -08:00
fjy 1d1baf5976 [maven-release-plugin] prepare release druid-0.6.33 2013-12-12 18:53:14 -08:00
fjy f920dd6fb2 [maven-release-plugin] prepare for next development iteration 2013-12-12 17:28:38 -08:00
fjy 857d24a4ca [maven-release-plugin] prepare release druid-0.6.32 2013-12-12 17:28:34 -08:00
fjy 7c744ffda0 [maven-release-plugin] prepare for next development iteration 2013-12-12 14:53:01 -08:00
fjy 365da245c2 [maven-release-plugin] prepare release druid-0.6.31 2013-12-12 14:52:57 -08:00
Gian Merlino 70c153592f CliPeon: Fix local mode 2013-12-12 14:22:57 -08:00
Gian Merlino 370e2f855a TaskSerdeTest: Fix IndexTask test by including an actual firehoseFactory 2013-12-12 13:58:44 -08:00
Gian Merlino 169f149cf9 TaskLifecycleTest: Fix broken setUp and broken assumptions. 2013-12-12 13:51:13 -08:00
Gian Merlino ba757b1e5a IndexTask: Actually make and publish segments for the correct intervals. 2013-12-12 13:50:53 -08:00
Gian Merlino be25d51a2c RemoteTaskRunner: Fix issues leading to failing tests 2013-12-12 13:49:49 -08:00
Gian Merlino c60158a21a RemoteTaskRunner: Remove task from pendingTaskPayloads on shutdown if needed 2013-12-12 10:59:16 -08:00
Gian Merlino 0129ea99cf RemoteTaskRunner changes to make bootstrapping actually work. 
- Workers are not added to zkWorkers until caches have been initialized.
- Worker status we haven't heard about will be added to runningTasks or
  completeTasks as appropriate. 
- TaskRunnerWorkItem now only needs a taskId, not the entire Task. This makes
  it possible to create them from TaskStatus objects, if that's all we have.
- Also remove some dead code.
 2013-12-12 10:44:46 -08:00
Gian Merlino d92b88718c OverlordResource: Fix comment 2013-12-12 08:46:24 -08:00
Gian Merlino b6a52610bc IndexTask: Call plumber.startJob() 2013-12-12 08:46:10 -08:00
Gian Merlino db9b515e71 IndexTask: Remove unnecessary args to determinePartitions. 2013-12-12 08:46:00 -08:00
Gian Merlino f4a09d4ee3 TaskAction: Add JsonSubType for LockTryAcquireAction 2013-12-12 08:45:23 -08:00
Gian Merlino b17dc6f744 Task interval, isReady hygiene 2013-12-11 22:42:20 -08:00
Gian Merlino 05e24bd85c RemoteTaskRunner: Fix typo 2013-12-11 22:38:04 -08:00
Gian Merlino bed263efa5 VersionConverterTask: Less goofy import for Preconditions 2013-12-11 22:37:55 -08:00
Gian Merlino 53d90efe30 TaskQueueConfig: Copyright header 2013-12-11 22:37:40 -08:00
Gian Merlino 0adda97776 AbstractFixedIntervalTask: Copyright header 2013-12-11 22:37:28 -08:00
Gian Merlino c4b8c8bc6f Rework indexing service internals to hopefully be more reliable. 
The TaskQueue directly manages the TaskRunner. The main management loop runs
periodically and checks that the runner is doing reasonable things. If not, it
attempts to adjust the runner. The management loop also runs on-demand when a
task is added to keep task assignment relatively low latency. The TaskConsumer
is no longer necessary and so it no longer exists.

Task interval locks are handled differently. Instead of some tasks acquiring
locks at runtime and some tasks having implicit fixed lock intervals, all tasks
ask for locks explicitly. This occurs either in "isReady" (which runs on the
overlord) or in "run" (which runs on the peon).

Other changes:
- The TaskQueue is attached to the leader lifecycle, instead of global
- The TaskLockbox is able to sync itself from storage and is no longer
  bootstrapped by the TaskQueue.
- RemoteTaskRunner does not clean up zk paths until asked to. This will
  prevent deletion of statuses that have not yet been committed.
- Added retries on DbTaskStorage operations.
- Removed SpawnTasksAction (no more subtasks)
- Removed obsolete EventReceiverFirehose configs
- Removed obsolete OldOverlordResource
- Removed TaskStorageQueryAdapter methods related to subtasks
 2013-12-11 15:05:16 -08:00
fjy 96f679f31c clean up for merge 2013-12-10 17:51:13 -08:00
Gian Merlino f3cfd1d781 Introduce FileTaskLogs, and move TaskLogs module from server to indexing-service 2013-12-10 17:39:43 -08:00
Gian Merlino 47c1c8cab2 TaskStorage: Rename getRunningTasks -> getActiveTasks 2013-12-10 17:39:42 -08:00
fjy a7cc559623 [maven-release-plugin] prepare for next development iteration 2013-12-09 20:07:38 -08:00
fjy 54755b8864 [maven-release-plugin] prepare release druid-0.6.30 2013-12-09 20:07:34 -08:00
fjy 7a134272b3 [maven-release-plugin] prepare for next development iteration 2013-12-09 18:45:27 -08:00
fjy 829658f171 [maven-release-plugin] prepare release druid-0.6.29 2013-12-09 18:45:23 -08:00
fjy 98a0f40227 [maven-release-plugin] prepare for next development iteration 2013-12-09 18:35:22 -08:00
fjy 939d2313c0 [maven-release-plugin] prepare release druid-0.6.28 2013-12-09 18:35:18 -08:00
fjy c03d573e84 [maven-release-plugin] prepare for next development iteration 2013-12-09 18:30:15 -08:00
fjy 71fe8a43da [maven-release-plugin] prepare release druid-0.6.27 2013-12-09 18:30:11 -08:00
fjy 303f6ff334 fix worker config setup problems 2013-12-09 18:25:29 -08:00
fjy 8aba4ad3d2 Merge pull request #311 from metamx/versions 
Add versions to static page
 2013-12-09 17:18:16 -08:00
nishantmonu51 2186bd6cb7 Minor fixes and documentation changes 2013-12-09 19:07:48 +05:30
fjy 7c7cfdf819 [maven-release-plugin] prepare for next development iteration 2013-12-04 18:57:30 -08:00
fjy 6bfb3b8d41 [maven-release-plugin] prepare release druid-0.6.26 2013-12-04 18:57:25 -08:00
Himadri Singh 47278ad109 Merge relevant changes from 'druid-version-info' 2013-12-04 10:21:19 +05:30
fjy 1a3a1221c2 [maven-release-plugin] prepare for next development iteration 2013-12-02 14:22:54 -08:00
fjy b18b65108a [maven-release-plugin] prepare release druid-0.6.25 2013-12-02 14:22:50 -08:00
fjy 730e3a2303 [maven-release-plugin] prepare for next development iteration 2013-11-25 17:55:01 -08:00
fjy c7171e3311 [maven-release-plugin] prepare release druid-0.6.24 2013-11-25 17:54:57 -08:00
fjy 5c6e647148 [maven-release-plugin] prepare for next development iteration 2013-11-20 12:00:15 -08:00
fjy 5b94895a36 [maven-release-plugin] prepare release druid-0.6.23 2013-11-20 12:00:11 -08:00
fjy ea95566cd9 [maven-release-plugin] prepare for next development iteration 2013-11-20 10:45:21 -08:00
fjy fe7122b717 [maven-release-plugin] prepare release druid-0.6.22 2013-11-20 10:45:17 -08:00
fjy 88378eb036 [maven-release-plugin] prepare for next development iteration 2013-11-20 10:28:57 -08:00
fjy 2e6056f537 [maven-release-plugin] prepare release druid-0.6.21 2013-11-20 10:28:52 -08:00
fjy 7fec5e6ff7 [maven-release-plugin] prepare for next development iteration 2013-11-19 09:59:48 -08:00
fjy 265ef23cd5 [maven-release-plugin] prepare release druid-0.6.20 2013-11-19 09:59:44 -08:00
fjy 5d7173ac98 fix autoscaling termination duration bug 2013-11-15 19:21:25 -08:00
fjy 48d275e5cb [maven-release-plugin] prepare for next development iteration 2013-11-14 18:20:29 -08:00
fjy c31c5ce1ac [maven-release-plugin] prepare release druid-0.6.19 2013-11-14 18:20:25 -08:00
fjy 346cf0e04c fix out of order in urls for hadoop classpath 2013-11-14 18:18:39 -08:00
fjy d935e1b47b [maven-release-plugin] prepare for next development iteration 2013-11-14 15:08:12 -08:00
fjy 32dbbe4f63 [maven-release-plugin] prepare release druid-0.6.18 2013-11-14 15:08:09 -08:00
fjy 6b41681424 fix ordering of urls, ugh, need to write more tests 2013-11-14 15:06:07 -08:00
fjy 373251a476 [maven-release-plugin] prepare for next development iteration 2013-11-14 14:31:26 -08:00
fjy 7120db20a4 [maven-release-plugin] prepare release druid-0.6.17 2013-11-14 14:31:22 -08:00
fjy 23ebca6d32 more hadoop dependency hell, getting the right urls to hadoop is hard 2013-11-14 14:29:42 -08:00
fjy 89e6ce3550 [maven-release-plugin] prepare for next development iteration 2013-11-14 13:18:24 -08:00
fjy 97293b3f04 [maven-release-plugin] prepare release druid-0.6.16 2013-11-14 13:18:20 -08:00
fjy c51eed060f actually use a class loader pull out dependencies from extension modules 2013-11-14 11:52:21 -08:00
fjy e7118cd9be [maven-release-plugin] prepare for next development iteration 2013-11-14 10:54:37 -08:00
fjy d3f46c826a [maven-release-plugin] prepare release druid-0.6.15 2013-11-14 10:54:32 -08:00
fjy fcf0c6ce06 change the classpath ordering for batch processing and prepare for next release 2013-11-14 10:52:51 -08:00
fjy 6e7e2428e8 [maven-release-plugin] prepare for next development iteration 2013-11-13 13:27:47 -08:00
fjy 9c4cecc015 [maven-release-plugin] prepare release druid-0.6.14 2013-11-13 13:27:43 -08:00
fjy 6c41cd8682 [maven-release-plugin] prepare for next development iteration 2013-11-13 11:38:44 -08:00
fjy b93550194a [maven-release-plugin] prepare release druid-0.6.13 2013-11-13 11:38:39 -08:00
fjy 9bbaade8f9 [maven-release-plugin] prepare for next development iteration 2013-11-13 10:37:12 -08:00
fjy 4dffef698e [maven-release-plugin] prepare release druid-0.6.12 2013-11-13 10:37:08 -08:00
fjy 12a188d8d6 [maven-release-plugin] prepare for next development iteration 2013-11-12 15:20:54 -08:00
fjy 63d9f7362a [maven-release-plugin] prepare release druid-0.6.11 2013-11-12 15:20:47 -08:00
fjy 0681fb614d [maven-release-plugin] prepare for next development iteration 2013-11-08 15:56:10 -08:00
fjy de5984ccea [maven-release-plugin] prepare release druid-0.6.10 2013-11-08 15:56:06 -08:00
fjy d212b3fc43 [maven-release-plugin] prepare for next development iteration 2013-11-08 15:14:46 -08:00
fjy 4cd4578539 [maven-release-plugin] prepare release druid-0.6.9 2013-11-08 15:14:41 -08:00
fjy fd4fb78e1f [maven-release-plugin] prepare for next development iteration 2013-11-08 11:34:12 -08:00
fjy f8d5d127a4 [maven-release-plugin] prepare release druid-0.6.8 2013-11-08 11:34:09 -08:00
fjy 64b93bf448 fix broken autoscaling and prepare for next release 2013-11-08 11:29:42 -08:00
fjy 90240e6f1e [maven-release-plugin] prepare for next development iteration 2013-11-07 23:29:02 -08:00
fjy c3d6233bc4 [maven-release-plugin] prepare release druid-0.6.7 2013-11-07 23:28:58 -08:00
fjy d70bcc5657 repair broken pom 2013-11-07 23:27:06 -08:00
fjy fafd9c8d27 [maven-release-plugin] prepare for next development iteration 2013-11-07 18:20:19 -08:00
fjy f945416538 [maven-release-plugin] prepare release druid-0.6.5 2013-11-07 18:20:16 -08:00
fjy ed328aca6e [maven-release-plugin] prepare for next development iteration 2013-11-07 18:03:39 -08:00
fjy e34f1a829b [maven-release-plugin] prepare release druid-0.6.4 2013-11-07 18:03:36 -08:00
fjy a049b42674 fix an issue with task tables not getting created automatically and prepare for next release 2013-11-07 18:01:35 -08:00
fjy 009646ed56 [maven-release-plugin] prepare for next development iteration 2013-11-07 17:40:12 -08:00
fjy 4cde47c9d1 [maven-release-plugin] prepare release druid-0.6.3 2013-11-07 17:40:06 -08:00
fjy e6a83c0339 [maven-release-plugin] prepare for next development iteration 2013-11-07 17:05:52 -08:00
fjy 2ac09b798d [maven-release-plugin] prepare release druid-0.6.2 2013-11-07 17:05:48 -08:00
fjy 7f85a126ac fix broken event receiving firehose 2013-11-07 16:59:01 -08:00
fjy 621133d6f2 removing dead code 2013-11-07 16:53:34 -08:00
fjy 084c90aa19 cleanup to prepare for next release 2013-11-07 15:55:51 -08:00
fjy bad1a7e9f8 fix according to code review 2013-11-07 15:52:34 -08:00
fjy aeb411a3a3 fix according to code review and fix broken examples 2013-11-07 15:42:48 -08:00
fjy e2e10fae1f clean up code 2013-11-07 15:03:14 -08:00
fjy 913ff3a082 clean up code 2013-11-07 15:00:11 -08:00
fjy 6b573c76f1 more fixes 2013-11-07 14:40:45 -08:00
fjy ee15fd9f85 [maven-release-plugin] prepare for next development iteration 2013-11-05 15:42:37 -08:00
fjy 713015199a [maven-release-plugin] prepare release druid-0.6.1 2013-11-05 15:42:33 -08:00
Gian Merlino 8660db93fc RemoteTaskRunner: Run taskComplete after a task times out 2013-11-04 10:56:26 -08:00
Gian Merlino 186bbd1cb6 WorkerTaskMonitor: Add log message 2013-11-04 10:25:16 -08:00
Gian Merlino 781673a8f8 ForkingTaskRunner: Fix pass-down of nodeType 2013-10-23 14:13:05 -07:00
fjy a684885839 [maven-release-plugin] prepare for next development iteration 2013-10-18 15:48:16 -07:00
fjy 88dcfe9a94 [maven-release-plugin] prepare release druid-0.6.0 2013-10-18 15:48:12 -07:00
Gian Merlino b68b3526e8 IndexingServiceFirehoseModule: Add header 2013-10-18 13:27:40 -07:00
Gian Merlino 2e9c46867f Fixes for indexing service. 
- Create IndexingServiceFirehoseModule so firehoses can be loaded by all mains
- Fix implicit lock acquisition in AbstractTask
 2013-10-18 11:14:33 -07:00
fjy 4862852b43 more docs about how to use different versions of hadoop in druid 2013-10-16 17:54:49 -07:00
fjy 6192602893 fix extensions config not getting picked up in hadoop index task 2013-10-16 16:52:23 -07:00
fjy a1c09df17f make the hadoop index task work again 2013-10-16 09:45:17 -07:00
fjy 9796a40b92 port docs over to 0.6 and a bunch of misc fixes 2013-10-11 18:38:53 -07:00
fjy a9a723bd11 clean up poms, add a new loading your own data tutorial, add new validation, clean up logs 2013-10-09 15:42:39 -07:00
fjy 4e509d1d09 Merge branch 'master' into is-docs 2013-10-09 14:05:10 -07:00
cheddar c47fe202c7 Fix HadoopDruidIndexer to work with the new way of things 
There are multiple and sundry changes in here.

First, "HadoopDruidIndexer" has been split into two pieces, (1) CliHadoop which pulls the hadoop version and builds up the right classpath with the proper hadoop version to run the indexer and (2) CliInternalHadoopIndexer which actually runs the indexer.

In order to work around a bunch of jets3t version conflicts with Hadoop and Druid, I needed to extract the S3 deep storage stuff into its own module.  I then also moved the HDFS stuff into its own module so that I could eliminate the dependency on Hadoop for druid-server.

In doing these changes, I wanted to make the extensions buildable with only the druid-api jar, so a few other things had to move out of Druid and into druid-api.  They are all API-level things, however, so they really belong in druid-api instead.

Lastly, I removed the druid-realtime module and put it all in druid-server.
 2013-10-09 15:15:44 -05:00
fjy 4ec4b8e024 rewrite indexing service docs 2013-10-08 16:34:58 -07:00
fjy 703b674800 add availability zone info to autoscaling 2013-10-07 12:16:50 -07:00
fjy ac330f72bb first set of changes to standarize the naming convention we use in druid 2013-10-03 16:36:48 -07:00
fjy 17874eeb67 make the CliPeon actually able to run on its own 2013-10-02 15:55:10 -07:00
fjy bc8db7daa5 1) make chat handler resource work again 
2) add more default configs
3) make examples work again
 2013-10-02 14:22:39 -07:00
Gian Merlino 384dcda7e4 Chat handlers still don't work, but, they're closer maybe. 2013-10-01 17:45:53 -07:00
fjy 59f2d0711d Merge branch 'master' into local-index 2013-10-01 13:21:43 -07:00
fjy 30df53671e remove line iterator factory because it is not needed 2013-10-01 13:21:20 -07:00
Gian Merlino 62eda5020c ShardSpec: Remove isInChunk(Map<String, String>) 2013-10-01 12:50:08 -07:00
fjy 5d0d71250b fix chat handler resources not correctly registering themselves 2013-10-01 11:25:39 -07:00
fjy 53698a135a add interface to new firehose as per code review comments 2013-09-30 18:00:59 -07:00
fjy f55a5199b1 add a firehose module to remove so much copy and pasted code 2013-09-30 16:29:20 -07:00
fjy ed9e0cf9f6 add a local firehose for indexing local files 2013-09-30 16:03:26 -07:00
Gian Merlino dc5dab8747 Fixes for property conversion, firehose registration, and the indexing service 2013-09-27 17:09:59 -07:00
fjy 0b04325ee8 fix things up according to code review comments 2013-09-27 10:17:45 -07:00
fjy e404295c1f make indexing service work 2013-09-26 17:44:21 -07:00
fjy 8bc56daa66 fix things up according to code review comments 2013-09-26 11:35:45 -07:00
fjy 87259321b6 port hadoop druid indexer to new guice framework 2013-09-26 11:04:42 -07:00
fjy 15843c3978 refactor how server service discovery is done 2013-09-24 10:36:26 -07:00
fjy dc8a119787 fix broken unit tests are a result of the last merge 2013-09-23 12:56:01 -07:00
cheddar 5712b29c8c Fix issues with bindings and handling extensions 
The way the Guice bindings were setup previously, each process only had bindings
for the things it cared about.  This became problematic when adding extension modules
that bound everything that they could possibly need expecting that the processes would
only instantiate what they actually do need.  Guice tries to fail-fast and verifies that all
 bindings exist before it does anything, which is a problem because the extension bind
 some objects that don't necessarily have all of their dependencies bound in all processes.

The fix for this is to build a single Injector with all bindings in it and let each of the
 processes only load the things that they care about.  This also requires the use of
 Module overrides and other such interesting things, which are node done.

 In doing the fix, I also swapped out the way that the DataSegmentPusher/Puller stuff is bound, as well as made the Cassandra stuff fail if its settings are not provided.  This all of a sudden made all of the things require Cassandra's settings, so I migrated the Cassandra deep storage stuff into its own module.

 In doing these changes, I also discovered that some properties weren't properly converting for the ConvertProperties command (specifically, the properties related to data segment loading and pushing), so I fixed that.
 2013-09-20 17:45:01 -05:00
cheddar dc728b5721 Update versions of poms and make IncrementalIndexStorageAdapterTest compile 2013-09-18 17:01:03 -05:00
cheddar 6bfe2310bb Merge branch 'master' into guice 
Conflicts:
	client/pom.xml
	client/src/main/java/com/metamx/druid/query/segment/LegacySegmentSpec.java
	indexing-common/pom.xml
	indexing-service/src/main/java/com/metamx/druid/indexing/common/TaskToolbox.java
	indexing-service/src/main/java/com/metamx/druid/indexing/common/TaskToolboxFactory.java
	indexing-service/src/main/java/com/metamx/druid/indexing/common/task/RealtimeIndexTask.java
	indexing-service/src/main/java/com/metamx/druid/indexing/worker/executor/ExecutorNode.java
	indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/RemoteTaskRunnerTest.java
	indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/TaskLifecycleTest.java
	indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/TaskQueueTest.java
	realtime/src/main/java/com/metamx/druid/realtime/RealtimeNode.java
	realtime/src/main/java/com/metamx/druid/realtime/firehose/IrcFirehoseFactory.java
	realtime/src/main/java/com/metamx/druid/realtime/firehose/WikipediaIrcDecoder.java
	realtime/src/main/java/com/metamx/druid/realtime/plumber/RealtimePlumberSchool.java
	server/src/main/java/com/metamx/druid/BaseServerNode.java
	server/src/main/java/com/metamx/druid/http/ComputeNode.java
	server/src/main/java/com/metamx/druid/index/v1/IncrementalIndexStorageAdapter.java
	server/src/main/java/com/metamx/druid/query/group/GroupByQueryEngine.java
 2013-09-18 16:46:54 -05:00
fjy cabae7993d port over multi threaded realtime and also fix broken realtime nodes that can't start up 2013-09-16 16:03:47 -07:00
Gian Merlino 163fa14e62 [maven-release-plugin] prepare for next development iteration 2013-09-13 19:02:46 -07:00
Gian Merlino a20f3fc6dd [maven-release-plugin] prepare release druid-0.5.58 2013-09-13 19:02:40 -07:00
fjy d0b0adbb68 [maven-release-plugin] prepare for next development iteration 2013-09-13 18:57:25 -07:00
fjy 23c5fdb107 [maven-release-plugin] prepare release druid-0.5.57 2013-09-13 18:57:20 -07:00
fjy b0a09ba1f1 [maven-release-plugin] prepare for next development iteration 2013-09-13 18:45:20 -07:00
fjy b17f5c403b [maven-release-plugin] prepare release druid-0.5.56 2013-09-13 18:45:14 -07:00
Gian Merlino 22a99c7589 [maven-release-plugin] prepare for next development iteration 2013-09-13 17:59:03 -07:00
Gian Merlino 542d564453 [maven-release-plugin] prepare release druid-0.5.55 2013-09-13 17:58:56 -07:00
Gian Merlino 24df6c482d Use shared query executor in indexing-service 2013-09-13 14:33:13 -07:00
fjy f7c10e3594 rework tests in indexing service to be more unit testy 2013-09-12 16:37:58 -07:00
cheddar a2dcc45a8e 1) Remove SingleSegmentLoader and replace with OmniSegmentLoader 2013-09-12 11:47:03 -05:00
cheddar 6c9a107356 1) remove duplicate package initialization.initialization 2013-09-09 17:02:57 -05:00
fjy e03c623467 [maven-release-plugin] prepare for next development iteration 2013-09-05 13:48:21 -07:00
fjy 552d0c061a [maven-release-plugin] prepare release druid-0.5.54 2013-09-05 13:48:11 -07:00
fjy 5a64e2a64a [maven-release-plugin] prepare for next development iteration 2013-09-04 14:24:04 -07:00
fjy 523355d00b [maven-release-plugin] prepare release druid-0.5.53 2013-09-04 14:23:56 -07:00
fjy 567168330b [maven-release-plugin] prepare for next development iteration 2013-09-04 14:06:33 -07:00
fjy 39bda32b81 [maven-release-plugin] prepare release druid-0.5.52 2013-09-04 14:06:25 -07:00
fjy f08b8287a4 [maven-release-plugin] prepare for next development iteration 2013-09-04 13:22:19 -07:00
fjy cf9038abb4 [maven-release-plugin] prepare release druid-0.5.51 2013-09-04 13:22:11 -07:00
cheddar 3c39f90c89 1) Move Firehose interface and dependencies to druid-api 
2) Move DataSegment* interfaces and dependencies to druid-api
 2013-08-31 16:43:28 -05:00
cheddar 5ab671050e No more com.metamx.druid, it is now all io.druid! 2013-08-30 19:42:12 -05:00
cheddar bd0756e360 More stuff moved, things still compiling and tests still passing. Yay! 2013-08-30 18:58:35 -05:00
cheddar 56e2b956d0 OMG!!! A lot of stuff has been moved. Modules have been created and destroyed, but everything is compiling and unit tests are passing, OMFG this is awesome.! 2013-08-30 18:21:04 -05:00
cheddar 386cc79420 [maven-release-plugin] prepare for next development iteration 2013-08-29 17:20:10 -05:00
cheddar 83e5e5434e [maven-release-plugin] prepare release druid-0.5.50 2013-08-29 17:20:05 -05:00
cheddar cb90ed05b0 Revert the previous commit. After going down this path, I realized that extracting things enough to allow Queries to be extended without depending on Druid proper was going to lead down a very nasty path. So, I've decided against that. Extending queries will require a tight dependency on Druid proper. 2013-08-29 16:45:03 -05:00
cheddar 2a46086e20 1) Didn't remove the io.druid files from client. Remove those and make sure things compile 
2) Switch DefaultObjectMapper to CommonObjectMapper
3) Create new DefaultObjectMapper in client that has Query stuff registered on it by default
 2013-08-29 15:25:36 -05:00
cheddar 5ad023cd3f Merge branch 'master' into guice 2013-08-29 13:59:59 -05:00
cheddar d9a8340dc7 [maven-release-plugin] prepare for next development iteration 2013-08-29 13:29:54 -05:00
cheddar 5118e1da12 [maven-release-plugin] prepare release druid-0.5.49 2013-08-29 13:29:49 -05:00
cheddar 9c30ced5ea 1) Move various "api" classes to io.druid packages and make sure things compile and stuff 2013-08-28 15:51:02 -05:00
cheddar ee1e73cfa1 1) Make it compile again after the merge 2013-08-27 14:36:01 -05:00
cheddar 8097450d8c Some things that didn't get committed with the merge for some reason!? 2013-08-27 14:29:03 -05:00
cheddar 5fa944dd26 Merge branch 'master' into guice 
Conflicts:
	client/src/main/java/com/metamx/druid/coordination/BatchDataSegmentAnnouncer.java
	client/src/main/java/com/metamx/druid/curator/announcement/Announcer.java
	client/src/main/java/com/metamx/druid/query/filter/SelectorDimFilter.java
	client/src/main/java/com/metamx/druid/query/search/SearchQueryQueryToolChest.java
	indexing-service/src/main/java/com/metamx/druid/indexing/common/tasklogs/S3TaskLogs.java
	indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/ForkingTaskRunner.java
	indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/RemoteTaskRunner.java
	indexing-service/src/main/java/com/metamx/druid/indexing/worker/WorkerCuratorCoordinator.java
	indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/RemoteTaskRunnerTest.java
	pom.xml
	server/src/main/java/com/metamx/druid/http/MasterMain.java
	server/src/main/java/com/metamx/druid/http/MasterServletModule.java
	server/src/main/java/com/metamx/druid/master/DruidMasterConfig.java
	server/src/test/java/com/metamx/druid/master/DruidMasterTest.java
	server/src/test/java/com/metamx/druid/query/group/GroupByQueryRunnerTest.java
 2013-08-27 14:27:32 -05:00
cheddar 3617ac17fc 1) Eliminate ExecutorMain and have it run using the new Main! 2013-08-27 14:11:05 -05:00
cheddar 269997dc94 1) ExecutorNode is working, except for the running of the task. Need to adjust it to be able to run a task and then everything will be wonderful 2013-08-26 18:08:41 -05:00
cheddar 6636ef1ea8 Remove unused files again 2013-08-23 18:00:56 -05:00
cheddar 55dbda2046 1) Worker appears to be running! It's also now known as the MiddleManager 2013-08-23 17:59:48 -05:00
cheddar 613ebd54b5 1) Delete unused things 2013-08-23 14:32:14 -05:00
cheddar b897c2cb22 1) IndexCoordinator appears to work as the CliOverlord now, yay! 2013-08-23 14:11:34 -05:00
cheddar 344ec1ec2f [maven-release-plugin] prepare for next development iteration 2013-08-23 13:20:09 -05:00
cheddar 5b0de6eb5c [maven-release-plugin] prepare release druid-0.5.48 2013-08-23 13:20:02 -05:00
fjy 4102dbd1e0 [maven-release-plugin] prepare for next development iteration 2013-08-21 21:50:55 -07:00
fjy 0daa2a95fe [maven-release-plugin] prepare release druid-0.5.47 2013-08-21 21:50:46 -07:00
fjy d92ab8bb58 more logs for RTR 2013-08-21 21:47:59 -07:00
fjy b1d0f989ec [maven-release-plugin] prepare for next development iteration 2013-08-21 20:28:15 -07:00
fjy ab467bf42a [maven-release-plugin] prepare release druid-0.5.46 2013-08-21 20:28:06 -07:00
fjy 1037a48a8e [maven-release-plugin] prepare for next development iteration 2013-08-21 18:55:49 -07:00
fjy 6b9ec95f01 [maven-release-plugin] prepare release druid-0.5.45 2013-08-21 18:55:40 -07:00
fjy 54f00479cc add explicit null check for moving tasks from pending to running 2013-08-21 13:02:35 -07:00
fjy 88661b26a0 bug fix for RTR removing workers race condition and partition chunks not being sorted by chunk number 2013-08-21 11:14:54 -07:00
fjy ed666d9d5f [maven-release-plugin] prepare for next development iteration 2013-08-20 19:37:52 -07:00
fjy 66dcfe8c3e [maven-release-plugin] prepare release druid-0.5.44 2013-08-20 19:37:44 -07:00
fjy e283de6831 fix another bug with RTR to remove things correctly from running tasks 2013-08-20 19:34:30 -07:00
fjy baedacf970 [maven-release-plugin] prepare for next development iteration 2013-08-20 19:29:05 -07:00
fjy a4eb025ba7 [maven-release-plugin] prepare release druid-0.5.43 2013-08-20 19:28:56 -07:00
fjy d02be16245 fix RTR closing PCC too early 2013-08-20 19:25:16 -07:00
Gian Merlino a3793eeba4 [maven-release-plugin] prepare for next development iteration 2013-08-20 17:58:28 -07:00
Gian Merlino b792bf2ba5 [maven-release-plugin] prepare release druid-0.5.42 2013-08-20 17:58:22 -07:00
Gian Merlino 70ab225770 Add missing license headers 2013-08-20 17:50:10 -07:00
Gian Merlino 455645e723 Workers announce TaskAnnouncement rather than TaskStatus 2013-08-20 16:14:36 -07:00
Gian Merlino 9609314765 ForkingTaskRunner: Make TaskInfo into ForkingTaskRunnerWorkItem 
This allows the API/GUI to return reasonable results when the primary
task runner is a ForkingTaskRunner.
 2013-08-20 14:04:28 -07:00
Gian Merlino 4e8325f963 Better tests and error messages for TaskResource 2013-08-20 14:01:38 -07:00
Gian Merlino b102d67173 Fix getResourceManagementScheduler for non-autoscaling configs 2013-08-20 13:39:05 -07:00
Gian Merlino 25e330780c Simplify AbstractTask constructor 2013-08-20 13:38:52 -07:00
Gian Merlino d8493f8e26 RealtimeIndexTask: Fix "resource" serde 2013-08-20 13:02:52 -07:00
fjy 47ec240703 [maven-release-plugin] prepare for next development iteration 2013-08-15 13:47:57 -07:00
fjy 33e2fb3e21 [maven-release-plugin] prepare release druid-0.5.41 2013-08-15 13:47:48 -07:00
fjy 5c600c7012 change error msg to alert 2013-08-15 13:27:41 -07:00
fjy 4e7dac18b9 fix condition where status would be returned even if worker not running task 2013-08-15 13:12:57 -07:00
fjy 1fb6107a37 fix the case where RTR does not clean up a completed task on startup 2013-08-15 13:09:02 -07:00
Gian Merlino a0fcd02b3c [maven-release-plugin] prepare for next development iteration 2013-08-14 17:03:40 -07:00
Gian Merlino b6c445c726 [maven-release-plugin] prepare release druid-0.5.40 2013-08-14 17:03:28 -07:00
fjy 36d02ab943 [maven-release-plugin] prepare for next development iteration 2013-08-14 14:05:49 -07:00
fjy 8c841e90a4 [maven-release-plugin] prepare release druid-0.5.39 2013-08-14 14:05:39 -07:00
fjy 761a411b16 [maven-release-plugin] prepare for next development iteration 2013-08-14 10:13:50 -07:00
fjy 4dd727d974 [maven-release-plugin] prepare release druid-0.5.38 2013-08-14 10:13:34 -07:00
fjy 9870cab855 [maven-release-plugin] prepare for next development iteration 2013-08-13 17:21:56 -07:00
fjy e4c998cfef [maven-release-plugin] prepare release druid-0.5.37 2013-08-13 17:21:46 -07:00
Gian Merlino 246bd0fbaa [maven-release-plugin] prepare for next development iteration 2013-08-13 07:27:24 -07:00
Gian Merlino a2e9441693 [maven-release-plugin] prepare release druid-0.5.36 2013-08-13 07:27:18 -07:00
fjy d4fc069e01 [maven-release-plugin] prepare for next development iteration 2013-08-12 18:58:14 -07:00
fjy dd8c96c734 [maven-release-plugin] prepare release druid-0.5.35 2013-08-12 18:58:06 -07:00
Gian Merlino 8d7a4f4493 Retries for S3TaskLogs, S3DataSegmentPusher 2013-08-12 14:27:34 -07:00
fjy 1f4e0ea40c [maven-release-plugin] prepare for next development iteration 2013-08-12 13:21:29 -07:00
fjy 1c2ad4f08c [maven-release-plugin] prepare release druid-0.5.34 2013-08-12 13:21:21 -07:00
fjy 68a3f1ab79 [maven-release-plugin] prepare for next development iteration 2013-08-08 22:15:04 -07:00
fjy ce5679554d [maven-release-plugin] prepare release druid-0.5.33 2013-08-08 22:14:55 -07:00
fjy ddc2e87c4c [maven-release-plugin] prepare for next development iteration 2013-08-06 21:21:28 -07:00
fjy ffbcd18cdc [maven-release-plugin] prepare release druid-0.5.32 2013-08-06 21:21:18 -07:00
fjy df883a9823 learn to type 2013-08-06 21:18:36 -07:00
fjy 4617d37e28 [maven-release-plugin] prepare for next development iteration 2013-08-06 21:08:02 -07:00
fjy 53b248dc28 [maven-release-plugin] prepare release druid-0.5.31 2013-08-06 21:07:52 -07:00
fjy 795657aedf fix bug where workers with same capacity would not be unique 2013-08-06 21:04:13 -07:00
Gian Merlino 99a3de1597 [maven-release-plugin] prepare for next development iteration 2013-08-06 17:41:58 -07:00
Gian Merlino c0ccfd66ea [maven-release-plugin] prepare release druid-0.5.30 2013-08-06 17:41:51 -07:00
fjy 930b089f7f [maven-release-plugin] prepare for next development iteration 2013-08-06 16:20:37 -07:00
fjy dd5f5ae49b [maven-release-plugin] prepare release druid-0.5.29 2013-08-06 16:20:29 -07:00
fjy ebf1ac47f0 Merge branch 'master' of github.com:metamx/druid 2013-08-06 15:38:25 -07:00
fjy 9d0e4a94f0 alert when task fails in RTR assign 2013-08-06 15:38:17 -07:00
cheddar 4a64ce37ed Finish the merging, wtf IntelliJ? 2013-08-06 13:34:24 -07:00
cheddar eee1efdcb5 Merge branch 'master' into guice 
Conflicts:
	client/src/main/java/com/metamx/druid/client/DruidServerConfig.java
	indexing-service/src/main/java/com/metamx/druid/indexing/common/index/ChatHandlerProvider.java
	indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/TaskMasterLifecycle.java
	indexing-service/src/main/java/com/metamx/druid/indexing/worker/executor/ExecutorNode.java
	indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/TaskLifecycleTest.java
 2013-08-06 13:33:31 -07:00
Gian Merlino ba847bff32 [maven-release-plugin] prepare for next development iteration 2013-08-05 19:58:40 -07:00
Gian Merlino 7101d9ad2e [maven-release-plugin] prepare release druid-0.5.28 2013-08-05 19:58:34 -07:00
Gian Merlino a1904c9b3b ChatHandlerResource: Fix Guice type errors 2013-08-05 19:56:03 -07:00
fjy db695f49c2 [maven-release-plugin] prepare for next development iteration 2013-08-05 18:26:20 -07:00
fjy c22e5b7ee0 [maven-release-plugin] prepare release druid-0.5.27 2013-08-05 18:26:10 -07:00
fjy d1b2a5a4b3 fix indexer console serde of running tasks 2013-08-05 18:22:12 -07:00
fjy 8ff0b53df4 [maven-release-plugin] prepare for next development iteration 2013-08-05 18:00:46 -07:00
fjy 1d3729906d [maven-release-plugin] prepare release druid-0.5.26 2013-08-05 18:00:36 -07:00
fjy 479f0cefca fix bug with RTR not assigning tasks when a new worker is available 2013-08-05 17:57:59 -07:00
fjy 90e082463b [maven-release-plugin] prepare for next development iteration 2013-08-05 16:10:19 -07:00
fjy eb276d34d7 [maven-release-plugin] prepare release druid-0.5.25 2013-08-05 16:10:11 -07:00
fjy 626cf14a6e fix bug where the curator config name was changed in one place but not another; make some info msgs into debug msgs; fix zkworker serialization 2013-08-05 16:02:26 -07:00
fjy a95d9c46e2 [maven-release-plugin] prepare for next development iteration 2013-08-05 14:47:35 -07:00
fjy 8e1f4d3256 [maven-release-plugin] prepare release druid-0.5.24 2013-08-05 14:47:26 -07:00
fjy 66c658305f Merge branch 'master' of github.com:metamx/druid 2013-08-05 14:44:09 -07:00
fjy 35f89d7232 make RTR idempotent to multiple run requests for same task, because higher level things in the indexing service require this behaviour 2013-08-05 14:44:01 -07:00
Gian Merlino efd34f3a8b TaskRunner: Fix comment 2013-08-05 14:20:31 -07:00
fjy fbb1211cbc [maven-release-plugin] prepare for next development iteration 2013-08-05 11:44:36 -07:00
fjy 22a90d36b7 [maven-release-plugin] prepare release druid-0.5.23 2013-08-05 11:44:28 -07:00
cheddar 2361e0112a Make it all compile again... 2013-08-02 10:14:46 -07:00
fjy c33f2f06ff fix logic of how to assign tasks to workers 2013-08-02 09:01:02 -07:00
fjy 584ccac833 move scanning of workers and tasks into RTR start, simplify bootstrap, make tests better 2013-08-01 17:50:05 -07:00
cheddar 9e78bb38f5 Merge branch 'master' into guice 
Conflicts:
	client/src/main/java/com/metamx/druid/QueryableNode.java
	client/src/main/java/com/metamx/druid/client/ServerInventoryView.java
	client/src/main/java/com/metamx/druid/coordination/SingleDataSegmentAnnouncer.java
	client/src/main/java/com/metamx/druid/initialization/CuratorDiscoveryConfig.java
	client/src/main/java/com/metamx/druid/query/MetricsEmittingExecutorService.java
	indexing-hadoop/src/test/java/com/metamx/druid/indexer/HadoopDruidIndexerConfigTest.java
	indexing-service/src/main/java/com/metamx/druid/indexing/common/TaskToolbox.java
	indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/http/IndexerCoordinatorNode.java
	indexing-service/src/main/java/com/metamx/druid/indexing/worker/executor/ExecutorNode.java
	indexing-service/src/main/java/com/metamx/druid/indexing/worker/http/WorkerNode.java
	pom.xml
	server/src/main/java/com/metamx/druid/coordination/ServerManager.java
	server/src/main/java/com/metamx/druid/coordination/ZkCoordinator.java
	server/src/main/java/com/metamx/druid/db/DatabaseRuleManager.java
	server/src/main/java/com/metamx/druid/db/DatabaseSegmentManager.java
	server/src/main/java/com/metamx/druid/http/ComputeNode.java
	server/src/main/java/com/metamx/druid/http/MasterMain.java
	server/src/main/java/com/metamx/druid/loading/SegmentLoaderConfig.java
	server/src/main/java/com/metamx/druid/loading/SingleSegmentLoader.java
	server/src/main/java/com/metamx/druid/master/DruidMaster.java
 2013-08-01 16:42:47 -07:00
cheddar 019bb5d453 1) Another whole bunch of changes to annotate things and create Modules and bind stuff. But OMFG, the compute node actually appears to be working! 
2) The compute node works with Guice
3) The compute node fires up with Guice and appears to work
4) Did I mention that the compute node, now called historical node, fires up with Guice and appears to work?
 2013-08-01 15:28:08 -07:00
fjy a4edc2221d fix RTR comments 2013-07-31 15:28:52 -07:00
fjy 215d147a69 Merge branch 'worker-resource' of github.com:metamx/druid into worker-resource 2013-07-31 15:23:49 -07:00
fjy e2b5cd6067 Merge branch 'master' of github.com:metamx/druid into worker-resource 
Conflicts:
	indexing-service/src/main/java/com/metamx/druid/indexing/worker/executor/ExecutorNode.java
 2013-07-31 15:23:13 -07:00
Gian Merlino eaddce06d5 Call TaskRunner.bootstrap immediately after starting it 2013-07-30 15:26:11 -07:00
fjy 50836798fa toggle between compressed and non compressed service discovery 2013-07-29 15:40:45 -07:00
fjy ad65c8111d fix logs 2013-07-29 11:41:42 -07:00
fjy 4ae8395538 1) on bootstrap, load all initial data and do a compare with bootstrapped tasks, delete any that are extra out there 
2) change autoscaling logic such that it only works with remote task runnrs
3) zk workers use their own status caches to determine what they are running
 2013-07-26 14:32:08 -07:00
cheddar 6b9963b472 1) Move the @Json and @Smile annotations to com.metamx.druid.guice.annotations 
2) Changed ServiceAnnouncer to take a DruidNode object
3) Remove ServiceInstanceFactory interface and implementations
4) Add registrations to DiscoveryModule so that you can register nodes that should be announced on startup
5) Make the set of default Monitors configurable
 2013-07-26 14:03:51 -07:00
fjy a1262760b2 Merge branch 'master' into worker-resource 2013-07-26 10:02:15 -07:00
fjy b0090a1de6 [maven-release-plugin] prepare for next development iteration 2013-07-25 17:26:44 -07:00
fjy 6868153905 [maven-release-plugin] prepare release druid-0.5.22 2013-07-25 17:26:36 -07:00
cheddar a6bc63732d Merge pull request #199 from metamx/indexing-service-mods 
Improve robustness and usability of RealtimeIndexTask and some code cleanup in the Indexing Service
 2013-07-25 17:09:15 -07:00
fjy cf0cb3b703 [maven-release-plugin] prepare for next development iteration 2013-07-25 16:55:28 -07:00
fjy 8297f068c3 [maven-release-plugin] prepare release druid-0.5.21 2013-07-25 16:55:20 -07:00
fjy f68a68112f [maven-release-plugin] prepare for next development iteration 2013-07-25 16:21:59 -07:00
fjy b6e8d11aec [maven-release-plugin] prepare release druid-0.5.20 2013-07-25 16:21:49 -07:00
fjy 217894d44a Merge branch 'master' into worker-resource 
Conflicts:
	pom.xml
 2013-07-25 16:18:42 -07:00
Gian Merlino 4dd42c7177 RemoteTaskActionClient: Log service uri when submitting action 2013-07-25 14:48:25 -07:00