Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
7,322 Commits 111 Branches 503 Tags 721 MiB
Commit Graph

442 Commits

Author SHA1 Message Date
Xavier Léauté f847a46ccf fix typo 2014-07-22 15:53:50 -07:00
fjy 8f2fc595ee allow router to override selection based on config 2014-07-22 10:12:02 -07:00
fjy bf71d079f3 fix #636 2014-07-16 15:20:20 -07:00
fjy 534acc9eab fix mm 2014-07-15 22:50:38 -07:00
fjy 935b642b2c fix overlord binding problems 2014-07-15 16:13:11 -07:00
jisookim0513 c4397bd65a fixed merge conflicts 2014-07-07 11:26:11 -07:00
fjy 4c40e71e54 address cr 2014-06-19 14:48:46 -07:00
fjy a870fe5cbe inject column config 2014-06-19 14:47:57 -07:00
jisookim0513 00856f0fec add retry feature on broker side; modify QueryRunner inferface and tests 2014-06-18 16:55:33 -07:00
fjy 0bc1915067 Merge pull request #578 from metamx/new-guava 
Update guava, java-util, and druid-api
 2014-06-18 14:23:32 -06:00
fjy 5227b43050 fix more test breakage 2014-06-17 10:35:01 -07:00
fjy a63cda3281 Merge branch 'master' into new-guava 
Conflicts:
	server/src/main/java/io/druid/server/QueryResource.java
 2014-06-13 10:08:10 -07:00
Xavier Léauté c506d014bf fix link to docs 2014-06-04 12:55:48 -07:00
fjy bb4105ed1a fix broken standalone hadoop ingestion 2014-06-04 09:23:46 -07:00
fjy c97768b27d Merge branch 'master' of github.com:metamx/druid into new-guava 2014-06-04 09:12:59 -07:00
nishantmonu51 d8338fc51d fix backwards compatibility 2014-06-04 16:40:06 +05:30
fjy 77ec4df797 update guava, java-util, and druid-api 2014-06-03 13:43:38 -07:00
fjy 49a8dc9e3e fix backwards incompatiable change with broker.balancer config 2014-05-28 13:27:43 -07:00
fjy b6fb4245aa Merge branch 'master' into new-schema 
Conflicts:
	indexing-hadoop/src/main/java/io/druid/indexer/HadoopDriverConfig.java
	indexing-hadoop/src/main/java/io/druid/indexer/HadoopDruidIndexerConfig.java
	indexing-hadoop/src/main/java/io/druid/indexer/HadoopDruidIndexerConfigBuilder.java
	pom.xml
	server/src/main/java/io/druid/segment/realtime/RealtimeManager.java
	server/src/main/java/io/druid/segment/realtime/firehose/EventReceiverFirehoseFactory.java
 2014-05-06 14:32:51 -07:00
fjy 544c0f08f6 Merge pull request #459 from metamx/move-firehose 
Move the event receiver firehose such that standalone realtime nodes can use it
 2014-05-05 17:09:43 -06:00
fjy e90eb4b83a add custom tier selector 2014-04-28 15:31:15 -07:00
fjy 9de170540b fix bug with selection in cli 2014-04-28 14:00:53 -07:00
fjy 3c3daf452e Smarter server selection at the router level 2014-04-28 13:58:10 -07:00
fjy 171d20d52d Merge branch 'move-firehose' of github.com:metamx/druid into move-firehose 2014-04-25 14:13:19 -07:00
fjy eef034ca7e Merge branch 'master' of github.com:metamx/druid into move-firehose 2014-04-25 14:13:08 -07:00
fjy 76e0a48527 Merge branch 'master' into new-schema 
Conflicts:
	indexing-hadoop/src/main/java/io/druid/indexer/DbUpdaterJob.java
	indexing-hadoop/src/test/java/io/druid/indexer/HadoopDruidIndexerConfigTest.java
	indexing-service/src/main/java/io/druid/indexing/common/task/HadoopIndexTask.java
	server/src/main/java/io/druid/segment/realtime/plumber/RealtimePlumber.java
	server/src/main/java/io/druid/segment/realtime/plumber/RealtimePlumberSchool.java
 2014-04-25 14:03:28 -07:00
Gian Merlino 8d650ae131 PullDependencies changes. 
- Pull default Hadoop version by default.
- Allow additional coordinates on the command line.
 2014-04-22 14:33:53 -07:00
fjy c00fb1d08e Merge pull request #445 from metamx/hadoop-version-update 
Hadoop version update
 2014-04-17 19:16:44 -06:00
fjy a87213e15f point code docs to latest instead of a version 2014-04-16 10:50:39 -07:00
fjy 51e2da9d8d use a separate annotation for the router, less confusing 2014-04-16 10:08:58 -07:00
fjy 24a0268036 cleanup code in router and add forwarding for get requests 2014-04-16 09:36:18 -07:00
fjy 508cf6a4f7 Merge branch 'master' of github.com:metamx/druid 2014-04-15 14:23:58 -07:00
fjy 9b06b6aa58 misc fixes for router 2014-04-15 14:23:54 -07:00
Xavier Léauté 077dd7c589 enable cache monitor on computes by default 2014-04-14 16:11:12 -07:00
fjy 88c592ccaa add more meaningful request logging 2014-04-10 10:37:25 -07:00
nishantmonu51 ea8b2dffd5 Add chatHandlerResource 2014-04-09 14:25:18 +05:30
fjy 7db756f819 prepare for next release 2014-04-08 16:25:59 -07:00
fjy a4df23f092 prepare for next release 2014-04-08 11:42:59 -07:00
nishantmonu51 c6f4b344cb add bindings for chat handler provider 2014-04-08 11:56:23 +05:30
fjy 4ef2bbbfc1 prepare for next release 2014-04-07 14:37:02 -07:00
fjy 2e9579a146 prepare for next release 2014-04-07 12:43:30 -07:00
fjy 1843316db6 commonalize event receiver firehose 2014-04-03 20:46:36 -07:00
fjy 5d8d506fe8 fix commented out filters 2014-04-03 17:06:28 -07:00
fjy d9062d91fa fix endpoint bugs and more docs 2014-04-03 17:01:33 -07:00
fjy b0bf08554a fix broken build 2014-04-02 22:22:42 -07:00
fjy e53d107472 revert broken cache 2014-04-02 22:12:22 -07:00
fjy ea9f31bbe0 learn to math and prepare for next release 2014-04-02 21:31:17 -07:00
fjy f185d215e1 fix jetty default thread pool size for jetty 9 and prepare for next release 2014-04-02 21:01:07 -07:00
fjy fc94b847c3 prepare for next release 2014-04-02 17:31:19 -07:00
fjy 7547e989f0 Merge pull request #456 from metamx/processing-threads-default 
more sensible defaults
 2014-04-02 18:25:43 -06:00
Xavier Léauté 2eb9493010 fix documentation link 2014-04-02 15:27:49 -07:00
fjy 1866cbad1b remove unneeded comment 2014-04-01 18:15:51 -07:00
fjy eaec0e3ab2 cleanup 2014-04-01 17:56:25 -07:00
fjy 6952eb7878 asynch servlet for query forwarding 2014-04-01 17:51:28 -07:00
nishantmonu51 06b53f02a8 upgrade default version in CluHadoopIndexer and merge from master 2014-03-27 03:37:45 +05:30
fjy 5169249ed5 Merge pull request #429 from metamx/compute-populate-cache 
Enable historical node to populate cache
 2014-03-25 12:03:26 -06:00
fjy eec07ad45f Merge branch 'master' of github.com:metamx/druid 
Conflicts:
	services/src/main/java/io/druid/cli/CliHadoopIndexer.java
 2014-03-25 10:04:14 -07:00
fjy b6dd2fae64 update some comments on standalone hadoop indexer 2014-03-25 10:03:40 -07:00
nishantmonu51 fe7a184096 review comments 2014-03-25 17:02:36 +05:30
nishantmonu51 0ba72ffb2b merge from master 2014-03-25 15:43:05 +05:30
nishantmonu51 6a34c09e05 Merge branch 'master' into compute-populate-cache 2014-03-25 13:23:00 +05:30
fjy 5ebb2d27e3 fix hadoop 2014-03-24 18:43:31 -07:00
fjy 75ec8bed9a address code review comments 2014-03-24 14:53:30 -07:00
fjy fcd7522596 Merge branch 'master' into router-node 2014-03-24 13:43:52 -07:00
fjy 55169175de prepare for next release 2014-03-21 11:05:54 -07:00
nishantmonu51 69e9b7a6c5 merge changes from master 2014-03-20 20:23:58 +05:30
fjy ab2524ba0a add router node 2014-03-18 13:44:29 -07:00
fjy 804a4051fa prepare for next rel 2014-03-18 08:10:59 -07:00
fjy 1d1ec1d855 clean up some defaults and prepare for next release 2014-03-17 21:41:34 -07:00
fjy 2adcf07f5f Merge branch 'master' into new-schema 
Conflicts:
	indexing-hadoop/src/main/java/io/druid/indexer/DetermineHashedPartitionsJob.java
	indexing-service/src/main/java/io/druid/indexing/common/task/RealtimeIndexTask.java
	indexing-service/src/test/java/io/druid/indexing/common/task/TaskSerdeTest.java
	processing/src/test/java/io/druid/segment/TestIndex.java
	server/src/main/java/io/druid/segment/realtime/RealtimeManager.java
	server/src/main/java/io/druid/segment/realtime/plumber/RealtimePlumberSchool.java
 2014-03-17 10:59:31 -07:00
nishantmonu51 468e8e0b31 Memcache changes 2014-03-17 18:54:30 +05:30
fjy f1da044cfa fix another breaking endpoint in druid coordinator 2014-03-14 15:15:08 -07:00
fjy f525361c3d fix broken tests and prepare for next release 2014-03-14 15:05:07 -07:00
fjy f0fdfbb2f7 fix broken constant post agg 2014-03-14 14:54:52 -07:00
fjy b3fb2fea9c prepare for next release 2014-03-14 10:57:43 -07:00
nishantmonu51 04f3d0a13a fix cli hadoop indexer 
* run determine partitions job using CLI
 2014-03-11 16:36:19 +05:30
fjy ae2dc6506c fix broken granularity 2014-03-10 15:03:43 -07:00
fjy 243af486cf prepare for next release 2014-03-10 13:24:53 -07:00
fjy e1fac64298 prepare for next release 2014-03-06 12:53:05 -08:00
fjy 7b0b90a860 Merge branch 'master' of github.com:metamx/druid into new-schema 2014-02-20 16:12:31 -08:00
fjy 5d2367f0fd unit tests pass at this point 2014-02-20 15:52:12 -08:00
fjy ab83cba790 Merge pull request #393 from metamx/group-by-improvements 
Group by memory usage improvements
 2014-02-19 19:52:37 -07:00
fjy fb93448e9d prepare for next release because my last one failzored out of bad internetz 2014-02-14 15:22:27 -08:00
fjy 1a8f9f2e12 prepare for next release 2014-02-14 15:07:44 -08:00
fjy a986166b9c prepare for next release 2014-02-12 20:01:05 -08:00
nishantmonu51 3302cff6db groupBy improvements 2014-02-12 00:51:10 +05:30
fjy 2497bd2e85 prepare for next release 2014-02-07 13:04:34 -08:00
fjy c8b350270f prepare for next release 2014-02-07 11:54:56 -08:00
fjy 8edf0cf136 more backwards compatability fix 2014-01-26 21:27:34 +08:00
fjy 158a725985 fix backwards compat issues 2014-01-26 20:07:09 +08:00
fjy f3e1cbc423 fix according to code review 2014-01-21 16:26:28 -08:00
fjy e2ea2ec7fd Merge branch 'master' into az 
Conflicts:
	server/src/main/java/io/druid/server/coordinator/DruidCoordinator.java
	server/src/main/java/io/druid/server/http/CoordinatorResource.java
	server/src/test/java/io/druid/client/DirectDruidClientTest.java
 2014-01-20 15:10:13 -08:00
fjy 215584891e Merge branch 'master' into moar-endpoints 2014-01-20 13:06:45 -08:00
fjy 6404967994 fixes 2014-01-20 13:05:05 -08:00
fjy 8d01a46d02 actually register resources 2014-01-17 18:01:09 -08:00
fjy f898c29e20 fix batch indexing and prepare for next release 2014-01-17 15:52:04 -08:00
fjy 0ceda034cd fix dependency issue and prepare for next release 2014-01-14 17:34:02 -08:00
fjy 187bb84b45 prepare for next release 2014-01-14 16:07:05 -08:00
fjy 9184906c39 prepare for next release 2014-01-14 09:54:42 -08:00
fjy e8d61ffe99 prepare for next release 2014-01-13 18:02:24 -08:00
fjy f123a14373 Merge branch 'master' into az 
Conflicts:
	docs/content/Realtime.md
	server/src/main/java/io/druid/client/selector/ServerSelector.java
 2014-01-09 19:09:38 -08:00
Xavier Léauté 632d6b0a78 make server selection configurable, add back random server selection 2014-01-09 14:37:33 -08:00
fjy f44509b530 queries working at this point, tests pass, still need more tests and fix rules interface 2014-01-07 16:24:53 -08:00
fjy e11952f3b6 bridge is kinda working for mastery stuff right now, tests pass 2014-01-06 17:01:50 -08:00
Xavier Léauté 95d92915bf more tests for Initialization + dependency cleanup 2013-12-30 16:31:10 -08:00
Gian Merlino bbe1b82347 Update versions 2013-12-20 12:26:27 -08:00
Gian Merlino 4bc8ff4d6a Update versions 2013-12-20 11:19:36 -08:00
fjy fba6caf7fd Merge pull request #326 from metamx/versions 
StatusResource
 2013-12-18 22:44:50 -08:00
Gian Merlino fc4dd51194 Update versions 2013-12-18 15:33:25 -08:00
fjy d112b2d211 prepare for next release 2013-12-17 17:47:04 -08:00
fjy 0051877f84 fix status endpoint 2013-12-17 17:46:16 -08:00
fjy 07875dd30b prepare for next release 2013-12-17 16:28:11 -08:00
fjy 69f33b0acd remove commented line 2013-12-17 16:24:15 -08:00
fjy bbb2754a7c fix redirects in druid 2013-12-17 16:22:36 -08:00
fjy f2241c8885 prepare for next release 2013-12-16 16:16:00 -08:00
Xavier Léauté ac2ca0e46c separate move and archive tasks 2013-12-16 14:00:55 -08:00
Xavier Léauté a417cd5df2 add archive task 2013-12-16 13:59:15 -08:00
fjy 84233238b1 Merge branch 'master' of github.com:metamx/druid 2013-12-16 12:01:55 -08:00
fjy 0998d835b0 fix redirection issues with master and overlord 2013-12-16 12:01:52 -08:00
Gian Merlino 7759ed4517 Coordinator: Link service announcement to leadership 2013-12-16 11:28:28 -08:00
Himadri Singh 3752122a7b implementing comments from review 2013-12-16 15:16:28 +05:30
fjy 87b83bceb1 fix task storage config serde and prepare for next release 2013-12-13 16:55:22 -08:00
fjy 01f9c1df31 fix broken task storage config and prepare for next release 2013-12-13 16:45:32 -08:00
fjy 9a5fd6b0e9 prepare for next release 2013-12-13 16:18:34 -08:00
Gian Merlino 600dc7546f Configurability of recency threshold 2013-12-13 16:02:54 -08:00
fjy 407f8caf94 remove server monitor from list of default monitors 2013-12-13 09:04:32 -08:00
fjy 25ebba2c38 fix default start delay of indexing service and prepare for next release 2013-12-12 18:51:25 -08:00
fjy 19eafe0e96 prepare for next release 2013-12-12 17:26:50 -08:00
fjy a58f90a689 prepare for next release 2013-12-12 14:51:14 -08:00
Gian Merlino 70c153592f CliPeon: Fix local mode 2013-12-12 14:22:57 -08:00
Gian Merlino c4b8c8bc6f Rework indexing service internals to hopefully be more reliable. 
The TaskQueue directly manages the TaskRunner. The main management loop runs
periodically and checks that the runner is doing reasonable things. If not, it
attempts to adjust the runner. The management loop also runs on-demand when a
task is added to keep task assignment relatively low latency. The TaskConsumer
is no longer necessary and so it no longer exists.

Task interval locks are handled differently. Instead of some tasks acquiring
locks at runtime and some tasks having implicit fixed lock intervals, all tasks
ask for locks explicitly. This occurs either in "isReady" (which runs on the
overlord) or in "run" (which runs on the peon).

Other changes:
- The TaskQueue is attached to the leader lifecycle, instead of global
- The TaskLockbox is able to sync itself from storage and is no longer
  bootstrapped by the TaskQueue.
- RemoteTaskRunner does not clean up zk paths until asked to. This will
  prevent deletion of statuses that have not yet been committed.
- Added retries on DbTaskStorage operations.
- Removed SpawnTasksAction (no more subtasks)
- Removed obsolete EventReceiverFirehose configs
- Removed obsolete OldOverlordResource
- Removed TaskStorageQueryAdapter methods related to subtasks
 2013-12-11 15:05:16 -08:00
Gian Merlino f3cfd1d781 Introduce FileTaskLogs, and move TaskLogs module from server to indexing-service 2013-12-10 17:39:43 -08:00
fjy 6cc0860657 fix s3 module problems and prepare for next release 2013-12-09 20:05:55 -08:00
fjy ab1f02b2bc fix version brokenness and prepare for next release 2013-12-09 18:38:22 -08:00
fjy 976affb3af prepare for next release 2013-12-09 18:28:31 -08:00
fjy 303f6ff334 fix worker config setup problems 2013-12-09 18:25:29 -08:00
fjy 8aba4ad3d2 Merge pull request #311 from metamx/versions 
Add versions to static page
 2013-12-09 17:18:16 -08:00
Himadri Singh b5f6dbc32f Code refactoring, one place! 2013-12-10 02:09:41 +05:30
Himadri Singh 8117fff4e2 Some minor fixes 2013-12-10 00:39:10 +05:30
fjy fbad1e2d7a fix convert properties with infoPath and prepare for next release 2013-12-04 18:55:35 -08:00
Himadri Singh 32200b9b41 removing debug line. 2013-12-04 11:00:47 +05:30
Himadri Singh 47278ad109 Merge relevant changes from 'druid-version-info' 2013-12-04 10:21:19 +05:30
fjy 05f72601c4 fix request time metric and prepare for next release 2013-12-02 14:20:59 -08:00
fjy ddb6e83f17 fix realtime default and prepare for next deploy 2013-11-25 17:53:17 -08:00
fjy fbb65988bd make query resource eagerly instantiate and prepare for next release 2013-11-20 11:58:25 -08:00
fjy c234dafee1 prepare for next release 2013-11-20 10:43:37 -08:00
fjy 84a6262e04 prepare for next release 2013-11-20 10:27:09 -08:00
fjy e7167ae00a add new query resource 2013-11-20 10:02:43 -08:00
fjy 984e24c1bb fix broken links in docs, prepare for next release 2013-11-19 09:57:56 -08:00
fjy 67a93ccd4d fix broken validation query config port 2013-11-18 15:53:49 -08:00
fjy 1820110818 clean up docs and fix server monitor double emitting metrics 2013-11-18 15:37:30 -08:00
fjy a86d3af17e fix docs 2013-11-14 18:29:27 -08:00
fjy 346cf0e04c fix out of order in urls for hadoop classpath 2013-11-14 18:18:39 -08:00
fjy 5d00e4075e prepare for next release 2013-11-14 15:06:26 -08:00
fjy 6b41681424 fix ordering of urls, ugh, need to write more tests 2013-11-14 15:06:07 -08:00
fjy 23ebca6d32 more hadoop dependency hell, getting the right urls to hadoop is hard 2013-11-14 14:29:42 -08:00
fjy 61367c6b13 prepare for next release 2013-11-14 13:16:38 -08:00
fjy c51eed060f actually use a class loader pull out dependencies from extension modules 2013-11-14 11:52:21 -08:00
fjy fcf0c6ce06 change the classpath ordering for batch processing and prepare for next release 2013-11-14 10:52:51 -08:00
fjy 60300bdad0 fix doc version 2013-11-13 13:34:57 -08:00
fjy c1009f0872 prepare for next release 2013-11-13 11:36:50 -08:00
fjy d0f990129a fix docs 2013-11-13 10:59:03 -08:00
fjy 9fe9ca3163 the emitter library is broken and also prepare for next release 2013-11-08 15:54:16 -08:00
fjy 822a30658d fix broken http post emitter and prepare for next release 2013-11-08 14:01:16 -08:00
fjy 64b93bf448 fix broken autoscaling and prepare for next release 2013-11-08 11:29:42 -08:00
fjy d70bcc5657 repair broken pom 2013-11-07 23:27:06 -08:00
fjy a0eedf7342 fix broken docs 2013-11-07 23:11:30 -08:00
fjy d1c3176bf5 update docs ver 2013-11-07 19:47:36 -08:00
fjy a049b42674 fix an issue with task tables not getting created automatically and prepare for next release 2013-11-07 18:01:35 -08:00
fjy d49554185a fix broken default table names 2013-11-07 17:38:00 -08:00
fjy 6b573c76f1 more fixes 2013-11-07 14:40:45 -08:00
fjy 33561031a2 update docs version to match next release 2013-11-05 15:30:41 -08:00
Xavier Léauté dce6e7e0eb hack to support remote repository authentication 2013-11-01 09:33:07 -07:00
Xavier Léauté ef88f37bbe add tools pull-deps to cli 2013-10-30 18:16:27 -07:00
fjy 21235ae533 fix configuration conversion bugs and docs and hdfs configs 2013-10-29 17:34:24 -04:00
Gian Merlino 2e9c46867f Fixes for indexing service. 
- Create IndexingServiceFirehoseModule so firehoses can be loaded by all mains
- Fix implicit lock acquisition in AbstractTask
 2013-10-18 11:14:33 -07:00
fjy a1c09df17f make the hadoop index task work again 2013-10-16 09:45:17 -07:00
fjy 9796a40b92 port docs over to 0.6 and a bunch of misc fixes 2013-10-11 18:38:53 -07:00
fjy a9a723bd11 clean up poms, add a new loading your own data tutorial, add new validation, clean up logs 2013-10-09 15:42:39 -07:00
fjy 4e509d1d09 Merge branch 'master' into is-docs 2013-10-09 14:05:10 -07:00
cheddar c47fe202c7 Fix HadoopDruidIndexer to work with the new way of things 
There are multiple and sundry changes in here.

First, "HadoopDruidIndexer" has been split into two pieces, (1) CliHadoop which pulls the hadoop version and builds up the right classpath with the proper hadoop version to run the indexer and (2) CliInternalHadoopIndexer which actually runs the indexer.

In order to work around a bunch of jets3t version conflicts with Hadoop and Druid, I needed to extract the S3 deep storage stuff into its own module.  I then also moved the HDFS stuff into its own module so that I could eliminate the dependency on Hadoop for druid-server.

In doing these changes, I wanted to make the extensions buildable with only the druid-api jar, so a few other things had to move out of Druid and into druid-api.  They are all API-level things, however, so they really belong in druid-api instead.

Lastly, I removed the druid-realtime module and put it all in druid-server.
 2013-10-09 15:15:44 -05:00
fjy 4ec4b8e024 rewrite indexing service docs 2013-10-08 16:34:58 -07:00
fjy ac330f72bb first set of changes to standarize the naming convention we use in druid 2013-10-03 16:36:48 -07:00
fjy bf157ea8ef fix the example assembly and have better error logging for rabbitmq 2013-10-03 10:54:25 -07:00
fjy 17874eeb67 make the CliPeon actually able to run on its own 2013-10-02 15:55:10 -07:00
fjy bc8db7daa5 1) make chat handler resource work again 
2) add more default configs
3) make examples work again
 2013-10-02 14:22:39 -07:00
Gian Merlino 384dcda7e4 Chat handlers still don't work, but, they're closer maybe. 2013-10-01 17:45:53 -07:00
fjy 5d0d71250b fix chat handler resources not correctly registering themselves 2013-10-01 11:25:39 -07:00
fjy d6445cd8f3 remove imports 2013-10-01 09:42:28 -07:00
fjy 53698a135a add interface to new firehose as per code review comments 2013-09-30 18:00:59 -07:00
fjy f55a5199b1 add a firehose module to remove so much copy and pasted code 2013-09-30 16:29:20 -07:00
fjy ed9e0cf9f6 add a local firehose for indexing local files 2013-09-30 16:03:26 -07:00
fjy 4a7a28a7f6 Merge branch 'master' into fix-config 2013-09-27 17:17:46 -07:00
Gian Merlino dc5dab8747 Fixes for property conversion, firehose registration, and the indexing service 2013-09-27 17:09:59 -07:00
fjy a79ad7bab4 make dynamic master resource configuration work again 2013-09-27 15:00:40 -07:00
fjy 0b04325ee8 fix things up according to code review comments 2013-09-27 10:17:45 -07:00
fjy e404295c1f make indexing service work 2013-09-26 17:44:21 -07:00
fjy 8bc56daa66 fix things up according to code review comments 2013-09-26 11:35:45 -07:00
fjy 87259321b6 port hadoop druid indexer to new guice framework 2013-09-26 11:04:42 -07:00
fjy 19276f6bad fix spacing issues and other code review comments 2013-09-24 14:34:09 -07:00
fjy be5bb7f2eb fix lifecycle startup/stop ordering problem with discovery module and make druid able to load local extensions 2013-09-24 14:20:56 -07:00
fjy 1ff04412a2 clean up realtime module and fix breakage in broker paths 2013-09-23 16:26:05 -07:00
cheddar 1750b702d7 Add missed com.metamx.druid.emitter.period propery to ConvertProperties 2013-09-23 16:46:37 -05:00
cheddar cc632cdb30 Make Jetty Server lazy 
JettyServerModule had been eagerly instantiating the Server object, which was causing things that didn't care about an HTTP interface to all of a sudden require host and port parameters.  The change makes the JettyServerModule only setup the bindings without eagerly instantiating the Jetty Server.  Each cli needs to register the Server class with the Lifecycle in order to make it actually get used.
 2013-09-23 16:02:24 -05:00
fjy dc8a119787 fix broken unit tests are a result of the last merge 2013-09-23 12:56:01 -07:00
cheddar fefc71e34f Refix the things that were lost in the merge. 2013-09-20 18:01:08 -05:00
cheddar b13ae3f405 Merge branch 'master' into fix-extensionization 
Conflicts:
	services/src/main/java/io/druid/cli/CliBroker.java
	services/src/main/java/io/druid/cli/CliCoordinator.java
 2013-09-20 17:49:13 -05:00
cheddar 5712b29c8c Fix issues with bindings and handling extensions 
The way the Guice bindings were setup previously, each process only had bindings
for the things it cared about.  This became problematic when adding extension modules
that bound everything that they could possibly need expecting that the processes would
only instantiate what they actually do need.  Guice tries to fail-fast and verifies that all
 bindings exist before it does anything, which is a problem because the extension bind
 some objects that don't necessarily have all of their dependencies bound in all processes.

The fix for this is to build a single Injector with all bindings in it and let each of the
 processes only load the things that they care about.  This also requires the use of
 Module overrides and other such interesting things, which are node done.

 In doing the fix, I also swapped out the way that the DataSegmentPusher/Puller stuff is bound, as well as made the Cassandra stuff fail if its settings are not provided.  This all of a sudden made all of the things require Cassandra's settings, so I migrated the Cassandra deep storage stuff into its own module.

 In doing these changes, I also discovered that some properties weren't properly converting for the ConvertProperties command (specifically, the properties related to data segment loading and pushing), so I fixed that.
 2013-09-20 17:45:01 -05:00
fjy f26e13c61a fix broken http endpoints 2013-09-19 17:33:58 -07:00
cheddar 6bfe2310bb Merge branch 'master' into guice 
Conflicts:
	client/pom.xml
	client/src/main/java/com/metamx/druid/query/segment/LegacySegmentSpec.java
	indexing-common/pom.xml
	indexing-service/src/main/java/com/metamx/druid/indexing/common/TaskToolbox.java
	indexing-service/src/main/java/com/metamx/druid/indexing/common/TaskToolboxFactory.java
	indexing-service/src/main/java/com/metamx/druid/indexing/common/task/RealtimeIndexTask.java
	indexing-service/src/main/java/com/metamx/druid/indexing/worker/executor/ExecutorNode.java
	indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/RemoteTaskRunnerTest.java
	indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/TaskLifecycleTest.java
	indexing-service/src/test/java/com/metamx/druid/indexing/coordinator/TaskQueueTest.java
	realtime/src/main/java/com/metamx/druid/realtime/RealtimeNode.java
	realtime/src/main/java/com/metamx/druid/realtime/firehose/IrcFirehoseFactory.java
	realtime/src/main/java/com/metamx/druid/realtime/firehose/WikipediaIrcDecoder.java
	realtime/src/main/java/com/metamx/druid/realtime/plumber/RealtimePlumberSchool.java
	server/src/main/java/com/metamx/druid/BaseServerNode.java
	server/src/main/java/com/metamx/druid/http/ComputeNode.java
	server/src/main/java/com/metamx/druid/index/v1/IncrementalIndexStorageAdapter.java
	server/src/main/java/com/metamx/druid/query/group/GroupByQueryEngine.java
 2013-09-18 16:46:54 -05:00
fjy f7df951610 broker needs to announce itself in service discovery 2013-09-18 09:23:46 -07:00
fjy cabae7993d port over multi threaded realtime and also fix broken realtime nodes that can't start up 2013-09-16 16:03:47 -07:00
fjy 5a8b71ddec fix how druid services is assembled 2013-09-13 18:41:33 -07:00
cheddar e0738b3b22 1) Add tool to convert properties from the old settings to the new ones. 2013-09-12 17:59:23 -05:00
cheddar a2dcc45a8e 1) Remove SingleSegmentLoader and replace with OmniSegmentLoader 2013-09-12 11:47:03 -05:00
cheddar 3e4a4f5566 1) Adjustments to allow for the addition of new Runnables via modules. Modules should implement CliCommandCreator and add their stuff there. 2013-09-11 17:41:20 -05:00
cheddar 6c9a107356 1) remove duplicate package initialization.initialization 2013-09-09 17:02:57 -05:00
Russell Jurney a11eae4144 Edited assembly.xml to exclude the selfcontained jars, and edited path of run_example_server to use the lib/ directory instead of selfcontained jars, thus reducing the size of the assembly tarball from 150MB to 50MB. 2013-09-04 12:53:38 -07:00
cheddar 740c70ed06 1) It's good when unit tests actually pass... 2013-08-31 17:27:15 -05:00
cheddar 5ab671050e No more com.metamx.druid, it is now all io.druid! 2013-08-30 19:42:12 -05:00
cheddar bd0756e360 More stuff moved, things still compiling and tests still passing. Yay! 2013-08-30 18:58:35 -05:00
cheddar 56e2b956d0 OMG!!! A lot of stuff has been moved. Modules have been created and destroyed, but everything is compiling and unit tests are passing, OMFG this is awesome.! 2013-08-30 18:21:04 -05:00
cheddar 3617ac17fc 1) Eliminate ExecutorMain and have it run using the new Main! 2013-08-27 14:11:05 -05:00
cheddar 269997dc94 1) ExecutorNode is working, except for the running of the task. Need to adjust it to be able to run a task and then everything will be wonderful 2013-08-26 18:08:41 -05:00
cheddar 55dbda2046 1) Worker appears to be running! It's also now known as the MiddleManager 2013-08-23 17:59:48 -05:00
cheddar b897c2cb22 1) IndexCoordinator appears to work as the CliOverlord now, yay! 2013-08-23 14:11:34 -05:00
fjy 6a8c160740 update code according to code review 2013-08-22 10:46:05 -07:00
fjy 85ee8bb267 port realtime to guice 2013-08-13 17:08:45 -07:00
cheddar 51f648ec11 1) Rename QueryToolChestModule back 2013-08-06 12:05:56 -07:00
cheddar 69f7831d82 1) Rename QueryToolChestModule because there was a case sensitivity issue, grrr 2013-08-06 12:04:04 -07:00
cheddar 9f71d42314 1) Broker works with Guice 
2) Extract ServerViewModule to handle various ServerView thingies
3) Extract QueryToolChestModule and QueryRunnerFactoryModule to reuse code for configuration of Query stuff
4) Extract QueryJettyServerInitializer to reuse between Historical and Broker nodes
5) Remove ClientMain, BrokerMain and BrokerNode
 2013-08-06 12:01:52 -07:00
cheddar 293c0e8418 1) Clean up the servlet registration stuff, resource registrations on CliHistorical weren't working, they do now. 2013-08-05 13:58:45 -07:00
cheddar d64bc2f762 1) Eliminate Old mains() that aren't needed anymore. 2013-08-05 13:33:43 -07:00
cheddar 9e78bb38f5 Merge branch 'master' into guice 
Conflicts:
	client/src/main/java/com/metamx/druid/QueryableNode.java
	client/src/main/java/com/metamx/druid/client/ServerInventoryView.java
	client/src/main/java/com/metamx/druid/coordination/SingleDataSegmentAnnouncer.java
	client/src/main/java/com/metamx/druid/initialization/CuratorDiscoveryConfig.java
	client/src/main/java/com/metamx/druid/query/MetricsEmittingExecutorService.java
	indexing-hadoop/src/test/java/com/metamx/druid/indexer/HadoopDruidIndexerConfigTest.java
	indexing-service/src/main/java/com/metamx/druid/indexing/common/TaskToolbox.java
	indexing-service/src/main/java/com/metamx/druid/indexing/coordinator/http/IndexerCoordinatorNode.java
	indexing-service/src/main/java/com/metamx/druid/indexing/worker/executor/ExecutorNode.java
	indexing-service/src/main/java/com/metamx/druid/indexing/worker/http/WorkerNode.java
	pom.xml
	server/src/main/java/com/metamx/druid/coordination/ServerManager.java
	server/src/main/java/com/metamx/druid/coordination/ZkCoordinator.java
	server/src/main/java/com/metamx/druid/db/DatabaseRuleManager.java
	server/src/main/java/com/metamx/druid/db/DatabaseSegmentManager.java
	server/src/main/java/com/metamx/druid/http/ComputeNode.java
	server/src/main/java/com/metamx/druid/http/MasterMain.java
	server/src/main/java/com/metamx/druid/loading/SegmentLoaderConfig.java
	server/src/main/java/com/metamx/druid/loading/SingleSegmentLoader.java
	server/src/main/java/com/metamx/druid/master/DruidMaster.java
 2013-08-01 16:42:47 -07:00
cheddar 019bb5d453 1) Another whole bunch of changes to annotate things and create Modules and bind stuff. But OMFG, the compute node actually appears to be working! 
2) The compute node works with Guice
3) The compute node fires up with Guice and appears to work
4) Did I mention that the compute node, now called historical node, fires up with Guice and appears to work?
 2013-08-01 15:28:08 -07:00
cheddar f4fc8b2f6e 1) A whole boondoggle of changes to Guicify the instantiation of things for the Historical nodes 
2) Add GuiceInjectableValues that supports @JacksonInject resolution via Guice
3) Refactor RequestLogger stuff and setup polymorphic deserialization method of creating objects
 2013-07-31 18:36:47 -07:00
cheddar 97fb971f84 1) Start the journey of making the historical nodes work with Guice. Doesn't work yet, but at least it builds. 2013-07-30 11:58:05 -07:00
cheddar 2b587cba85 1) Introduce airlift/airline for CLI stuff and add a Coordinator CLI verb to make it all happiness 2013-07-29 15:14:20 -07:00
Russell Jurney 28af252bca Include multiple configurations, one per node type, in assembly package for examples. 2013-07-01 16:36:30 -07:00
Steve Harris 5a9bf98b6c Some changes to automatically build a kit with samples. 2013-05-04 20:04:16 -07:00
cheddar 12fe145cc8 1) Setup assembly to grab all dependencies -- it's broken right now, http://jira.codehaus.org/browse/MASSEMBLY-646 2013-05-02 16:35:06 -05:00
cheddar d1a8b93b3f 1) druid-services -> services 2013-05-01 17:37:38 -05:00