Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
2,489 Commits 113 Branches 504 Tags 726 MiB
Commit Graph

657 Commits

Author SHA1 Message Date
cheddar f4fc8b2f6e 1) A whole boondoggle of changes to Guicify the instantiation of things for the Historical nodes 
2) Add GuiceInjectableValues that supports @JacksonInject resolution via Guice
3) Refactor RequestLogger stuff and setup polymorphic deserialization method of creating objects
 2013-07-31 18:36:47 -07:00
fjy e2b5cd6067 Merge branch 'master' of github.com:metamx/druid into worker-resource 
Conflicts:
	indexing-service/src/main/java/com/metamx/druid/indexing/worker/executor/ExecutorNode.java
 2013-07-31 15:23:13 -07:00
fjy f76540c16c move reference counting classes out of server manager and more tests for concurrency 2013-07-31 15:14:39 -07:00
Dhruv Parthasarathy dd689ca828 changed urls 2013-07-31 11:43:24 -07:00
fjy 5d1fe50703 additional ut for reference counting 2013-07-30 17:55:57 -07:00
fjy 404b84823e reference count to make sure we arent accessing any instances of a closed mmapped segment 2013-07-30 17:36:56 -07:00
cheddar 843b6650a2 Merge pull request #206 from metamx/peontimeout 
Add a timeout in LoadQueuePeon to fail an assign if it takes too long
 2013-07-30 15:08:46 -07:00
fjy e7c6dd8112 get and reset failed assign count 2013-07-30 14:50:24 -07:00
fjy 35d8a82879 move the close method to the segment object 2013-07-30 12:24:01 -07:00
cheddar 97fb971f84 1) Start the journey of making the historical nodes work with Guice. Doesn't work yet, but at least it builds. 2013-07-30 11:58:05 -07:00
cheddar 1c5c2e0ef5 Merge pull request #205 from metamx/compress-curator 
Toggle between compressed and non compressed service discovery
 2013-07-30 11:33:43 -07:00
cheddar 66adef5d64 Merge pull request #179 from zanox/master 
Fix path used when storing segments to HDFS
 2013-07-30 11:13:29 -07:00
fjy f55e12040f fix according to code review 2013-07-29 18:39:14 -07:00
Dhruv Parthasarathy b8940b4524 added support for dynamicConfigs for Master 2013-07-29 18:16:57 -07:00
Dhruv Parthasarathy 4e4d582e07 formatted files 2013-07-29 18:10:08 -07:00
Dhruv Parthasarathy ac0f0afdbf turned Iterable to ArrayList, fixed potential null pointer in reservoir sampling 2013-07-29 18:07:49 -07:00
fjy 6a96c1fb76 fix according to code review comments 2013-07-29 17:56:19 -07:00
Dhruv Parthasarathy b0d9889c83 added web console for changing dynamicConfigs 2013-07-29 17:49:09 -07:00
fjy 091dce11c8 Add a timeout in LQP to fail an assign if it takes too long 2013-07-29 15:47:05 -07:00
fjy 50836798fa toggle between compressed and non compressed service discovery 2013-07-29 15:40:45 -07:00
fjy 1226190279 add the ability to close mmapped files 2013-07-29 15:22:34 -07:00
cheddar 4c9be81912 1) Pull try/catch check in the JettyServerInitializer out of the initializer. 2013-07-29 14:35:30 -07:00
cheddar 9cd26c70c0 1) Adjust the JettyServerModule and other things such that Jetty will actually serve the static artifacts and the various resources of the Master 2013-07-29 14:26:51 -07:00
Dhruv Parthasarathy 31ffed466c first set of changes 2013-07-29 11:39:24 -07:00
cheddar 6b9963b472 1) Move the @Json and @Smile annotations to com.metamx.druid.guice.annotations 
2) Changed ServiceAnnouncer to take a DruidNode object
3) Remove ServiceInstanceFactory interface and implementations
4) Add registrations to DiscoveryModule so that you can register nodes that should be announced on startup
5) Make the set of default Monitors configurable
 2013-07-26 14:03:51 -07:00
Dhruv Parthasarathy ba4a9c28a7 removed unecessary line 2013-07-26 10:52:50 -07:00
Dhruv Parthasarathy 1e4f64b119 redid testing 2013-07-26 10:50:06 -07:00
fjy 217894d44a Merge branch 'master' into worker-resource 
Conflicts:
	pom.xml
 2013-07-25 16:18:42 -07:00
Dhruv Parthasarathy 8e4fba431c moved profiler 2013-07-25 14:37:03 -07:00
Dhruv Parthasarathy 34a9525966 added tests for random sampler and changed abstract class to interface 2013-07-25 14:14:16 -07:00
Dhruv Parthasarathy 8ee71c8ceb added strategy pattern and RandomBalancerStrategy 2013-07-25 00:00:38 -07:00
Stefán Freyr Stefánsson b3530fb15d Adding a configuration property to disable setting an ACL on the segment pushed to S3. 2013-07-24 23:37:21 +00:00
Dhruv Parthasarathy a03dcc6429 cleaned up comments 2013-07-24 15:34:08 -07:00
Dhruv Parthasarathy bcb3e089d8 profiler added, commented out initialTotalCost normalization and normalizedInitialCost 2013-07-24 14:57:02 -07:00
Dhruv Parthasarathy 6dc8d09b00 partial changes 2013-07-23 17:25:36 -07:00
cheddar 02ffd805c8 1) Move DbConnector into its own module 
2) Move JacksonConfig stuff into its own module
3) Fix LogEmitterModule to inject properly with a Supplier<>
4) Set Guice to require explicit bindings, to protect from weirdness like was initially experienced with LoggingEmitterConfig
5) Master fires up and appears to all work with Guice!  Whoot!
 2013-07-23 14:55:36 -07:00
fjy 980b09d903 fix broken test 2013-07-23 10:19:26 -07:00
fjy ea7c7d1d66 Merge branch 'master' into batch-zk 
Conflicts:
	client/src/main/java/com/metamx/druid/QueryableNode.java
 2013-07-23 10:14:26 -07:00
fjy ef27810458 rename to fix case issue 2013-07-22 18:02:10 -07:00
fjy 3f97ac9253 add read to batch zk 2013-07-22 18:01:13 -07:00
cheddar 7504f3ab65 1) Adjust the Announcer to cleanup parent nodes that it creates. This stops it from leaving little turdlings lying around on zookeeper as nodes exit 
2) Rename *CuratorDataSegmentAnnouncer because neither of them depend directly on Curator.
 2013-07-22 16:31:01 -07:00
cheddar 85a35201b7 1) The Master compiles and runs with Guice bindings!!!! 
2) The service discovery stuff really needs to be reworked.  It's unhappy now.
 2013-07-19 16:29:54 -07:00
fjy 4e2f6380e6 Merge branch 'master' of github.com:metamx/druid 2013-07-18 13:25:08 -07:00
fjy 7d58a2d8db fix bug with enabling segments and improve replication logic 2013-07-18 13:24:52 -07:00
cheddar e906481704 Merge pull request #191 from metamx/s3-puller-retries 
S3 puller retries
 2013-07-17 17:07:57 -07:00
fjy 4e84fb5f03 better logic for exiting load rule assign if replicants cannot be created 2013-07-16 16:50:52 -07:00
fjy 49a80572b7 fix master speed issue with immutable copy of in druid server, better exception handling in moving and dropping segments 2013-07-16 13:24:43 -07:00
Gian Merlino 6258d77398 Retry s3 operations on non-wrapped IOExceptions 
Can happen if we get a socket related mishap while fetching an s3 object.
 2013-07-16 08:25:23 -07:00
Gian Merlino 320f1fe840 Move retryS3Operation to S3Utils 2013-07-16 08:07:14 -07:00
Gian Merlino 945bc9a370 Limited retries on s3 segment puller exceptions 
Useful for reducing spurious alerts due to general s3 flakiness.
 2013-07-16 08:03:47 -07:00
cheddar e892d14f77 1) Configuration fixes, slowly gonna make the MasterMain actually start up! 2013-07-09 16:16:56 -07:00
fjy cc35841e23 Merge branch 'batch-zk' into worker-resource 2013-07-08 11:04:05 -07:00
fjy 454f23d7c0 Merge branch 'master' of github.com:metamx/druid into batch-zk 2013-07-08 10:40:30 -07:00
Jan Rudert d5b4417280 Merge branch 'master' of github.com:metamx/druid 2013-07-08 11:50:15 +02:00
Jan Rudert 18ec081855 moved getHdfsStorageDir() to DataSegmentPusherUtil, extended test 2013-07-08 08:39:08 +02:00
cheddar b42c5743c9 Merge pull request #177 from nebrera/master 
Postgresql adaptation
 2013-07-05 16:42:57 -07:00
Jan Rudert 6f06d701d9 Building a correct segment path for HDFS 2013-07-04 16:00:20 +02:00
fjy be2f153985 Merge branch 'bug-fix' into worker-resource 2013-07-02 16:25:07 -07:00
fjy d3eb491155 Merge branch 'batch-zk' into worker-resource 
Conflicts:
	pom.xml
	server/src/main/java/com/metamx/druid/coordination/ZkCoordinator.java
 2013-07-02 16:24:59 -07:00
fjy e21aa41bdd Merge branch 'master' into worker-resource 
Conflicts:
	server/src/main/java/com/metamx/druid/coordination/ZkCoordinator.java
	server/src/main/java/com/metamx/druid/loading/SingleSegmentLoader.java
 2013-07-02 16:23:53 -07:00
fjy fde874ea56 fix according to code review 2013-07-02 16:11:12 -07:00
fjy 9818d28a26 Merge branch 'master' into bug-fix 2013-07-02 15:59:52 -07:00
fjy 5ac2d1db0b Merge branch 'master' of github.com:metamx/druid into batch-zk 
Conflicts:
	server/src/main/java/com/metamx/druid/coordination/ZkCoordinator.java
	server/src/main/java/com/metamx/druid/loading/SingleSegmentLoader.java
 2013-07-02 15:57:31 -07:00
fjy 7219ed15d3 fix according to code review 2013-07-02 15:56:12 -07:00
cheddar 797a083b69 1) Adjust SingleSegmentLoader to allow for storing segments on multiple different mount points. The specification language is really janky right now, so this is remaining a stealth feature for the time being. 2013-07-01 14:56:57 -07:00
cheddar e8afeda046 Merge branch 'master' of github.com:metamx/druid 2013-07-01 14:54:32 -07:00
fjy 5a57539736 fix hdfs config 2013-06-28 10:18:10 -07:00
Pablo Nebrera e9ce6b596c PostgresSQL integration with druid 2013-06-28 12:49:24 +02:00
fjy 2969d9a713 Merge branch 'master' into batch-zk 
Conflicts:
	client/src/main/java/com/metamx/druid/QueryableNode.java
 2013-06-27 16:46:37 -07:00
fjy a6787ab201 fix up configs 2013-06-27 16:40:17 -07:00
fjy d5306c5dd9 Merge branch 'master' into worker-resource 
Conflicts:
	client/src/main/java/com/metamx/druid/QueryableNode.java
 2013-06-27 12:52:58 -07:00
fjy cce62cbd8b The highly important commit of changing a log line 2013-06-26 17:27:04 -07:00
fjy 15a4dae2b1 bunch of issue fixes 2013-06-26 10:45:54 -07:00
cheddar cd0fa34a73 1) Log exception when there is an error removing a segment 2013-06-25 11:18:04 -07:00
cheddar 149c2e3560 1) A bunch more changes to make things actually run. Not quite done, but close! 2013-06-20 17:19:51 -07:00
fjy caa68e101a first commit; things working right now 2013-06-19 15:56:45 -07:00
cheddar 11ea15fc1a 1) Introduce Jetty 8 
2) Fix up modules based on exceptions at startup for MasterMain
 2013-06-17 10:53:50 -07:00
cheddar 06f7e7e665 1) Fix DatabaseSegmentManager so that Unit Test passes 2013-06-10 14:13:28 -07:00
fjy 3312d9b802 fix merging problem with linear shard specs 2013-06-10 14:07:36 -07:00
cheddar 2f56c24259 1) Inject IndexingServiceClient 
2) Switch all the DBI references to IDBI
 2013-06-07 17:37:33 -07:00
cheddar 06f90bc2e1 Merge branch 'master' of github.com:metamx/druid 2013-06-07 14:15:39 -07:00
cheddar a949f1672d 1) Fix bug with Master stopping trying to be Master after he losses it once 2013-06-07 14:14:58 -07:00
fjy 37a3994003 add delegating executor service and fix bug with query priortization 2013-06-07 14:08:51 -07:00
cheddar f68df7ab69 1) Make tests work and continue trying to make the DruidMaster start up with just Guice 2013-06-07 12:01:46 -07:00
fjy f1dea971f2 Merge pull request #156 from metamx/query-prioritization 
Query prioritization
 2013-06-06 16:05:47 -07:00
fjy 19c5dacf3a more code review fixes 2013-06-06 15:57:18 -07:00
cheddar 9df458a065 1) Initial commit of conversion to using Guice modules for bootstrapping. Things don't actually completely work yet. 2013-06-06 15:43:15 -07:00
fjy e20f547261 fix according to code review 2013-06-06 15:37:19 -07:00
cheddar b8feb98616 1) Make Master honor the druid.monitoring.monitorSystem property to turn off Sigar 2013-06-06 14:55:23 -07:00
fjy d48b4d8d97 Merge branch 'master' into query-prioritization 2013-06-06 14:19:50 -07:00
fjy 8202bc70eb optimize spatial and update poms everywhere 2013-06-05 15:26:40 -07:00
fjy 61074daa67 fix merge error 2013-06-04 18:56:20 -07:00
fjy 63b379ae90 Merge branch 'master' into refactor-indexing 2013-06-04 17:25:16 -07:00
fjy 06931ee0f5 introduce availability groups 2013-06-04 17:12:19 -07:00
cheddar 1a02637bea 1) add full parameter to master endpoint /info/rules/dataSourceName that returns the fully resolved rules, not just the specified rules 2013-06-04 17:10:25 -07:00
fjy 42cc87a294 Merge branch 'master' into refactor-indexing 
Conflicts:
	indexing-service/src/main/java/com/metamx/druid/indexing/common/task/IndexTask.java
	pom.xml
 2013-05-31 17:28:59 -07:00
fjy 09f7b181d4 first commit, working UT 2013-05-29 17:04:25 -07:00
xvrl 192b24c355 enable gzip compression for HTTP requests 2013-05-22 23:02:37 -07:00
fjy 1fb8c9db7f fix spatial bugs and more tests 2013-05-22 13:35:12 -07:00
fjy 47f2d3b0aa fix druidsetup properties path bug 2013-05-22 10:25:11 -07:00
fjy 39f55a010c additional bug fixes for hadoop based spatial indexing 2013-05-21 21:33:06 -07:00
fjy 6e5b893e6d make spatial filters work 2013-05-21 17:10:44 -07:00
fjy 69c7131a7b cleanup interval determination in II 2013-05-20 21:06:32 -07:00
fjy efc92e7ca6 bug fixes for segment metadata queries and spatial indexing 2013-05-20 20:19:18 -07:00
fjy 1b4a543f25 add default cassandra configs 2013-05-20 13:44:33 -07:00
fjy a792525063 Merge tag 'druid-0.4.12.5' 
[maven-release-plugin]  copy for tag druid-0.4.12.5

Conflicts:
	client/pom.xml
	common/pom.xml
	examples/pom.xml
	index-common/pom.xml
	indexer/pom.xml
	merger/pom.xml
	pom.xml
	realtime/pom.xml
	server/pom.xml
	services/pom.xml
 2013-05-20 12:14:23 -07:00
fjy 54c48aac6d fix DruidSetup not working 2013-05-20 12:06:21 -07:00
fjy 8af9598157 fix currSize bug and maxTime bugs 2013-05-18 13:21:45 -07:00
fjy b0a58af05d Merge pull request #140 from boneill42/cassandra-segments 
Initial implementation of Cassandra Data Segment Pusher and Puller
 2013-05-17 09:56:44 -07:00
Brian O'Neill c8ff5ca2fb Using string formatter in log statements. 
Clean up of imports on the pusher.
 2013-05-17 12:54:03 -04:00
fjy c6ad0753f0 bug fixes 2013-05-16 16:02:20 -07:00
Brian O'Neill 61c014c49f Cleaned up imports. 2013-05-16 13:33:39 -04:00
Brian O'Neill 23998f3f01 - Added cleanup to the puller. 
- Edited the documentation to remove reference to real-time node.
 2013-05-16 13:04:46 -04:00
fjy 6213c0b63c Merge branch 'master' into refactor-indexing 2013-05-15 17:14:40 -07:00
fjy 20ae1d8b6b lots of cleanups and refactorings 2013-05-15 15:37:04 -07:00
fjy 803e8ff69e add configurable spatial dimensions to hadoop indexer 2013-05-14 17:38:05 -07:00
fjy 5af188f18d Merge branch 'master' into spatial 2013-05-14 16:10:18 -07:00
cheddar 86415e70ea Merge pull request #142 from metamx/improve-bard 
Enable the broker to select a compute node will the least number of open connections
 2013-05-14 15:18:04 -07:00
cheddar fe5627e119 1) Left some log lines in the tests... That's no good. 2013-05-14 16:59:34 -05:00
cheddar 8bd19e9d04 1) Take in g9yuayon's pull request, fix merge conflicts and formatting 2013-05-14 16:55:43 -05:00
fjy e8ab0893ff fix column size reporting 2013-05-14 14:52:38 -07:00
fjy 1a978fe525 fix predicates 2013-05-12 15:26:19 -07:00
fjy bcfeac2d8c fix NPE 2013-05-12 15:14:57 -07:00
Brian O'Neill 8e0c0e6d47 Switched to default the aws properties. 2013-05-08 16:02:11 -04:00
Brian O'Neill 3eb0f4dfca Fix hierarchy in Pusher (to use CassandraStorage superclass) 2013-05-08 15:50:58 -04:00
fjy 986bb2482d Merge branch 'master' into spatial 2013-05-08 10:37:38 -07:00
fjy 43d3c53806 fix order by double sum agg 2013-05-08 10:14:40 -07:00
fjy 32f82364f1 Merge branch 'master' into spatial 2013-05-07 14:12:05 -07:00
Brian O'Neill 863b8808cc Merge branch 'master' of https://github.com/metamx/druid into cassandra-segments 2013-05-07 11:36:49 -04:00
Brian O'Neill 41e9f8fcb4 Working Push & Pull. 2013-05-07 11:35:14 -04:00
Brian O'Neill 8e8736291f Added puller. 2013-05-06 23:14:18 -04:00
cheddar 01c1257cee Merge pull request #134 from metamx/event-push 
Event push
 2013-05-06 15:10:44 -07:00
Brian O'Neill 142668cda5 Merge branch 'master' of https://github.com/metamx/druid into cassandra-segments 2013-05-06 17:54:56 -04:00
Brian O'Neill 79d802fede Working data segment pushed for C*. 2013-05-06 17:54:12 -04:00
fjy 8902986069 Merge branch 'master' into spatial 2013-05-03 23:09:30 -07:00
fjy 344a2b5d24 numerous bug fixes and improvements according to code review 2013-05-03 23:09:05 -07:00
cheddar 665f3ef0fd 1) Unit tests for order by and limit stuff for groupBy queries 
2) Adjustments to order by and limit stuff in light of unit tests-n-stuff
 2013-05-03 17:47:18 -05:00
fjy d57141f46e checking stuff in but stuff not really working yet 2013-05-03 10:35:26 -07:00
Gian Merlino af08ea7617 ServiceAnnouncer: 
- Interface with two impls, Noop and Curator
- Used wherever fine services are advertised

Initialization:
- Don't automatically register with service discovery
- Add makeServiceAnnouncer, announceDefaultService methods
- Replace serviceInstance with makeServiceInstanceFactory
 2013-05-03 14:29:17 +03:00
Brian O'Neill b8c8ed8d68 Formatting. 2013-05-01 15:17:15 -04:00
Brian O'Neill bf01399a2a Cassandra Data Segment Pusher, initial commit. 2013-05-01 15:11:50 -04:00
fjy 6d4c0850ca Merge branch 'master' into spatial 
Conflicts:
	client/src/main/java/com/metamx/druid/QueryableNode.java
 2013-05-01 11:49:00 -07:00
fjy 63ccb24ef6 Merge branch 'master' into spatial 
Conflicts:
	pom.xml
	server/src/main/java/com/metamx/druid/http/InfoResource.java
 2013-05-01 10:12:04 -07:00
cheddar 0a3492c9f5 Move all the code for examples under the examples directory. Also add initial take on flights demo. 2013-04-30 18:27:38 -05:00
cheddar 8922adb1f2 1) Make the demos work again. 
2) First step in restructuring the demos into a single repository
 2013-04-30 18:13:55 -05:00
cheddar 60b279b0d3 1) Add support for storing segments in HDFS 2013-04-30 12:47:43 -05:00
cheddar a665e8b3b7 whitespace! 2013-04-29 14:02:30 -05:00
cheddar 2630d75cd3 More explicit comments around Zookeeper race condition 2013-04-29 13:58:57 -05:00
cheddar 405eed7c60 Better name for ServerInventoryThingie 2013-04-29 11:48:42 -05:00
cheddar 99305b29c8 Fixorz unit tests! 2013-04-26 19:19:57 -05:00
cheddar 163a73bed1 More logging stuff 2013-04-26 19:19:57 -05:00
cheddar de5c8c9d64 It's better when things compile 2013-04-26 19:19:56 -05:00
cheddar 1eb2dfe17e 1) Don't need a ServiceEmitter in ZkCoordinator anymore. 2013-04-26 19:19:56 -05:00
cheddar 38b5d94ffc 1) Imports... 2013-04-26 19:19:56 -05:00
cheddar 67ce1b6f26 1) Adjust all of the uses of zookeeper paths to be based on ZkPathsConfig 2013-04-26 19:19:56 -05:00
cheddar 3e94204c44 1) Switch from com.netflix.curator to org.apache.curator 2013-04-26 19:19:55 -05:00
cheddar b8ba9138ff 1) Fix bug with Master not starting up without an indexer specified 
2) Fix bug with PotentiallyGzippedCompressionProvider only catching ZipExceptions and not IOException (java 6 throws IO, java 7 throws Zip)
3) Create DruidServerMetadata and use that instead of DruidServer to represent the current server
 2013-04-26 19:19:55 -05:00
cheddar f71b941a1a 1) Refactor the announcement of segments to all exist inside the DataSegmentAnnouncer 
2) Adjust the ExecutorNodes to expose the correct nodeType given the refactorings
 2013-04-26 19:19:54 -05:00
cheddar 7370b0f2fc 1) Cleanup TODO comments 2013-04-26 19:19:54 -05:00
cheddar 57c31656d1 1) Eliminate PhoneBook 2013-04-26 19:19:53 -05:00
cheddar dde50a0d87 1) Remove references to zkclient 
2) Eradicate zkclient from the poms!
 2013-04-26 19:19:53 -05:00
cheddar 43d630c098 1) Replace InventoryManagers with Curator-based Inventory Managers 
2) Replace ZkSegmentAnnouncer with CuratorSegmentAnnouncer
3) Adjust LoadQueuePeons to operate using Curator
4) Remove ZkPhoneBook

Things left to do:
1) Unit tests
2) Remove references to zkclient library
3) Remove references to PhoneBook interface
 2013-04-26 19:19:23 -05:00
fjy e01a78515e many more fixes and unit tests 2013-04-25 19:01:45 -07:00
fjy 13b617cfa9 fix some interval bugs 2013-04-23 16:53:57 -07:00
fjy 7184e528fd fix some config bugs 2013-04-23 13:16:50 -07:00
fjy 70937af7bd Merge branch 'master' of github.com:metamx/druid into spatial 
Conflicts:
	server/src/main/java/com/metamx/druid/index/v1/IndexStorageAdapter.java
	server/src/main/java/com/metamx/druid/index/v1/QueryableIndexStorageAdapter.java
 2013-04-23 11:40:44 -07:00
fjy 9b9c516e30 more tests 2013-04-22 18:39:41 -07:00
fjy 6c4e844f97 semi working spatial 2013-04-19 16:56:56 -07:00
Fangjin Yang 71269d7e88 somehow tests are passing 2013-04-17 17:13:24 -07:00
xvrl 50998671ae formatting 2013-04-16 09:33:50 -07:00
xvrl 4597d7f44e remove IndexStorageAdapter 2013-04-16 09:33:50 -07:00
xvrl 2afe2bdeae formatting 2013-04-16 09:33:50 -07:00
xvrl c83631928c object column selectors 2013-04-16 09:33:50 -07:00
xvrl 0c68bd1b1d rename MetricSelectorFactory to ColumnSelectorFactory 2013-04-16 09:33:49 -07:00
Fangjin Yang cc929b2a07 slightly better configuration when talking to indexing service 2013-04-15 11:09:04 -07:00
Eric Tschetter 55648c47a7 1) Adjust the GroupByQuery to also be able to merge results on the local node. Fixes #116 
2) Make the GroupByQuery operate in a multi-threaded fashion by default (this is configurable via druid.query.groupBy.singleThreaded).  Fixes #96
3) Fix up some post aggregation computation stuff.  I believe #72 is fixed
4) Fix case sensitivity issue with post aggregations on GroupBy queries
 2013-04-09 19:13:08 -07:00
cheddar a678f08b67 Merge pull request #118 from darxriggs/master 
cleanup & refactoring
 2013-04-09 18:56:59 -07:00
Fangjin Yang 120b822c18 Merge branch 'master' of github.com:metamx/druid 
Conflicts:
	client/src/main/java/com/metamx/druid/QueryableNode.java
	server/src/main/java/com/metamx/druid/http/ComputeNode.java
 2013-04-08 11:24:17 -07:00
Fangjin Yang 925c104dd2 update emitter version and fix some NPEs 2013-04-08 11:23:08 -07:00
René Scheibe 6c27cd2e8e cleanup - removed unused imports 2013-04-07 23:31:22 +02:00
René Scheibe a6dc29cf85 cleanup - removed extra empty lines 
- there should not be any adjacent empty lines
 2013-04-07 14:32:09 +02:00
Nelson Ray b1f2186c1f use reservoir sampling in pickSegmentToMove to avoid IndexOutOfBoundsException when a segment gets dropped mid-run 2013-04-02 12:22:11 -07:00
René Scheibe c9d541364e fix cut-n-paste mistake in error message (issue #81) 2013-04-02 18:42:30 +02:00
Fangjin Yang 0730976c80 fix broken UT 2013-04-01 17:16:19 -07:00
Fangjin Yang 258f18471e nodes will not try and download segments during startup 2013-04-01 16:55:03 -07:00
Fangjin Yang b56ab64572 fix segment partition deletion bug 2013-04-01 14:26:02 -07:00
Fangjin Yang 5bdcc3613c fix inconsistent druid http paths + fix master console bugs 2013-03-27 10:40:44 -07:00
Fangjin Yang a7d836a0dd fix NPE for master balancing; introduce max number of workers for indexing service 2013-03-26 10:25:52 -07:00
fjy 05fdf08644 Merge pull request #107 from metamx/aggregator-close 
add close method to aggregators
 2013-03-25 13:09:19 -07:00
fjy 669f792668 Merge pull request #109 from metamx/fix-groupby-allgran-timestamp 
fix output timestamps in groupby queries with granularity "all".
 2013-03-25 12:58:32 -07:00
Nelson Ray e57249bf15 refactor balancer metric emission code 2013-03-25 11:49:46 -07:00
Nelson Ray a28de5fa88 fix per tier metrics emission for new balancer 2013-03-25 10:32:02 -07:00
xvrl c420fe3b56 fix output timestamps in groupby queries with granularity "all". 2013-03-23 23:42:43 -07:00
xvrl 3687cba18e close resources as we iterate over them 2013-03-22 17:42:10 -07:00
xvrl 16e4cd4291 fix cursor overshooting granularity boundaries in IncrementalIndexStorageAdapter as well 2013-03-21 10:38:39 -07:00
xvrl c0397aa67c add test case for granularities not aligned with segment boundaries 2013-03-21 10:38:06 -07:00
xvrl d722c99530 fix cursor overshooting granularity boundaries, due to incorrect gran.next() usage 2013-03-20 23:36:52 -07:00
xvrl 57f8810721 properly close rowiterators 2013-03-20 14:07:06 -07:00
xvrl 0b04114c12 more squigglies 2013-03-20 12:17:20 -07:00
xvrl 8eec41f934 squigglies ftw 2013-03-20 11:42:17 -07:00
Eric Tschetter cc08d6029f 1) getDimensionCardinality() on QueryIndexStorageAdapter wasn't case insensitive. 2013-03-20 12:03:07 -05:00
xvrl c68bd66945 add close method to aggregators 2013-03-19 16:03:48 -07:00
Eric Tschetter 71aa8b8077 1) It's great when log lines don't generate exceptions 2013-03-18 17:06:51 -05:00
Eric Tschetter 57f52f71fc 1) Try checking for parent dirs first and creating if they aren't there when moving files from the legacy directory to the new one. 2013-03-18 17:04:11 -05:00
Fangjin Yang d316dfc8e1 Merge branch 'master' of github.com:metamx/druid 2013-03-15 14:29:19 -07:00
Fangjin Yang 6f771796b6 fix js bugs with indexer console 2013-03-15 14:28:55 -07:00
Eric Tschetter a933438e4e 1) Fix bugs with VersionConverterTask 
2) Fix bugs with NPEs on indexing
 2013-03-15 13:48:55 -05:00
Gian Merlino f6752799bc Merge branch 'master' into realtime-index-task 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/actions/SegmentInsertAction.java
 2013-03-14 13:45:59 -07:00
cheddar 2a94bd508d Merge pull request #104 from metamx/javascript-dimfilters 
support for Javascript DimFilters
 2013-03-14 11:47:55 -07:00
Gian Merlino df8e4d4061 Merge branch 'master' into realtime-index-task 2013-03-13 22:43:01 -07:00
Eric Tschetter 4c165b4880 1) Better logging of master doing version checking 
2) Exception out when the scv cannot find its indexer
 2013-03-13 19:15:29 -05:00
Gian Merlino f8d9a3b19b IndexGranularity: Fix increment for day, week 2013-03-11 11:08:44 -07:00
Eric Tschetter d1161ae0ec 1) Master should create the config table if it's not there. 2013-03-08 12:41:46 -06:00
Nelson Ray 7a57609c5b emit metrics corresponding to cost-based balancing 2013-03-07 16:31:57 -08:00
Nelson Ray 9f405c0b1e fix constant segment balancing and separate out the differences between segment assignment and balancing 2013-03-07 16:31:57 -08:00
Eric Tschetter 1c3ef48f34 1) Adjust the Config stuff that WorkerSetupManager was using to be reusable 
2) Use new ConfigManager to pull out a whitelist for the Master
 2013-03-07 17:45:57 -06:00
Eric Tschetter 6864007c05 1) Remove Task parameter from other methods off of TaskToolbox that required it 
2) Move getTaskDir() off of TaskConfig and onto TaskToolbox
3) Rename SegmentKiller interface to DataSegmentKiller
4) Change signature of "DataSegmentKiller.kill(Collection<DataSegment>) throws ServiceException" to just kill(DataSegment) throws SegmentLoadingException
5) Add various log messages
6) Update the version of the segment that has been converted
 2013-03-07 17:44:06 -06:00
Eric Tschetter c9b411c0ca 1) Remove the need for TaskActions to require a Task as a constructor parameter 2013-03-07 17:44:05 -06:00
Eric Tschetter fd0fcc7319 Whitespace! 2013-03-07 16:18:31 -06:00
Eric Tschetter 82aa6ed7dd 1) Adjust the SingleSegmentLoader to be able to use cache locations from previous versions as well without re-downloading 2013-03-07 15:55:23 -06:00
Fangjin Yang 9d9efadf72 working indexer console 2013-03-06 10:38:40 -08:00
xvrl 4cb3ca00bc support for javascript dimfilters 2013-03-06 09:59:53 -08:00
Fangjin Yang 92ab51e70c fix bugs in balancing logic and removing auto-scaled node 2013-03-04 13:03:15 -08:00
Gian Merlino f862d9205d Merge branch 'master' into hadoop-index-task 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
 2013-02-27 06:53:06 -08:00
Nelson Ray d1634fe5ef fix cost balancing so assignment/balancing doesn't get stuck 2013-02-26 18:58:16 -08:00
Nelson Ray e67a8757ba Merge remote-tracking branch 'origin/master' into new_balancer 
Conflicts:
	server/src/main/java/com/metamx/druid/master/DruidMaster.java
 2013-02-26 14:18:07 -08:00
Gian Merlino d32a6284ce Merge branch 'task-stuff' into indexing_refactor 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/worker/config/WorkerConfig.java
 2013-02-26 13:03:56 -08:00
Gian Merlino 49700dffe9 Merge branch 'master' into task-stuff 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskToolbox.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexGeneratorTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/MergeTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
	merger/src/main/java/com/metamx/druid/merger/worker/http/WorkerNode.java
 2013-02-25 12:18:31 -08:00
Eric Tschetter f8c54a72c2 1) Changes to allow for local storage 2013-02-21 15:47:01 -06:00
Eric Tschetter dc3459d3f9 1) Initial commit of refactorings on top of housejester's to simplify the zipping and simplify the creation of other methods of loading data 2013-02-21 15:47:01 -06:00
James Estes d1626576c0 Working toward making it easier to add new SegmentPullers. 
1) Move the local cacheFile logic out of the S3 pullers into the SingleSegmentLoader
2) Make the S3SegmentPuller just pull down the file
3) Make the Loader do the unzip, ungzip, or rename
4) 2 and 3 make S3ZippedSegmentPuller not necessary (still there, just deprecated and empty)
4) Tweak the TaskToolbox so that the Pullers returned by getSegmentGetters behave the same as they did before
 2013-02-21 15:47:01 -06:00
Deep Ganguli 60da9ffddb Merge branch 'master' into hadoop-index-task 
Conflicts:
	common/src/main/java/com/metamx/druid/utils/JodaUtils.java
	indexer/src/main/java/com/metamx/druid/indexer/DbUpdaterJob.java
	indexer/src/main/java/com/metamx/druid/indexer/HadoopDruidIndexerConfig.java
	indexer/src/main/java/com/metamx/druid/indexer/IndexGeneratorJob.java
 2013-02-15 13:22:31 -08:00
Nelson Ray 728e208f49 merge master 2013-02-15 11:39:22 -08:00
xvrl 05d347c304 Merge remote-tracking branch 'origin/master' into speedup-timeboundary 
Conflicts:
	client/pom.xml
	common/pom.xml
	druid-services/pom.xml
	examples/pom.xml
	examples/rand/pom.xml
	examples/twitter/pom.xml
	index-common/pom.xml
	indexer/pom.xml
	merger/pom.xml
	pom.xml
	realtime/pom.xml
	server/pom.xml
 2013-02-15 11:25:47 -08:00
Eric Tschetter 5ee7c0c264 1) Remove stray import of java.util.List 2013-02-15 12:15:53 -06:00
Gian Merlino 548c901c0b Additional fix for columns with cardinality 0 2013-02-14 23:45:02 -08:00
Gian Merlino 22d1f2a0c0 IndexMergerTest: Empty column test 2013-02-14 21:11:31 -08:00
Eric Tschetter 4f11eb5209 1) Fix alert sent from line DruidMasterBalancer:71 to not include all of the segments that a server is holding 2013-02-14 18:24:51 -06:00
Fangjin Yang bf0d9c601d Merge branch 'task-stuff' into indexing_refactor 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/TaskToolbox.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/KillTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
	merger/src/main/java/com/metamx/druid/merger/common/task/V8toV9UpgradeTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/RemoteTaskRunner.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskMasterLifecycle.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskQueue.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskRunner.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskRunnerWorkQueue.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorServletModule.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
 2013-02-14 15:42:07 -08:00
Gian Merlino 18daa19fc9 Merge branch 'master' into task-stuff 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskHolder.java
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/task/AbstractTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexGeneratorTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskContext.java
	merger/src/main/java/com/metamx/druid/merger/worker/TaskMonitor.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/TaskQueueTest.java
	server/src/main/java/com/metamx/druid/master/DruidMaster.java
 2013-02-14 13:10:29 -08:00
Nelson Ray a616f86128 add DateTime import back 2013-02-14 09:09:22 -08:00
Nelson Ray 754b065c54 merge master 2013-02-14 09:08:25 -08:00
Eric Tschetter c5b78e75cf 1) Move up to Jackson 2.1 
2) Some bugs were fixed, but they ended up getting mixed in with the Jackson upgrade :(
 2013-02-13 18:51:32 -08:00
Fangjin Yang 82f02a4fe4 Merge branch 'master' into indexing_refactor 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexDeterminePartitionsTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/Task.java
	merger/src/main/java/com/metamx/druid/merger/common/task/V8toV9UpgradeTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/MergerDBCoordinator.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/RemoteTaskRunner.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
 2013-02-13 16:34:14 -08:00
Fangjin Yang c7b4973700 bug fixes for RTR 2013-02-13 14:19:11 -08:00
Nelson Ray 429e23d3b2 change break to continue in BalancerCostAnalyzer to skip calculating cost when it's not necessary 2013-02-13 08:57:39 -08:00
Eric Tschetter a0b159fed7 1) Move up to Jackson 2.1 
2) Some bugs were fixed, but they ended up getting mixed in with the Jackson upgrade :(
 2013-02-12 20:58:17 -06:00
Nelson Ray fcbac96193 revert change to findNewSegmentHome: now return single ServerHolder 2013-02-12 10:47:30 -08:00
Fangjin Yang b27a459f71 more refactoring and UT passing at this point 2013-02-11 18:13:01 -08:00
xvrl e9f546c4ca add default implementations to QueryToolChest methods 2013-02-11 16:51:40 -08:00
Nelson Ray 695b78b19c make findNewSegmentHome return a priority queue so assignment can assign replicants to other servers 2013-02-11 15:28:17 -08:00
xvrl e10025c841 filter out unneeded segments for TimeBoundary query 2013-02-11 13:21:37 -08:00
Fangjin Yang 69d0d98df6 t Merge branch 'master' into indexing_refactor 2013-02-11 12:37:20 -08:00
Fangjin Yang a9176e738a Merge branch 'master' into indexing_refactor 2013-02-11 12:36:49 -08:00
Nelson Ray 1b9764fffb add binary version to datasegments for balancer test 2013-02-11 11:36:45 -08:00
Nelson Ray e12da51057 merge master 2013-02-11 11:28:50 -08:00
cheddar 05168808c2 Merge pull request #77 from metacret/master 
Adding LinearShardSpec and ACL on S3 object
 2013-02-11 11:12:31 -08:00
Fangjin Yang 07c005a0f7 Merge branch 'master' of github.com:metamx/druid 2013-02-11 10:56:39 -08:00
Fangjin Yang ac93f752b0 fix a case where a segment may get stuck in the terminate queue 2013-02-11 10:56:30 -08:00
Fangjin Yang 111df9a90e Merge branch 'master' into indexing_refactor 
Conflicts:
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
	server/src/main/java/com/metamx/druid/master/DruidMaster.java
 2013-02-07 10:35:32 -08:00
xvrl 6ff42f379c fix cut-n-paste mistake in error message 2013-02-07 10:11:36 -08:00
Fangjin Yang 267c797023 fix bug in throttle replicants 2013-02-06 18:40:29 -08:00
Fangjin Yang 0d303bee96 reset lifetimes for the replication throttle 2013-02-06 17:41:08 -08:00
Fangjin Yang 7540dd58ca fix broken UT 2013-02-06 16:52:24 -08:00
fjy e2862fe1fb Merge pull request #79 from metamx/bug_fixes 
Throttle replicants that are created by rules
 2013-02-06 16:50:04 -08:00
Fangjin Yang 84baef0462 fix some bugs from last commit 2013-02-06 13:57:56 -08:00
Fangjin Yang 97999c06c4 throttle replicants 2013-02-06 11:08:44 -08:00
Jae Hyeon Bae 4f70bbc687 adding linear shard spec and ACL for s3 bucket 2013-02-04 10:27:09 -08:00
Eric Tschetter 34b4383934 1) Adjust DataSegment to have a binaryVersion field that indicates the expected binary version of the segment files 
2) Eliminate race condition from RemoteTaskRunnerTest.testAlreadyExecutedTask()
 2013-02-01 18:59:33 -06:00
Eric Tschetter 1a9edbebd3 1) whitespace 2013-02-01 16:21:26 -06:00
Eric Tschetter 5f2c3cec1b Merge branch 'master' of github.com:metamx/druid 2013-02-01 16:19:26 -06:00
Eric Tschetter 7de5c7806e 1) Some whitespace changes 
2) Fix up the alerting to actually include the segment and server for when the MasterBalancer has something stuck in the queue
 2013-02-01 16:19:17 -06:00
Fangjin Yang 9e88197845 change endpoints for killing segments 2013-02-01 14:08:52 -08:00
Nelson Ray 1c9270add3 change format of line comment 2013-02-01 11:25:41 -08:00
Nelson Ray 0b61f96965 extra documentation 2013-02-01 11:25:05 -08:00
Fangjin Yang f50c214514 Merge branch 'master' into task-stuff 2013-01-31 09:59:52 -08:00
Nelson Ray dc49cccf56 javadoc format 2013-01-29 18:32:52 -08:00
Nelson Ray 73f774e768 make masterbalancertest more stringent 2013-01-29 18:17:38 -08:00
Nelson Ray a1ce4b294e simplify balancer test 2013-01-29 18:16:49 -08:00
Nelson Ray b205f7b3c4 use existing currentlyMovingSegments instead of creating a reundant segmentsBeingMoved 2013-01-29 18:16:34 -08:00
Nelson Ray 034d652569 make balancer tests take into account loadqueue status 2013-01-29 17:13:01 -08:00
cheddar 9a22e46eca Merge pull request #71 from metamx/queryrunner-onerow-test 
Bug fixes for incremental index storage adapter and one row indexes
 2013-01-29 13:41:15 -08:00
cheddar a7abd2ca86 Merge pull request #70 from metamx/timezone-tests 
More time zone tests
 2013-01-29 13:40:04 -08:00
Fangjin Yang 8fc10a61a0 bug fix for incremental index and 1 row 2013-01-29 13:23:57 -08:00
Fangjin Yang 41ca33c734 Merge branch 'master' of github.com:metamx/druid into task-stuff 2013-01-29 12:57:56 -08:00
Gian Merlino 61569f62be TimeseriesQueryRunner tests with small incremental indexes 2013-01-29 12:54:52 -08:00
Eric Tschetter da914b835e 1) Add caching for SegmentMetadataQuery 2013-01-29 13:04:57 -06:00
xvrl 27e09987c6 timezone test 2013-01-29 10:38:22 -08:00
Eric Tschetter 08d3253f3d Merge branch 'master' of github.com:metamx/druid 2013-01-29 11:29:42 -06:00
Eric Tschetter f2cce28297 1) Make SegmentMetadataQuery work 2013-01-29 11:29:26 -06:00
Nelson Ray 6438401a32 split pick segment logic into its own method 2013-01-28 15:06:09 -08:00
Eric Tschetter 7439a2e820 1) Make tests pass 2013-01-28 16:42:51 -06:00
xvrl c0822325e7 proper groupby tests 2013-01-28 16:41:03 -06:00
cheddar 5f18f368e3 Merge pull request #61 from metamx/fix-cache-timezone 
Cache did not preserve timezone information
 2013-01-28 14:39:00 -08:00
Fangjin Yang 74057600f9 fix worker node still using db connection 2013-01-27 09:57:37 -08:00
Gian Merlino f3b04d3f5f S3SegmentKiller: Add TODO note 2013-01-25 11:33:50 -08:00
Gian Merlino e6a618ca76 Merge branch 'killsegments' into task-stuff 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexGeneratorTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/MergeTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/LocalTaskRunner.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskQueue.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/exec/TaskConsumer.java
	merger/src/main/java/com/metamx/druid/merger/worker/TaskMonitor.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/TaskQueueTest.java
 2013-01-25 11:30:10 -08:00
xvrl 5e1cac6d9f timezone test 2013-01-24 17:57:10 -08:00
Deep Ganguli 017d4779d6 Implemented Hadoop Index Task which takes as input a HadoopDruidIndexConfig and 
generates index segments.

The HadoopIndexTask run method wraps a HadoopDruidIndexerJob run method. The
key modifications to the HadoopDruidIndexerJob are as follows:

- The UpDaterJobSpec field of the config that is used to set up the indexer job
  is set to null. This ensures that the job does not push a list of published
segments to the database, in order to allow the indexing service to handle this
later.
- Set the version field of the config file based on the TaskContext. Also
  changed config.setVersion method to take a string (as opposed to a Date) as
input, and propogated this change where necessary.
- Set the SegmentOutputDir field of the config file based on the TaskToolbox,
  to allow the indexing service to handle where to write the segments too.
- Added a method to IndexGeneratorJob called getPublishedSegments, that simply
  returns a list of published segments without publishing this list to the
database.
 2013-01-23 19:27:14 -08:00
Eric Tschetter 67feee3cd6 1) Indexes don't always have an index.drd file anymore 2013-01-23 18:36:52 -06:00