Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,268 Commits 111 Branches 503 Tags 726 MiB
Commit Graph

498 Commits

Author SHA1 Message Date
Xavier Léauté 9bc20ef8bf prefer druid.curator.compress to druid.indexer.runner.compressZnodes 2014-11-06 11:28:51 -08:00
Xavier Léauté 350bb09605 refactor sql storage to abstract task storage 2014-11-05 17:19:37 -08:00
Xavier Léauté 1872b8f979 make it easier to test 2014-10-31 14:49:07 -07:00
Xavier Léauté 97a2f5af4a rename db->metadata 2014-10-31 10:54:33 -07:00
Xavier Léauté 9c06db021f rename db->metadata postgres->postgresql 2014-10-31 10:30:27 -07:00
Xavier Léauté fb4d41cedb make the injection gods happy 2014-10-30 21:16:36 -07:00
Xavier Léauté 377151beda better abstraction for metadatastorage 2014-10-30 18:23:35 -07:00
jisookim0513 aa754b86e8 build success! 2014-10-24 11:28:42 -07:00
fjy bef74104d9 merge with 0.7.x and resolve any conflicts 2014-10-23 17:24:06 -07:00
Gian Merlino ca0a4bd8a5 RemoteTaskActionClient: Better error messages for non-2xx responses. 2014-10-22 18:02:22 -07:00
jisookim0513 02e79d6b15 attempted to solve merge-conflict; IncrementalIndex has unresolved classes after updates - needs to be fixed 2014-10-22 00:18:17 -07:00
jisookim0513 37979282fe enabled ansi-quote in mysql; insert statement should now work 2014-10-21 00:09:19 -07:00
jisookim0513 b8cbe2457a fixed variable name 2014-10-17 00:13:30 -07:00
jisookim0513 7d5c5f2083 fixed createTable; fixed miscellaneous stuff; added DerbyMetadataRuleManagerProvider 2014-10-17 00:10:36 -07:00
nishantmonu51 f4a97aebbc fix rollup for hashed partitions 
truncate timestamp while calculating the partitionNumber
 2014-10-15 22:32:56 +05:30
nishantmonu51 bce388fb27 merge changes from 0.7.x branch 2014-10-14 18:46:02 +05:30
nishantmonu51 b5d66381f3 more cleanup 2014-10-14 18:32:40 +05:30
nishantmonu51 454acd3f5a remove backwards compatible code 
1) remove backwards compatible and deprecated code
2) make hashed partitions spec default
 2014-10-13 19:30:44 +05:30
Gian Merlino e1fedbe741 RemoteTaskRunner should respect worker version changes (fixes #787). 2014-10-12 11:27:44 -07:00
jisookim0513 521398267c fixed inconsistent variable names 2014-10-10 17:00:50 -07:00
fjy c7b4d5b7b4 Merge branch 'master' into druid-0.7.x 
Conflicts:
	processing/src/test/java/io/druid/segment/filter/SpatialFilterTest.java
 2014-10-02 18:12:10 -07:00
fjy c3bea245a7 fix up some bugs 2014-09-30 17:20:52 -07:00
fjy a0782d4c54 fix compile 2014-09-30 16:56:44 -07:00
jisookim0513 0e50852985 fixed MetadataTaskStorage and handler 2014-09-30 14:09:23 -07:00
Gian Merlino 0781781b99 Merge pull request #766 from metamx/extend-rtr 
make the worker selection strategy in remotetaskrunner extendable
 2014-09-30 12:52:12 -07:00
fjy fab7caafff final code reviews 2014-09-30 12:32:14 -07:00
fjy 06757034f2 add default impl 2014-09-30 11:54:29 -07:00
fjy 2b2b028e5c why am i so bad at coding 2014-09-30 11:53:18 -07:00
fjy b1b9e0a267 i suck 2014-09-30 11:45:19 -07:00
fjy 575d51b0ce fix compilation error 2014-09-30 11:44:50 -07:00
fjy 4c23a5e9f6 address cr again 2014-09-30 11:40:29 -07:00
fjy 55db06ccb1 address cr 2014-09-30 10:29:02 -07:00
nishantmonu51 358ff915bb fix merge conflicts 2014-09-30 22:19:18 +05:30
nishantmonu51 2789536bed merge changes from druid-0.7.x 2014-09-30 22:05:49 +05:30
nishantmonu51 61c7fd2e6e make ingestOffheap tuneable 2014-09-30 15:30:02 +05:30
Gian Merlino 1e6ce8ac9a TaskLogs fixes and cleanups. 
- Fix negative offsets in FileTaskLogs and HdfsTaskLogs.
- Consolidate file offset code into LogUtils (currently used in two places).
- Clean up style for HdfsTaskLogs and related classes.
- Remove unused code in ForkingTaskRunner.
 2014-09-29 16:20:34 -07:00
fjy 4a09678739 make the selection strategy in rtr extendable 2014-09-29 14:24:02 -07:00
jisookim0513 74565c9371 cleaned up the code 2014-09-27 13:10:01 -07:00
jisookim0513 6a641621b2 finished merging into druid-0.7.x; derby not working (to be fixed) 2014-09-26 14:24:53 -07:00
jisookim0513 43cc6283d3 trying to revert files that have overwritten changes 2014-09-26 12:38:04 -07:00
fjy eaf0a48b92 Merge branch 'master' into druid-0.7.x 
Conflicts:
	cassandra-storage/pom.xml
	common/pom.xml
	examples/pom.xml
	hdfs-storage/pom.xml
	histogram/pom.xml
	indexing-hadoop/pom.xml
	indexing-service/pom.xml
	kafka-eight/pom.xml
	kafka-seven/pom.xml
	pom.xml
	processing/pom.xml
	processing/src/main/java/io/druid/guice/PropertiesModule.java
	rabbitmq/pom.xml
	s3-extensions/pom.xml
	server/pom.xml
	services/pom.xml
 2014-09-26 11:39:24 -07:00
jisookim0513 3bf39cc9f8 attempted to fix merge-conflicts 2014-09-24 15:55:42 -07:00
nishantmonu51 f51ab84386 merge changes from druid-0.7.x 2014-09-22 23:48:45 +05:30
jisookim0513 273205f217 initial attempt for abstraction; druid cluster works with Derby as a default 2014-09-19 17:39:59 -07:00
nishantmonu51 8eb6466487 revert buffer size and add back rowFlushBoundary 2014-09-19 23:06:04 +05:30
nishantmonu51 e6d93a3070 fix NPE 
fix NPE when the dimension of metric is not present one of the segments
to be reIndexed.
 2014-09-17 15:57:58 +05:30
nishantmonu51 f006de8639 fix #732 
fix metric list discovery
 2014-09-16 22:12:36 +05:30
Xavier Léauté c8b8e3f6e9 negating compare is bad 2014-09-15 13:00:06 -07:00
Xavier Léauté 137ad50bf1 classes that should be static 2014-09-15 13:00:06 -07:00
Xavier Léauté e57e2d97ba make constants final 2014-09-15 12:53:40 -07:00
Xavier Léauté cfa92e8217 fix incorrect nullable annotations 2014-09-15 12:13:52 -07:00
fjy 469ccbbe5e Merge branch 'master' into druid-0.7.x 
Conflicts:
	cassandra-storage/pom.xml
	common/pom.xml
	examples/pom.xml
	hdfs-storage/pom.xml
	histogram/pom.xml
	indexing-hadoop/pom.xml
	indexing-service/pom.xml
	kafka-eight/pom.xml
	kafka-seven/pom.xml
	pom.xml
	processing/pom.xml
	processing/src/main/java/io/druid/query/FinalizeResultsQueryRunner.java
	processing/src/main/java/io/druid/query/UnionQueryRunner.java
	processing/src/main/java/io/druid/query/groupby/GroupByQueryRunnerFactory.java
	processing/src/main/java/io/druid/query/topn/TopNQueryEngine.java
	processing/src/main/java/io/druid/query/topn/TopNQueryRunnerFactory.java
	rabbitmq/pom.xml
	s3-extensions/pom.xml
	server/pom.xml
	server/src/test/java/io/druid/server/initialization/JettyTest.java
	services/pom.xml
 2014-09-11 16:20:50 -07:00
fjy fec7b43fcb make making v9 segments something completely configurable 2014-09-10 15:28:30 -07:00
fjy 351afb8be7 allow legacy index generator 2014-09-09 17:04:35 -07:00
Xavier Léauté 508e982190 Merge remote-tracking branch 'origin/master' into druid-0.7.x 
Conflicts:
	cassandra-storage/pom.xml
	common/pom.xml
	examples/config/historical/runtime.properties
	examples/config/overlord/runtime.properties
	examples/config/realtime/runtime.properties
	examples/pom.xml
	hdfs-storage/pom.xml
	histogram/pom.xml
	indexing-hadoop/pom.xml
	indexing-service/pom.xml
	kafka-eight/pom.xml
	kafka-seven/pom.xml
	pom.xml
	processing/pom.xml
	rabbitmq/pom.xml
	s3-extensions/pom.xml
	server/pom.xml
	server/src/main/java/io/druid/server/ClientQuerySegmentWalker.java
	services/pom.xml
 2014-08-30 22:42:36 -07:00
Xavier Léauté 58ab759fc6 remove unused imports 2014-08-29 14:03:47 -07:00
Xavier Léauté ac05836833 make Java 8 javadoc happy 2014-08-29 13:58:50 -07:00
Gian Merlino 8bcb73a0eb Java programs are best run with a java command. 2014-08-28 18:10:01 -07:00
Gian Merlino 5564a67bfc File.pathSeparator is better than ":". 2014-08-28 18:09:33 -07:00
Gian Merlino 68aeafaacd Allow indexing tasks to specify extra classpaths. 
This could be used by Hadoop tasks to reference configs for different clusters, assuming
that the possible configs have been pre-distributed to middle managers.
 2014-08-28 18:00:26 -07:00
fjy d64879ccca more cleanup 2014-08-20 13:22:42 -07:00
fjy 1614f40f1a fix index task 2014-08-20 13:11:00 -07:00
fjy 92f26d9a1f cleanup rowflushboundary 2014-08-20 13:09:37 -07:00
nishantmonu51 fe105d52ee use bufferSize for IndexTask 2014-08-20 22:41:34 +05:30
nishantmonu51 33354cf7fe replace maxRowsInMemory with BufferSize 2014-08-20 20:59:44 +05:30
nishantmonu51 e525562767 review comments - cleanup ColumnSelectorFactory 2014-08-20 15:04:43 +05:30
nishantmonu51 60906c3244 Revert "make valueType configurable" 
This reverts commit 6f60a3f604.
 2014-08-20 11:55:26 +05:30
fjy 4fd5479559 fix typo 2014-08-19 12:34:10 -07:00
fjy 77e514688a Merge branch 'druid-0.7.x' into offheap-incremental-index 2014-08-18 11:14:19 -07:00
Xavier Léauté 1fd30ab588 default service/host/port for all nodes 2014-08-15 17:14:05 -07:00
nishantmonu51 6f60a3f604 make valueType configurable 2014-08-13 14:37:57 +05:30
nishantmonu51 1b0a72751b Add support for LongColumn 2014-08-13 08:52:36 +05:30
nishantmonu51 c6712739dc merge changes from druid-0.7.x 2014-08-12 15:47:42 +05:30
fjy 91ebe45b4e support both rejectionPolicy and rejectionPolicyFactory in serde 2014-08-07 10:06:27 -07:00
nishantmonu51 637bd35785 merge changes from druid-0.7.x 2014-07-31 16:07:22 +05:30
Gian Merlino 09fcfc3b6d Fix race in RemoteTaskRunner that could lead to zombie tasks. 2014-07-18 11:41:50 -07:00
nishantmonu51 4ce12470a1 Add way to skip determine partitions for index task 
Add a way to skip determinePartitions for IndexTask by manually
specifying numShards.
 2014-07-18 18:52:15 +05:30
fjy d8b8826c2e Merge branch 'cleanup-ingest' of github.com:metamx/druid into cleanup-ingest 
Conflicts:
	server/src/test/java/io/druid/realtime/firehose/CombiningFirehoseFactoryTest.java
 2014-07-17 20:26:11 -07:00
fjy 291f4c00ae Merge branch 'master' of github.com:metamx/druid into cleanup-ingest 2014-07-17 20:24:59 -07:00
nishantmonu51 0e0454a34c switch reingest task to noop & fix compilation 
switch back to noop task, its confusing to have a reinvest task that
does nothing.
fix compilation
 2014-07-18 06:50:58 +05:30
fjy ded83557dd Merge pull request #640 from metamx/disable-worker 
An alternative way to disable middlemanagers based on worker version
 2014-07-17 19:07:27 -06:00
fjy beac0be45b fix enabled endpoint 2014-07-17 18:04:36 -07:00
fjy c6078ca841 address code review 2014-07-17 13:34:05 -07:00
fjy ba978d8b79 some minor cleanups to ingest firehose 2014-07-17 13:05:59 -07:00
fjy bc650a1c80 Merge pull request #627 from metamx/druid-firehose 
Functionality to ingest a Druid segment and change the schema
 2014-07-17 13:41:16 -06:00
fjy 5197ea527a disable middlemanagers based on worker version 2014-07-17 12:35:45 -07:00
nishantmonu51 e59c9ebdbc minor fixes 
fix IndexOutOfBoundsException
fix ingestFirehose
 2014-07-17 17:24:57 +05:30
nishantmonu51 972c5dac31 improve memory usage and rename firehose 2014-07-14 21:17:53 +05:30
nishantmonu51 f5f05e3a9b Sync changes from branch new-ingestion PR #599 
Sync and Resolve Conflicts
 2014-07-11 16:15:10 +05:30
nishantmonu51 7168adcca7 Use Ingest task instead of Noop Task 2014-07-08 21:20:17 +05:30
fjy ce478be899 Remove redundant forking task runner config 2014-07-07 18:08:37 -06:00
nishantmonu51 97b58eb193 review comments 
1) Rename firehose to IngestSegment
2) fix segment overlapping, intervals
3) fix overshadow
 2014-07-04 12:31:33 +05:30
nishantmonu51 518ab473f3 improve port finding strategy for task runner 
1) Recycle free ports
2) Choose only ports that are free & not used by any other application
 2014-07-03 09:58:12 +05:30
nishantmonu51 1de390801f Druid Fireshose 
Add druidFirehose, can be used to reIndex filtered data from a
datasource
 2014-07-02 20:34:18 +05:30
fjy 4c40e71e54 address cr 2014-06-19 14:48:46 -07:00
fjy a870fe5cbe inject column config 2014-06-19 14:47:57 -07:00
Xavier Léauté 09346b0a3c make column cache configurable 2014-06-19 14:43:03 -07:00
fjy 0bc1915067 Merge pull request #578 from metamx/new-guava 
Update guava, java-util, and druid-api
 2014-06-18 14:23:32 -06:00
fjy b0517dc55d Merge pull request #610 from metamx/ec2-user-data 
More flexible EC2 user data classes.
 2014-06-18 11:36:13 -06:00
Gian Merlino 5c4b9a7d5b Indexing: Allow null userData. 2014-06-18 10:08:50 -07:00
Gian Merlino 65c1553a76 Fix busted TaskConfig. 2014-06-17 17:27:41 -07:00
Gian Merlino 9f16f0a955 More flexible EC2 user data classes. 2014-06-17 17:10:17 -07:00
fjy 5227b43050 fix more test breakage 2014-06-17 10:35:01 -07:00
fjy 8a13e34c50 fix broken ut 2014-06-17 10:31:50 -07:00
Gian Merlino 54833fa43d Indexing: Make default hadoop coordinates configurable. 2014-06-17 08:53:39 -07:00
nishantmonu51 0629be622c remove unnecessary changes & fix index closing subquery 2014-06-16 18:50:49 +05:30
fjy 5bd4ad3f0f address code review 2014-06-13 11:29:49 -07:00
fjy a63cda3281 Merge branch 'master' into new-guava 
Conflicts:
	server/src/main/java/io/druid/server/QueryResource.java
 2014-06-13 10:08:10 -07:00
nishantmonu51 0ddaf4c307 fix race while index closing and querying 2014-06-13 15:36:57 +05:30
nishantmonu51 025814cfff Merge branch 'master' into offheap-incremental-index 
Conflicts:
	processing/src/main/java/io/druid/query/GroupByParallelQueryRunner.java

processing/src/main/java/io/druid/query/groupby/GroupByQueryRunnerFactor
y.java

processing/src/main/java/io/druid/segment/incremental/IncrementalIndexSt
orageAdapter.java
 2014-06-12 20:02:27 +05:30
nishantmonu51 a7e19ad892 configure buffer sizes 2014-06-12 19:32:37 +05:30
fjy ad7db018d5 add default tuning config for index task 2014-06-09 16:28:47 -07:00
fjy 5e48f3907b fix index task new schema defaults 2014-06-09 16:24:12 -07:00
nishantmonu51 5bdc4a761a close IncrementalIndex properly and free up buffer 2014-06-05 19:41:43 +05:30
nishantmonu51 6265613bb9 Merge branch 'master' into offheap-incremental-index 2014-06-05 17:42:57 +05:30
nishantmonu51 01e8a713b6 unit tests passing with offheap-indexing 2014-06-05 17:42:53 +05:30
fjy 9f4cc5ca1f fix test 2014-06-04 16:29:20 -07:00
fjy 77ec4df797 update guava, java-util, and druid-api 2014-06-03 13:43:38 -07:00
Gian Merlino 8a6384862f Better task log errors. 2014-06-02 12:55:17 -07:00
Gian Merlino 48511e15cf Autoscaling: Fix case where target < current, but target is too low. 
This can happen when workers are provisioned manually.
 2014-06-02 12:36:48 -07:00
fjy 7ffe60ca60 clean up a bit of the logic in RTR failure with exceptions in announceTask 2014-06-02 10:49:28 -07:00
nishantmonu51 6f176cee85 fail task on exception 
If the task announcement throw exception, set status to fail instead of
retrying it again and again.
 2014-06-02 22:52:50 +05:30
fjy 4c13327297 more logging for determine hashed 2014-05-30 16:19:20 -07:00
Xavier Léauté 1a8eb25852 update realtime tasks to use FilteredServerView 2014-05-29 16:41:26 -07:00
fjy 7be93a770a make all firehoses work with tasks, add a lot more documentation about configuration 2014-05-28 16:33:59 -07:00
fjy 09ad32c5c5 fix race condition with merge and persist and sink adding 
Conflicts:
	indexing-service/src/main/java/io/druid/indexing/common/task/IndexTask.java
	server/src/main/java/io/druid/segment/realtime/RealtimeManager.java
	server/src/main/java/io/druid/segment/realtime/plumber/RealtimePlumber.java
 2014-05-16 15:21:09 -07:00
fjy d75cc7b9b8 fix more serde 2014-05-06 15:17:38 -07:00
fjy 1100d2f2a1 rename configs to make a bit more sense 2014-05-06 14:52:50 -07:00
fjy b6fb4245aa Merge branch 'master' into new-schema 
Conflicts:
	indexing-hadoop/src/main/java/io/druid/indexer/HadoopDriverConfig.java
	indexing-hadoop/src/main/java/io/druid/indexer/HadoopDruidIndexerConfig.java
	indexing-hadoop/src/main/java/io/druid/indexer/HadoopDruidIndexerConfigBuilder.java
	pom.xml
	server/src/main/java/io/druid/segment/realtime/RealtimeManager.java
	server/src/main/java/io/druid/segment/realtime/firehose/EventReceiverFirehoseFactory.java
 2014-05-06 14:32:51 -07:00
fjy 79e6d4eb56 Merge pull request #528 from metamx/union-query-source 
Union query source
 2014-05-06 14:13:33 -06:00
Gian Merlino bdf9e74a3b Allow config-based overriding of hadoop job properties. 2014-05-06 09:11:31 -07:00
nishantmonu51 728a606d32 Add support for union queries 2014-05-01 01:54:52 +05:30
fjy 171d20d52d Merge branch 'move-firehose' of github.com:metamx/druid into move-firehose 2014-04-25 14:13:19 -07:00
fjy eef034ca7e Merge branch 'master' of github.com:metamx/druid into move-firehose 2014-04-25 14:13:08 -07:00
fjy 76e0a48527 Merge branch 'master' into new-schema 
Conflicts:
	indexing-hadoop/src/main/java/io/druid/indexer/DbUpdaterJob.java
	indexing-hadoop/src/test/java/io/druid/indexer/HadoopDruidIndexerConfigTest.java
	indexing-service/src/main/java/io/druid/indexing/common/task/HadoopIndexTask.java
	server/src/main/java/io/druid/segment/realtime/plumber/RealtimePlumber.java
	server/src/main/java/io/druid/segment/realtime/plumber/RealtimePlumberSchool.java
 2014-04-25 14:03:28 -07:00
fjy 174ff66a88 Merge pull request #504 from metamx/shard-count-limit-fix 
Fix shard count limit
 2014-04-23 10:12:05 -06:00
nishantmonu51 6eb282b3a8 Fix shard count limit 
Fix shard count limit at another place,
Use partitioned as a flag
 2014-04-23 13:56:06 +05:30
Gian Merlino 8d650ae131 PullDependencies changes. 
- Pull default Hadoop version by default.
- Allow additional coordinates on the command line.
 2014-04-22 14:33:53 -07:00
fjy c00fb1d08e Merge pull request #445 from metamx/hadoop-version-update 
Hadoop version update
 2014-04-17 19:16:44 -06:00
fjy 9b06b6aa58 misc fixes for router 2014-04-15 14:23:54 -07:00
fjy 9bfc738032 im deeply saddened but another case sensitivity problem and want to resolve these once and for all 2014-04-14 16:20:31 -07:00
nishantmonu51 0c95c0b689 moved file 2014-04-09 14:25:31 +05:30
fjy 1843316db6 commonalize event receiver firehose 2014-04-03 20:46:36 -07:00
Xavier Léauté e0ff2aa0d6 make isPostgreSQL check URI instead of metadata 2014-03-27 12:13:10 -07:00
nishantmonu51 a24fc84d0d Merge branch 'master' into hadoop-version-update 2014-03-27 03:32:11 +05:30
nishantmonu51 a9a6682a0e Upgrade to Hadoop 2.3.0 2014-03-26 11:43:31 +05:30
fjy 4360c050eb fix broken ut 2014-03-25 11:13:52 -07:00
fjy 771aa2ae68 backwards compat 2014-03-25 09:23:48 -07:00
fjy 5ebb2d27e3 fix hadoop 2014-03-24 18:43:31 -07:00
fjy 2adcf07f5f Merge branch 'master' into new-schema 
Conflicts:
	indexing-hadoop/src/main/java/io/druid/indexer/DetermineHashedPartitionsJob.java
	indexing-service/src/main/java/io/druid/indexing/common/task/RealtimeIndexTask.java
	indexing-service/src/test/java/io/druid/indexing/common/task/TaskSerdeTest.java
	processing/src/test/java/io/druid/segment/TestIndex.java
	server/src/main/java/io/druid/segment/realtime/RealtimeManager.java
	server/src/main/java/io/druid/segment/realtime/plumber/RealtimePlumberSchool.java
 2014-03-17 10:59:31 -07:00