Apache/druid
1
0
Fork 0
mirror of https://github.com/apache/druid.git synced 2025-02-09 11:34:54 +00:00
Code Issues Packages Projects Releases Wiki Activity
13,840 Commits 116 Branches 507 Tags
Commit Graph

4221 Commits

Author SHA1 Message Date
Gian Merlino
f3b04d3f5f S3SegmentKiller: Add TODO note 2013-01-25 11:33:50 -08:00
Gian Merlino
e6a618ca76 Merge branch 'killsegments' into task-stuff 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/common/TaskStatus.java
	merger/src/main/java/com/metamx/druid/merger/common/task/DeleteTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexGeneratorTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/IndexTask.java
	merger/src/main/java/com/metamx/druid/merger/common/task/MergeTask.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/LocalTaskRunner.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/TaskQueue.java
	merger/src/main/java/com/metamx/druid/merger/coordinator/exec/TaskConsumer.java
	merger/src/main/java/com/metamx/druid/merger/worker/TaskMonitor.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/RemoteTaskRunnerTest.java
	merger/src/test/java/com/metamx/druid/merger/coordinator/TaskQueueTest.java
 2013-01-25 11:30:10 -08:00
xvrl
5e1cac6d9f timezone test 2013-01-24 17:57:10 -08:00
Deep Ganguli
017d4779d6 Implemented Hadoop Index Task which takes as input a HadoopDruidIndexConfig and 
generates index segments.

The HadoopIndexTask run method wraps a HadoopDruidIndexerJob run method. The
key modifications to the HadoopDruidIndexerJob are as follows:

- The UpDaterJobSpec field of the config that is used to set up the indexer job
  is set to null. This ensures that the job does not push a list of published
segments to the database, in order to allow the indexing service to handle this
later.
- Set the version field of the config file based on the TaskContext. Also
  changed config.setVersion method to take a string (as opposed to a Date) as
input, and propogated this change where necessary.
- Set the SegmentOutputDir field of the config file based on the TaskToolbox,
  to allow the indexing service to handle where to write the segments too.
- Added a method to IndexGeneratorJob called getPublishedSegments, that simply
  returns a list of published segments without publishing this list to the
database.
 2013-01-23 19:27:14 -08:00
Eric Tschetter
67feee3cd6 1) Indexes don't always have an index.drd file anymore 2013-01-23 18:36:52 -06:00
xvrl
55ae4c87dd timezone support in groupby query 2013-01-23 08:51:02 -08:00
xvrl
35058786d9 match query interval to granularity for this test 2013-01-23 08:50:43 -08:00
xvrl
86a6d112e3 proper groupby tests 2013-01-22 16:54:14 -08:00
Fangjin Yang
272d737517 cleaning up some interactions with RTR and workers 2013-01-22 16:21:38 -08:00
xvrl
f05c050c53 add test for timezone 2013-01-21 15:49:39 -08:00
Nelson Ray
94b72e8878 replace param BalancerCostAnalyzer getter with a factory 2013-01-21 15:32:29 -08:00
Fangjin Yang
bab9ee8827 Merge branch 'master' into killsegments 
Conflicts:
	merger/src/main/java/com/metamx/druid/merger/coordinator/http/IndexerCoordinatorNode.java
 2013-01-21 14:47:49 -08:00
Nelson Ray
2d7113b263 huge simplification of balancing code 2013-01-21 14:28:25 -08:00
Eric Tschetter
5b1e03530c 1) Fix some bugs found by external test suite 2013-01-16 21:06:57 -06:00
Fangjin Yang
21613bc73b initial commit to hard delete segments 2013-01-16 17:31:01 -08:00
Eric Tschetter
c8cb96b006 1) Remove vast majority of usages of IndexIO.mapDir() and deprecated it. IndexIO.loadIndex() is the new IndexIO.mapDir() 
2) Fix bug with IndexMerger and null columns
3) Add QueryableIndexIndexableAdapter so that QueryableIndexes can be merged
4) Adjust twitter example to have multiple values for each hash tag
5) Adjusted GroupByQueryEngine to just drop dimensions that don't exist instead of throwing an NPE
 2013-01-16 17:10:33 -06:00
Eric Tschetter
538d00e75e Merge branch 'master' of github.com:metamx/druid 2013-01-16 10:02:01 -06:00
Eric Tschetter
8b31d8db9f 1) Adjust IndexMerger to create convert the indexes it creates from the old format to the new. This is done quite sub-optimally, but it will work for now... 2013-01-16 10:01:46 -06:00
Fangjin Yang
7e074e8158 fix pom breakage 2013-01-15 12:04:12 -08:00
Nelson Ray
e847faf02f change name of MAX_SEGMENTS_TO_MOVE since it's not a final 2013-01-15 11:29:46 -08:00
Nelson Ray
ef9e077d9a remove getConfig method from DruidMaster 2013-01-14 16:01:29 -08:00
Eric Tschetter
998f0bf3c8 1) Better error message when aggregators want more memory than is available. 2013-01-14 18:00:51 -06:00
Nelson Ray
5f40dc6d9c put BalancerCostAnalyzer into params 2013-01-10 10:27:43 -08:00
Nelson Ray
617907d85b move empty server holder list check outside of loop 2013-01-08 16:12:16 -08:00
Nelson Ray
4549c3a8c3 add some finals, reorganize 2013-01-08 09:07:43 -08:00
Nelson Ray
2b4dbdde03 lots of changes 2013-01-07 17:28:53 -08:00
Nelson Ray
6fda5330fd finish making MAX_SEGMENTS_TO_MOVE a config item 2013-01-07 16:56:55 -08:00
Nelson Ray
ba82fe746a put MAX_SEGMENTS_TO_MOVE in the config 2013-01-07 16:28:27 -08:00
Eric Tschetter
4ef31de62d 1) Fix stray import, take 2... yay IntelliJ 2013-01-07 13:11:28 -08:00
Nelson Ray
d2ff88e923 add more stats to be emitted 2013-01-07 13:09:18 -08:00
Nelson Ray
a7553776fa make sampling of segments uniformly at random and add unit test 2013-01-07 12:51:51 -08:00
Eric Tschetter
b62bdf915b 1) Fix stray import... yay IntelliJ 2013-01-07 10:09:19 -08:00
Nelson Ray
2ea164acd9 get rid of extraneous analyzer.inits 2013-01-07 09:09:56 -08:00
Eric Tschetter
b184c46ae0 1) Introduce idea of value types and generic columns to the serialization format 
2) Create SegmentLoader interface for the loading of segments!
3) Setup serialization of new format and conversion function inside IndexIO
4) Make some adjustments to the twitter demo to make it make a bit more sense based on some email feedback from someone kicking the tires.
 2013-01-04 18:12:49 -06:00
Nelson Ray
402ddbf261 update 2013-01-04 11:20:06 -08:00
Nelson Ray
117cac2795 update 2013-01-03 14:12:10 -08:00
Nelson Ray
bb31fa7b68 update 2013-01-03 10:48:05 -08:00
xvrl
9627fdcd1c [maven-release-plugin] prepare for next development iteration 2013-01-03 10:09:52 -08:00
xvrl
f2c2d3f4cf [maven-release-plugin] prepare release druid-0.1.25 2013-01-03 10:09:43 -08:00
Nelson Ray
5441662cb8 Merge remote-tracking branch 'origin/master' into new_balancer 2013-01-03 09:10:48 -08:00
xvrl
a9b8d8f1ce fix end of month bug in PeriodDropRule as well 2013-01-02 16:10:23 -08:00
xvrl
0543322c56 tested timestamps should be deterministic 2013-01-02 14:31:51 -08:00
xvrl
19c3fd3363 make reference timestamp explicit for time based rules 2013-01-02 14:27:34 -08:00
xvrl
2a188996cd [maven-release-plugin] prepare for next development iteration 2012-12-31 14:56:10 -08:00
xvrl
6cdcc2becc [maven-release-plugin] prepare release druid-0.1.24 2012-12-31 14:56:03 -08:00
xvrl
d0290b1975 fix flawed interval overlap rule test 
fails around month end
 2012-12-31 14:52:42 -08:00
Nelson Ray
72758e1345 update 2012-12-31 13:55:48 -08:00
Nelson Ray
3eda9dc5f2 working balancercostanalyzer 2012-12-31 11:08:37 -08:00
Nelson Ray
366216f9f3 initial cost analyzer commit 2012-12-28 16:30:21 -08:00
Gian Merlino
5c1b2360d6 - Add optional query/wait metric to MetricsEmittingQueryRunner 
- Add MetricsEmittingExecutorService decorator, and use it on compute nodes
 2012-12-20 16:11:08 -08:00