Xavier Léauté
9ec7c71e0f
fix compilation error with updated druid-api
2014-05-19 14:06:23 -07:00
fjy
1100d2f2a1
rename configs to make a bit more sense
2014-05-06 14:52:50 -07:00
fjy
b6fb4245aa
Merge branch 'master' into new-schema
...
Conflicts:
indexing-hadoop/src/main/java/io/druid/indexer/HadoopDriverConfig.java
indexing-hadoop/src/main/java/io/druid/indexer/HadoopDruidIndexerConfig.java
indexing-hadoop/src/main/java/io/druid/indexer/HadoopDruidIndexerConfigBuilder.java
pom.xml
server/src/main/java/io/druid/segment/realtime/RealtimeManager.java
server/src/main/java/io/druid/segment/realtime/firehose/EventReceiverFirehoseFactory.java
2014-05-06 14:32:51 -07:00
Gian Merlino
bdf9e74a3b
Allow config-based overriding of hadoop job properties.
2014-05-06 09:11:31 -07:00
fjy
f9523274ac
remove extra println
2014-05-01 15:06:51 -07:00
nishantmonu51
5137031304
use same logic for compression
...
Use same logic for compression across creating files, reading from
files, and checking file existence
2014-05-01 15:20:47 +05:30
nishantmonu51
728f1e8ee3
fix exists check with compression
2014-05-01 15:01:10 +05:30
nishantmonu51
01e84f10b7
add the checks again.
...
removing these checks breaks when there is no data for any interval
2014-05-01 14:35:09 +05:30
fjy
76e0a48527
Merge branch 'master' into new-schema
...
Conflicts:
indexing-hadoop/src/main/java/io/druid/indexer/DbUpdaterJob.java
indexing-hadoop/src/test/java/io/druid/indexer/HadoopDruidIndexerConfigTest.java
indexing-service/src/main/java/io/druid/indexing/common/task/HadoopIndexTask.java
server/src/main/java/io/druid/segment/realtime/plumber/RealtimePlumber.java
server/src/main/java/io/druid/segment/realtime/plumber/RealtimePlumberSchool.java
2014-04-25 14:03:28 -07:00
fjy
2d1f33e59f
Merge pull request #500 from metamx/batch-ingestion-fixes
...
Batch ingestion fixes
2014-04-22 17:59:24 -06:00
nishantmonu51
357bbf5127
add all the shard specs
2014-04-23 05:23:11 +05:30
nishantmonu51
625a5418d2
minor fix
2014-04-23 05:05:51 +05:30
nishantmonu51
1ca61237c1
review comments- use final variables
2014-04-23 03:33:28 +05:30
nishantmonu51
0d8c1ffe54
review comments and add partitioner
2014-04-23 03:30:30 +05:30
nishantmonu51
ea4a80e8d2
Add serde test for shardCount
2014-04-23 00:24:08 +05:30
nishantmonu51
e920cec5d0
remove unused import
2014-04-23 00:13:30 +05:30
nishantmonu51
0748eabe9b
batch ingestion fixes
...
1) Fix path when mapped output is compressed
2) Add number of reducers to the determine hashed partitions job
manually
3) Add a way to disable determine partitions and specify shardCount in
HashedPartitionsSpec
2014-04-23 00:05:08 +05:30
Crystark
40a6804192
Support for postgresql
...
I think it was the last request using 'end' missing the postgresql support.
2014-04-07 17:37:03 +02:00
fjy
2adcf07f5f
Merge branch 'master' into new-schema
...
Conflicts:
indexing-hadoop/src/main/java/io/druid/indexer/DetermineHashedPartitionsJob.java
indexing-service/src/main/java/io/druid/indexing/common/task/RealtimeIndexTask.java
indexing-service/src/test/java/io/druid/indexing/common/task/TaskSerdeTest.java
processing/src/test/java/io/druid/segment/TestIndex.java
server/src/main/java/io/druid/segment/realtime/RealtimeManager.java
server/src/main/java/io/druid/segment/realtime/plumber/RealtimePlumberSchool.java
2014-03-17 10:59:31 -07:00
nishantmonu51
4ec1959c30
Use druid implementation of HyperLogLog
...
remove dependency on clear spring analytics
2014-03-07 00:06:40 +05:30
fjy
5db00afb37
clean up and default values
2014-03-04 14:38:27 -08:00
fjy
c4c4d80336
make local testing pass
2014-03-03 14:52:43 -08:00
fjy
46b9ac78e7
Merge branch 'master' into new-schema
...
Conflicts:
indexing-hadoop/src/test/java/io/druid/indexer/HadoopDruidIndexerConfigTest.java
pom.xml
publications/whitepaper/druid.pdf
publications/whitepaper/druid.tex
2014-03-03 14:48:15 -08:00
fjy
13c7f1c7b1
remove dead code
2014-02-27 15:52:19 -08:00
fjy
bf2ddda897
unit tests passing after more refactoring
2014-02-27 15:21:09 -08:00
nishantmonu51
5e0d418b4b
fix determine partitions partitioner to work in local mode
2014-02-26 16:31:42 +05:30
nishantmonu51
1ed5254d5b
improvements
...
1) Number of reducers use 1 only when intervals are to be determined
2) Read only useful bytes from BytesWritable
2014-02-26 02:51:45 +05:30
nishantmonu51
8af63005a6
refactor randomPartitionsSpec to hashedPartitionsSpec
...
refactor to a more appropriate name
2014-02-25 03:07:31 +05:30
fjy
5d2367f0fd
unit tests pass at this point
2014-02-20 15:52:12 -08:00
fjy
20cac8c506
not compiling yet but close
2014-02-19 15:54:27 -08:00
fjy
4b7c76762d
unit tests passingn at this point, finished rt port maybe
2014-02-18 15:14:38 -08:00
nishantmonu51
fde7269c86
check published segments before the intermediate files are cleaned up
2014-02-15 04:30:28 +05:30
fjy
3979eb270c
Revert "Revert "Merge branch 'determine-partitions-improvements'""
...
This reverts commit 189b3e2b9bc3c79da9c98b805df624f024109469.
2014-02-14 12:58:56 -08:00
fjy
a8c4362d72
rejiggering druid api
2014-02-14 12:57:52 -08:00
fjy
189b3e2b9b
Revert "Merge branch 'determine-partitions-improvements'"
...
This reverts commit 7ad228ceb580f9ca757e91b019bd8931f710f87a, reversing
changes made to 9c55e2b7794d37cd8713bbc0465aaad23e23472d.
2014-02-14 12:47:34 -08:00
nishantmonu51
48d0c37f98
documentation for random partition spec
2014-02-05 15:30:44 +05:30
nishantmonu51
bacc72415f
correct locking and partitionsSpec
2014-02-05 03:17:47 +05:30
nishantmonu51
569452121e
fix partitioner for loca mode
2014-01-31 21:59:17 +05:30
nishantmonu51
82b748ad43
review comments
2014-01-31 20:19:33 +05:30
nishantmonu51
97e5d68635
determine intervals working with determine partitions
2014-01-31 19:04:52 +05:30
nishantmonu51
5fd76067cd
remove logging and use new determine partition job
2014-01-31 13:51:38 +05:30
nishantmonu51
7ca87d59df
Determine partitions using cardinality
2014-01-31 00:49:11 +05:30
fjy
f898c29e20
fix batch indexing and prepare for next release
2014-01-17 15:52:04 -08:00
fjy
3b17c4c03c
a whole bunch of docs and fixes
2014-01-13 18:01:56 -08:00
fjy
1ecc94cfb6
another attempt at index task
2014-01-10 17:56:22 -08:00
Hagen Rother
52746b8ea6
fix hadoop intake's parser exception catching (was too specific)
2013-12-19 07:04:47 +01:00
fjy
a1c09df17f
make the hadoop index task work again
2013-10-16 09:45:17 -07:00
cheddar
c47fe202c7
Fix HadoopDruidIndexer to work with the new way of things
...
There are multiple and sundry changes in here.
First, "HadoopDruidIndexer" has been split into two pieces, (1) CliHadoop which pulls the hadoop version and builds up the right classpath with the proper hadoop version to run the indexer and (2) CliInternalHadoopIndexer which actually runs the indexer.
In order to work around a bunch of jets3t version conflicts with Hadoop and Druid, I needed to extract the S3 deep storage stuff into its own module. I then also moved the HDFS stuff into its own module so that I could eliminate the dependency on Hadoop for druid-server.
In doing these changes, I wanted to make the extensions buildable with only the druid-api jar, so a few other things had to move out of Druid and into druid-api. They are all API-level things, however, so they really belong in druid-api instead.
Lastly, I removed the druid-realtime module and put it all in druid-server.
2013-10-09 15:15:44 -05:00
fjy
a79ad7bab4
make dynamic master resource configuration work again
2013-09-27 15:00:40 -07:00
fjy
8bc56daa66
fix things up according to code review comments
2013-09-26 11:35:45 -07:00
