Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,878 Commits 111 Branches 503 Tags 726 MiB
Commit Graph

1052 Commits

Author SHA1 Message Date
Fangjin Yang e48f6dd660 Merge pull request #1736 from gianm/additional-ingest-segment-timeline-test 
IngestSegmentFirehostFactoryTimelineTest for overshadowing of the middle of a segment.
 2015-09-17 14:42:29 -07:00
Gian Merlino 64e33b2bcb IngestSegmentFirehostFactoryTimelineTest for overshadowing of the middle of a segment. 2015-09-16 10:17:43 -07:00
Himanshu Gupta 74f4572bd4 Lazily deserialize "parser" to InputRowParser in DataSchema 
so that user hadoop related InputRowParsers are created only when needed
this allows overlord to accept a HadoopIndexTask with a hadoopy InputRowParser
and not fail because hadoopy InputRowParser might need hadoop libraries
 2015-09-16 10:58:13 -05:00
Charles Allen f5ed6e885c Merge pull request #1702 from himanshug/double_datasource_in_storage_dir 
do not have dataSource twice in path to segment storage on hdfs
 2015-09-15 14:00:35 -07:00
Nishant 4681ff22ed add task duration in response for completed tasks 2015-09-10 13:51:50 +05:30
Himanshu Gupta fe0233adf2 removing unused imports from HadoopIndexTask 2015-09-09 11:12:01 -05:00
Nishant 47aac991ec add null check for task context. 
make variable final
 2015-09-04 22:19:01 +05:30
Fangjin Yang 75a582974b Merge pull request #1639 from gianm/new-plumber 
New plumber
 2015-09-03 18:52:57 -07:00
Gian Merlino 062a47fba4 Modify Plumbers in these ways, 
1) Persist using Committer instead of Runnable. (Although the metadata object
   is ignored in this patch)

2) Remove the getSink method.

3) Plumbers are now responsible for time-based and hydrant-full-based periodic
   committing. (FireChief, RealtimeIndexTask, and IndexTask used to do this)
 2015-09-03 11:13:06 -07:00
Nishant 726326abc3 Add Task Context and ability to override task specific properties 
override javaOpts

fix compilation

review comments

Add Test for typecast

review comments - remove unused method.
 2015-09-03 23:36:32 +05:30
Gian Merlino 940e1aa3eb Replace funky imports with standard ones. 
1) Lots of Guava imports were not coming from the actual Guava
2) junit.framework.Assert should be org.junit.Assert
 2015-08-28 18:02:05 -07:00
Gian Merlino 414a6fb477 Fix overlapping segments in IngestSegmentFirehose, DatasourceInputFormat. 
Fixes #1678. IngestSegmentFirehose (and its users) need to remember which
windows of which segments should actually be read, based on a timeline.
 2015-08-28 07:32:41 -07:00
Himanshu Gupta 2e0dd1d792 adding UTs and addressing review comments to 
firehoseV2 addition to Realtime[Manager|Plumber],
essential segment metadata persist support,
kafka-simple-consumer-firehose extension patch
 2015-08-27 20:50:46 -05:00
lvjq 2237a8cf0f kafka 8 simple consumer firehose 2015-08-27 20:50:46 -05:00
Nishant b306739e9c fix convert segment task 
1) fix serde
2) fix wrong parameter being passed when creating subtask

remove sysout
 2015-08-27 11:34:41 +05:30
Charles Allen e38cf54bc8 Migrate TestDerbyConnector to a JUnit @Rule 2015-08-26 21:47:40 -07:00
Xavier Léauté fdb6a6651b Merge pull request #1669 from metamx/upgrade-dependencies 
Upgrade dependencies
 2015-08-25 21:30:22 -07:00
Xavier Léauté 5c19ffa98c Merge pull request #1663 from gianm/segment-insert-constraints 
TaskActionToolbox: Remove allowOlderVersions, lift interval constraint
 2015-08-25 18:11:46 -07:00
Xavier Léauté 51f6a9a2c9 update jackson to 2.6.1 2015-08-25 16:07:01 -07:00
Gian Merlino 33681525e3 TaskActionToolbox: Remove allowOlderVersions switch, lift interval constraint. 
allowOlderVersions has been stuck true for a while due to a bug (introduced in
566a3a61), but I think it's actually OK this way. I think it's reasonable to
expect tasks to choose versions in some way that makes sense, so long as they
don't choose one larger than their taskLock version. This is still verified.

The interval constraint was introduced to force tasks to break up their
segment insert lists into manageable chunks. They are already doing this, and
I think it's reasonable to expect them to do so without enforcement.

Lifting these constraints paves the way for transactional insertion of segments
that have varying versions and may be for varying intervals.
 2015-08-25 14:17:38 -07:00
Paul Otto 2301b60365 Add ability to provide taskResource for IndexTask. 2015-08-24 17:38:31 -07:00
Xavier Léauté 3b2e41e42a update for next release 2015-08-18 17:16:46 -07:00
Himanshu Gupta 15fa43dd43 changing DatasourcePathSpec, to get segment list, so that hadoop indexer uses overlord action to get list of segments and passes when running as an overlord task. and, uses metadata store directly when running as standalone hadoop indexer 
also, serialized list of segments is passed to DatasourcePathSpec so that hadoop classloader issues do not creep up
 2015-08-16 14:07:35 -05:00
Himanshu Gupta 4d4aa8bfc6 refactor IngestSegmentFirehoseFactory so that IngestSegmentFirehose becomes reusable 
Conflicts:
	indexing-service/src/main/java/io/druid/indexing/firehose/IngestSegmentFirehoseFactory.java
 2015-08-14 14:44:22 -05:00
Gian Merlino bc0c7dd65d Avoid the Hadoop objectMapper in the local IndexTask. Fixes #1545. 2015-08-11 10:40:53 -07:00
Charles Allen 1ddaa3fb33 Merge pull request #1592 from metamx/clean-test-files 
clean temporary files
 2015-08-03 11:47:20 -07:00
Nishant 2679efee7a clean temporary files 2015-08-03 23:32:58 +05:30
Fangjin Yang 6f65e6d3ef Merge pull request #1547 from pjain1/improve_overlord_test 
add test to OverlordResourceTest
 2015-07-28 07:35:48 -10:00
Parag Jain 2e1b617346 add more tests 2015-07-24 15:12:08 -05:00
Fangjin Yang 97242356b4 Merge pull request #1480 from guobingkun/kill_task_test 
Unit tests for KillTask and MetadataTaskStorage
 2015-07-20 16:31:45 -07:00
Xavier Léauté 4cfb00bc8a inrement version 2015-07-15 13:09:05 -07:00
Fangjin Yang 3f7ba58227 Merge pull request #1504 from metamx/fix-1447 
fix for #1447
 2015-07-14 08:50:08 -07:00
Himanshu e2ddfb7a1a Merge pull request #1511 from pjain1/remove_test 
remove flaky overlord test
 2015-07-13 18:38:34 -05:00
Parag Jain 59dec89f6a remove flaky overlord test 2015-07-13 15:32:12 -05:00
Himanshu 725086cc89 Merge pull request #1506 from gianm/realtime-plumber-nulls 
Consider null inputRows and parse errors as unparseable during realtime ingestion.
 2015-07-13 10:12:12 -05:00
Gian Merlino 9068bcd062 Consider null inputRows and parse errors as unparseable during realtime ingestion. 
Also, harmonize exception handling between the RealtimeIndexTask and the RealtimeManager.
Conditions other than null inputRows and parse errors bubble up in both.
 2015-07-11 20:40:03 -07:00
Himanshu cac722968e Merge pull request #1503 from metamx/fix-leaking-zk-nodes 
Fix leaking Status Path nodes in ZK
 2015-07-10 17:40:18 -05:00
Fangjin Yang 9f19e96658 Merge pull request #1477 from pjain1/overlord_test 
overlord and task master test
 2015-07-10 14:27:14 -07:00
Parag Jain 55c4fe64f3 overlord and task master test 2015-07-10 16:17:45 -05:00
Nishant 5fe27fe4ad fix for #1447 
fixes #1447
 2015-07-09 19:05:48 +05:30
Nishant 8d7a566bae Fix leaking Status Path nodes in ZK 
- remove  ZK status path nodes for workers after they are removed
 2015-07-09 17:20:09 +05:30
Charles Allen c0b60c0d2f I'm not your mom, indexing-service/test... cleanup after yourself 2015-07-01 15:00:09 -07:00
Bingkun Guo 282a0f9760 Unit tests for KillTask and MetadataTaskStorage 2015-06-29 17:55:41 -05:00
Himanshu b5b9ca1446 Merge pull request #1470 from pjain1/rtindex_test 
Realtime Index Task test
 2015-06-29 16:51:35 -05:00
Parag Jain 284b80b09e Realtime Index Task test 2015-06-29 09:52:41 -05:00
Himanshu 4a83a22f8c Merge pull request #1445 from metamx/JSWorkerSelectStrategy 
JavaScript Worker Select Strategy
 2015-06-22 17:19:13 -05:00
nishant fb4052d577 JavaScript Worker Select Strategy 
this PR adds a JavaScriptWorkerSelectStrategy which allows defining
arbitrary logic for selecting workers to run task using a JavaScript
function.

This gives users full control to implement complex worker selection
strategies based on task attributes.

more tests and a complex javascript config

fix for java8 modify for nashorn compatibility
 2015-06-20 02:01:34 +05:30
Xavier Léauté 0a5bb909a2 [maven-release-plugin] prepare for next development iteration 2015-06-18 17:35:19 -07:00
Xavier Léauté 59c6b2b279 [maven-release-plugin] prepare release druid-0.8.0-rc1 2015-06-18 17:35:14 -07:00
Charles Allen acc0a3fbf7 Add jitter to the retries for RemoteTaskActionClient 2015-06-12 17:43:25 -07:00