Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
5,913 Commits 113 Branches 506 Tags 732 MiB
Commit Graph

1062 Commits

Author SHA1 Message Date
Charles Allen d2e400f063 Merge pull request #1740 from metamx/validate-locks 
fix #1715
 2015-09-29 09:38:42 -07:00
Xavier Léauté 25bbc0b923 Merge pull request #1778 from gianm/redirect-fixes 
Redirect fixes
 2015-09-25 09:54:48 -07:00
Gian Merlino 348172203f OverlordRedirectInfo: Fix ability to detect that there is no leader. 2015-09-25 09:30:09 -07:00
Parag Jain b630720164 fail task if finishjob throws any exception 
add realtime task failure test
 2015-09-25 10:55:45 -05:00
Fangjin Yang aa9d90355e Merge pull request #1772 from gianm/fix-overlord-startup 
RemoteTaskRunner: Fix for starting an overlord before any workers ever existed.
 2015-09-24 21:55:03 -07:00
Gian Merlino 63bf021077 RemoteTaskRunner: Fix for starting an overlord before any workers ever existed. 2015-09-24 21:15:36 -07:00
Himanshu Gupta 6e550d5346 update doc about aggregation field in merge task and a null check 2015-09-24 22:25:07 -05:00
Nishant b638400acb fix #1715 
fixes #1715
- TaskLockBox has a set of active tasks
- lock requests throws exception for if they are from a task not in
active task set.
- TaskQueue is responsible for updating the active task set on
tasklockbox

fix #1715

fixes #1715
- TaskLockBox has a set of active tasks
- lock requests throws exception for if they are from a task not in
active task set.
- TaskQueue is responsible for updating the active task set on
tasklockbox

review comment

remove duplicate line

use ISE instead

organise imports
 2015-09-24 10:06:50 +05:30
Himanshu 61b0743943 Merge pull request #1748 from metamx/forkingJavaOptionsWithQuotes 
Allow ForkingTaskRunner javaOpts to have quoted arguments which contain spaces
 2015-09-21 21:03:00 -05:00
Charles Allen 465035e531 Allow ForkingTaskRunner javaOpts to have quoted arguments which contain spaces 2015-09-21 17:32:27 -07:00
Fangjin Yang e48f6dd660 Merge pull request #1736 from gianm/additional-ingest-segment-timeline-test 
IngestSegmentFirehostFactoryTimelineTest for overshadowing of the middle of a segment.
 2015-09-17 14:42:29 -07:00
Gian Merlino 64e33b2bcb IngestSegmentFirehostFactoryTimelineTest for overshadowing of the middle of a segment. 2015-09-16 10:17:43 -07:00
Himanshu Gupta 74f4572bd4 Lazily deserialize "parser" to InputRowParser in DataSchema 
so that user hadoop related InputRowParsers are created only when needed
this allows overlord to accept a HadoopIndexTask with a hadoopy InputRowParser
and not fail because hadoopy InputRowParser might need hadoop libraries
 2015-09-16 10:58:13 -05:00
Charles Allen f5ed6e885c Merge pull request #1702 from himanshug/double_datasource_in_storage_dir 
do not have dataSource twice in path to segment storage on hdfs
 2015-09-15 14:00:35 -07:00
Nishant 4681ff22ed add task duration in response for completed tasks 2015-09-10 13:51:50 +05:30
Himanshu Gupta fe0233adf2 removing unused imports from HadoopIndexTask 2015-09-09 11:12:01 -05:00
Nishant 47aac991ec add null check for task context. 
make variable final
 2015-09-04 22:19:01 +05:30
Fangjin Yang 75a582974b Merge pull request #1639 from gianm/new-plumber 
New plumber
 2015-09-03 18:52:57 -07:00
Gian Merlino 062a47fba4 Modify Plumbers in these ways, 
1) Persist using Committer instead of Runnable. (Although the metadata object
   is ignored in this patch)

2) Remove the getSink method.

3) Plumbers are now responsible for time-based and hydrant-full-based periodic
   committing. (FireChief, RealtimeIndexTask, and IndexTask used to do this)
 2015-09-03 11:13:06 -07:00
Nishant 726326abc3 Add Task Context and ability to override task specific properties 
override javaOpts

fix compilation

review comments

Add Test for typecast

review comments - remove unused method.
 2015-09-03 23:36:32 +05:30
Gian Merlino 940e1aa3eb Replace funky imports with standard ones. 
1) Lots of Guava imports were not coming from the actual Guava
2) junit.framework.Assert should be org.junit.Assert
 2015-08-28 18:02:05 -07:00
Gian Merlino 414a6fb477 Fix overlapping segments in IngestSegmentFirehose, DatasourceInputFormat. 
Fixes #1678. IngestSegmentFirehose (and its users) need to remember which
windows of which segments should actually be read, based on a timeline.
 2015-08-28 07:32:41 -07:00
Himanshu Gupta 2e0dd1d792 adding UTs and addressing review comments to 
firehoseV2 addition to Realtime[Manager|Plumber],
essential segment metadata persist support,
kafka-simple-consumer-firehose extension patch
 2015-08-27 20:50:46 -05:00
lvjq 2237a8cf0f kafka 8 simple consumer firehose 2015-08-27 20:50:46 -05:00
Nishant b306739e9c fix convert segment task 
1) fix serde
2) fix wrong parameter being passed when creating subtask

remove sysout
 2015-08-27 11:34:41 +05:30
Charles Allen e38cf54bc8 Migrate TestDerbyConnector to a JUnit @Rule 2015-08-26 21:47:40 -07:00
Xavier Léauté fdb6a6651b Merge pull request #1669 from metamx/upgrade-dependencies 
Upgrade dependencies
 2015-08-25 21:30:22 -07:00
Xavier Léauté 5c19ffa98c Merge pull request #1663 from gianm/segment-insert-constraints 
TaskActionToolbox: Remove allowOlderVersions, lift interval constraint
 2015-08-25 18:11:46 -07:00
Xavier Léauté 51f6a9a2c9 update jackson to 2.6.1 2015-08-25 16:07:01 -07:00
Gian Merlino 33681525e3 TaskActionToolbox: Remove allowOlderVersions switch, lift interval constraint. 
allowOlderVersions has been stuck true for a while due to a bug (introduced in
566a3a61), but I think it's actually OK this way. I think it's reasonable to
expect tasks to choose versions in some way that makes sense, so long as they
don't choose one larger than their taskLock version. This is still verified.

The interval constraint was introduced to force tasks to break up their
segment insert lists into manageable chunks. They are already doing this, and
I think it's reasonable to expect them to do so without enforcement.

Lifting these constraints paves the way for transactional insertion of segments
that have varying versions and may be for varying intervals.
 2015-08-25 14:17:38 -07:00
Paul Otto 2301b60365 Add ability to provide taskResource for IndexTask. 2015-08-24 17:38:31 -07:00
Xavier Léauté 3b2e41e42a update for next release 2015-08-18 17:16:46 -07:00
Himanshu Gupta 15fa43dd43 changing DatasourcePathSpec, to get segment list, so that hadoop indexer uses overlord action to get list of segments and passes when running as an overlord task. and, uses metadata store directly when running as standalone hadoop indexer 
also, serialized list of segments is passed to DatasourcePathSpec so that hadoop classloader issues do not creep up
 2015-08-16 14:07:35 -05:00
Himanshu Gupta 4d4aa8bfc6 refactor IngestSegmentFirehoseFactory so that IngestSegmentFirehose becomes reusable 
Conflicts:
	indexing-service/src/main/java/io/druid/indexing/firehose/IngestSegmentFirehoseFactory.java
 2015-08-14 14:44:22 -05:00
Gian Merlino bc0c7dd65d Avoid the Hadoop objectMapper in the local IndexTask. Fixes #1545. 2015-08-11 10:40:53 -07:00
Charles Allen 1ddaa3fb33 Merge pull request #1592 from metamx/clean-test-files 
clean temporary files
 2015-08-03 11:47:20 -07:00
Nishant 2679efee7a clean temporary files 2015-08-03 23:32:58 +05:30
Fangjin Yang 6f65e6d3ef Merge pull request #1547 from pjain1/improve_overlord_test 
add test to OverlordResourceTest
 2015-07-28 07:35:48 -10:00
Parag Jain 2e1b617346 add more tests 2015-07-24 15:12:08 -05:00
Fangjin Yang 97242356b4 Merge pull request #1480 from guobingkun/kill_task_test 
Unit tests for KillTask and MetadataTaskStorage
 2015-07-20 16:31:45 -07:00
Xavier Léauté 4cfb00bc8a inrement version 2015-07-15 13:09:05 -07:00
Fangjin Yang 3f7ba58227 Merge pull request #1504 from metamx/fix-1447 
fix for #1447
 2015-07-14 08:50:08 -07:00
Himanshu e2ddfb7a1a Merge pull request #1511 from pjain1/remove_test 
remove flaky overlord test
 2015-07-13 18:38:34 -05:00
Parag Jain 59dec89f6a remove flaky overlord test 2015-07-13 15:32:12 -05:00
Himanshu 725086cc89 Merge pull request #1506 from gianm/realtime-plumber-nulls 
Consider null inputRows and parse errors as unparseable during realtime ingestion.
 2015-07-13 10:12:12 -05:00
Gian Merlino 9068bcd062 Consider null inputRows and parse errors as unparseable during realtime ingestion. 
Also, harmonize exception handling between the RealtimeIndexTask and the RealtimeManager.
Conditions other than null inputRows and parse errors bubble up in both.
 2015-07-11 20:40:03 -07:00
Himanshu cac722968e Merge pull request #1503 from metamx/fix-leaking-zk-nodes 
Fix leaking Status Path nodes in ZK
 2015-07-10 17:40:18 -05:00
Fangjin Yang 9f19e96658 Merge pull request #1477 from pjain1/overlord_test 
overlord and task master test
 2015-07-10 14:27:14 -07:00
Parag Jain 55c4fe64f3 overlord and task master test 2015-07-10 16:17:45 -05:00
Nishant 5fe27fe4ad fix for #1447 
fixes #1447
 2015-07-09 19:05:48 +05:30