Apache/druid
1
0
Fork 0
mirror of https://github.com/apache/druid.git synced 2025-02-10 12:05:00 +00:00
Code Issues Packages Projects Releases Wiki Activity
8,235 Commits 116 Branches 507 Tags
Commit Graph

1362 Commits

Author SHA1 Message Date
Charles Allen
9dff0e5dbd Add hadoop classloader isolation tests for HadoopTask 2016-02-19 11:15:53 -08:00
Fangjin Yang
ddf913d626 Merge pull request #2508 from gianm/ftr-shutdown-logging 
ForkingTaskRunner: Better logging during orderly shutdown.
 2016-02-19 10:02:24 -08:00
Gian Merlino
c0c6cf77fa ForkingTaskRunner: Better logging during orderly shutdown. 2016-02-19 09:17:16 -08:00
Gian Merlino
243ac5399b Harmonize realtime indexing loop across the task and standalone nodes. 
- Both now catch ParseExceptions on plumber.add (see https://groups.google.com/d/topic/druid-user/wmiRDvx2RvM/discussion)
- Standalone now treats IndexSizeExceededException as fatal (previously only the task did)
 2016-02-19 07:34:15 -08:00
Charles Allen
87752be740 Make HadoopTasks's classloader a single one 2016-02-18 20:58:09 -08:00
Andrés Gomez
07d714b1b5 Fixed equal distribution strategy when exist disable middleManager with same currCapacityUsed. 2016-02-17 08:40:42 +01:00
Himanshu
5779b32742 Merge pull request #2439 from metamx/fix2435 
Make QuotableWhiteSpaceSplitter able to take JSON
 2016-02-11 13:14:43 -06:00
Charles Allen
40ade32a1f Fix dependencies. 
* Don't put druid****selfcontained.jar at the end of the hadoop isolated classpath
* Add `<scope>provided</scope>` to prevent repeated dependency inclusion in the extension directories
 2016-02-11 07:30:14 -08:00
Charles Allen
3a6452c6d4 Make QuotableWhiteSpaceSplitter able to take json 
* Fixes #2435
 2016-02-10 16:42:14 -08:00
Xavier Léauté
91f23583f5 Merge pull request #2436 from gianm/mm-less-suppressey 
Harmonize znode writing code in RTR and Worker.
 2016-02-10 16:11:30 -08:00
Gian Merlino
fa92b77f5a Harmonize znode writing code in RTR and Worker. 
- Throw most exceptions rather than suppressing them, which should help
  detect problems. Continue suppressing exceptions that make sense to
  suppress.
- Handle payload length checks consistently, and improve error message.
- Remove unused WorkerCuratorCoordinator.announceTaskAnnouncement method.
- Max znode length should be int, not long.
- Add tests.
 2016-02-10 14:52:00 -08:00
Charles Allen
2bde8b1d68 Make hadoop classpath isolation more explicit 
* Fixes #2428
 2016-02-10 12:09:17 -08:00
Charles Allen
a0728fa854 Allow ScalingStats to be null 
* Fixes #2378
 2016-02-02 18:01:01 -08:00
Parag Jain
7853a9cc41 clean up TaskLifecycleTest 2016-01-31 11:19:20 -06:00
Gian Merlino
5fd4b79373 RealtimeIndexTask: Fix NPE caused by calling stopGracefully before a firehose had been connected. 2016-01-29 11:20:23 -08:00
Gian Merlino
c4fde52160 Fix 'graceful shutdown aborted' log message in ThreadPoolTaskRunner. 2016-01-29 11:07:17 -08:00
Nishant
dcb7830330 Merge pull request #984 from drcrallen/thread-priority-rebase 
Use thread priorities. (aka set `nice` values for background-like tasks)
 2016-01-21 15:02:34 +05:30
Charles Allen
66e74b1a63 Minor field name change in RemoteTaskRunnerFactory to be more descriptive 
* Addresses https://github.com/druid-io/druid/pull/2309#discussion_r50335081
 2016-01-20 17:43:20 -08:00
Charles Allen
3152d08844 Fix overlord scheduled executor injection 
* Fixes https://github.com/druid-io/druid/issues/2308
 2016-01-20 14:16:14 -08:00
Charles Allen
2e1d6aaf3d Use thread priorities. (aka set nice values for background-like tasks) 
* Defaults the thread priority to java.util.Thread.NORM_PRIORITY in io.druid.indexing.common.task.AbstractTask
 * Each exec service has its own Task Factory which is assigned a priority for spawned task. Therefore each priority class has a unique exec service
 * Added priority to tasks as taskPriority in the task context. <0 means low, 0 means take default, >0 means high. It is up to any particular implementation to determine how to handle these numbers
 * Add options to ForkingTaskRunner
    * Add "-XX:+UseThreadPriorities" default option
    * Add "-XX:ThreadPriorityPolicy=42" default option
 * AbstractTask - Removed unneded @JsonIgnore on priority
 * Added priority to RealtimePlumber executors. All sub-executors (non query runners) get Thread.MIN_PRIORITY
 * Add persistThreadPriority and mergeThreadPriority to realtime tuning config
 2016-01-20 14:00:31 -08:00
Nishant
ac6c90e657 Merge pull request #1953 from metamx/taskRunnerResourceManagement 
Move resource managemnt to be the responsibility of the TaskRunner
 2016-01-20 22:27:47 +05:30
Jonathan Wei
df2906a91c Merge pull request #2290 from gianm/index-merger-v9-stuff 
Respect buildV9Directly in PlumberSchools, so it works on standalone realtime.
 2016-01-19 13:04:00 -08:00
Fangjin Yang
0c31f007fc Merge pull request #1728 from himanshug/aggregators_in_segment_metadata 
Store AggregatorFactory[] in segment metadata
 2016-01-19 12:55:49 -08:00
Himanshu
fe841fd961 Merge pull request #2118 from guobingkun/fix_segment_loading 
Fix loading segment for historical
 2016-01-19 14:25:48 -06:00
Himanshu Gupta
a99aef29a1 adding aggregators to segment metadata 2016-01-19 14:23:39 -06:00
Gian Merlino
1dcf22edb7 Respect buildV9Directly in PlumberSchools, so it works on standalone realtime nodes. 
Also parameterize some tests to run with/without buildV9Directly:

- IndexGeneratorJobTest
- RealtimeIndexTaskTest
- RealtimePlumberSchoolTest
 2016-01-19 12:15:06 -08:00
Bingkun Guo
c4ad50f92c Fix loading segment for historical 
Historical will drop a segment that shouldn't be dropped in the following scenario:
Historical node tried to load segmentA, but failed with SegmentLoadingException,
then ZkCoordinator called removeSegment(segmentA, blah) to schedule a runnable that would drop segmentA by deleting its files. Now, before that runnable executed, another LOAD request was sent to this historical, this time historical actually succeeded on loading segmentA and announced it. But later on, the scheduled drop-of-segment runnable started executing and removed the segment files, while historical is still announcing segmentA.
 2016-01-19 10:29:49 -06:00
Himanshu Gupta
164b0aad7a removing Map<String,Object> segmentMetadata from methods in Index[Maker/Merger] and using Metadata class 
instead of a Map to store segment metadata
 2016-01-18 22:03:46 -06:00
Kurt Young
82ff98c2bf add config for build v9 directly and update docs 2016-01-16 11:26:34 +08:00
Charles Allen
976d4c965b Move resource managemnt to be the responsibility of the TaskRunner 2016-01-13 10:38:22 -08:00
Himanshu
82bdfbbbf1 Merge pull request #2155 from metamx/taskConfigTmpdir 
Make TaskConfig pull from java.io.tmpdir
 2016-01-05 13:58:39 -06:00
Nishant
45f402f22f increase timeout 
tune timeouts
 2016-01-05 19:06:04 +05:30
Charles Allen
e18301d99c Make TaskConfig pull from java.io.tmpdir 
* Also makes paths built off of java.nio.file.Paths instead of String.format
 2016-01-04 10:17:08 -08:00
fjy
b5c094d951 Fixes #2180 2016-01-01 16:56:41 -08:00
Nishant
b68265399c Merge pull request #2168 from druid-io/remove-indexmaker 
Remove IndexMaker
 2015-12-30 12:24:29 +05:30
Nishant
df893dbaf8 Merge pull request #2141 from gianm/fix-restoring-realtime 
Fix some problems with restoring
 2015-12-30 10:44:45 +05:30
Fangjin Yang
7ffa706655 Merge pull request #2152 from metamx/add-taskId 
Add taskId to realtimeMetrics
 2015-12-29 10:33:40 -08:00
fjy
38b0f1fbc2 fix transient failures in unit tests 2015-12-28 20:03:30 -08:00
fjy
faf421726b remove IndexMaker 2015-12-28 14:19:02 -08:00
Fangjin Yang
8cb52bddd8 Merge pull request #2140 from navis/fix-sporadic-testfail4 
Fix sporadic fail of RemoteTaskRunnerTest#testWorkerRemoved
 2015-12-27 14:55:49 -08:00
Fangjin Yang
9aa62e4631 Merge pull request #2154 from navis/fix-testfail-WorkerTaskMonitorTest 
Fix sporadic fail of WorkerTaskMonitorTest#testRunTask
 2015-12-23 20:52:33 -08:00
navis.ryu
a8f6c6110d Fix sporadic fail of WorkerTaskMonitorTest#testRunTask 2015-12-24 02:31:30 +09:00
navis.ryu
2c3c4a3f8f Another try to fix xxServerViewTests 2015-12-24 02:13:40 +09:00
Nishant
978a3fd8ae Add taskId to realtimeMetrics 
Add task Id to Realtime Metrics
 2015-12-23 18:05:25 +05:30
Gian Merlino
32edd1538d RealtimeIndexTask: Fix a couple of problems with restoring. 
- Shedding locks at startup is bad, we actually want to keep them. Stop doing that.
- stopGracefully now interrupts the run thread if had started running finishJob. This avoids
  waiting for handoff unnecessarily.
 2015-12-22 16:04:47 -08:00
Gian Merlino
f4ce2b9bc5 TaskLockbox: Consider active tasks active even if they have no locks. 2015-12-22 16:04:16 -08:00
Gian Merlino
bad270b6c4 druid.indexer.task.restoreTasksOnRestart configuration. 2015-12-22 10:59:15 -08:00
navis.ryu
8a179fc273 Fix sporadic fail of RemoteTaskRunnerTest#testWorkerRemoved 2015-12-22 14:33:37 +09:00
Himanshu Gupta
5e178499e8 trying to fix transient errors in testRealtimeIndexTask() by increasing overall timeout and unlimited wait for segment publish 2015-12-21 00:11:20 -06:00
Fangjin Yang
14229ba0f2 Merge pull request #1922 from metamx/jsonIgnoresFinalFields 
Change DefaultObjectMapper to NOT overwrite final fields unless explicitly asked to
 2015-12-18 15:38:32 -08:00