Nishant
dcb7830330
Merge pull request #984 from drcrallen/thread-priority-rebase
...
Use thread priorities. (aka set `nice` values for background-like tasks)
2016-01-21 15:02:34 +05:30
Fangjin Yang
500c11a35a
Merge pull request #2311 from metamx/commentsIn2309
...
Minor field name change in RemoteTaskRunnerFactory to be more descriptive
2016-01-20 19:27:49 -08:00
Charles Allen
66e74b1a63
Minor field name change in RemoteTaskRunnerFactory to be more descriptive
...
* Addresses https://github.com/druid-io/druid/pull/2309#discussion_r50335081
2016-01-20 17:43:20 -08:00
Gian Merlino
5a932d28c1
Merge pull request #2288 from tubemogul/index-merge-bug2
...
Null check in IncrementalIndexAdapter.getDimValueLookup()
2016-01-20 17:07:15 -08:00
Charles Allen
2a69a58570
Merge pull request #2149 from binlijin/master
...
Do persist IncrementalIndex in another thread in IndexGeneratorReducer
2016-01-20 17:06:42 -08:00
Fangjin Yang
067baf10f2
Merge pull request #2310 from metamx/testFor2308
...
Add basic tests to some CliXXX classes
2016-01-20 17:00:06 -08:00
Fangjin Yang
996c1173c6
Merge pull request #2223 from navis/besteffort-split-locations
...
Best effort to find locations for input splits
2016-01-20 16:53:43 -08:00
Charles Allen
eddc18ecf2
Add basic tests to some CliXXX classes
2016-01-20 16:19:15 -08:00
Charles Allen
551d08da2a
Merge pull request #2309 from metamx/fix2308
...
Fix overlord scheduled executor injection
2016-01-20 16:04:57 -08:00
Charles Allen
3152d08844
Fix overlord scheduled executor injection
...
* Fixes https://github.com/druid-io/druid/issues/2308
2016-01-20 14:16:14 -08:00
Charles Allen
2e1d6aaf3d
Use thread priorities. (aka set `nice` values for background-like tasks)
...
* Defaults the thread priority to java.util.Thread.NORM_PRIORITY in io.druid.indexing.common.task.AbstractTask
* Each exec service has its own Task Factory which is assigned a priority for spawned task. Therefore each priority class has a unique exec service
* Added priority to tasks as taskPriority in the task context. <0 means low, 0 means take default, >0 means high. It is up to any particular implementation to determine how to handle these numbers
* Add options to ForkingTaskRunner
* Add "-XX:+UseThreadPriorities" default option
* Add "-XX:ThreadPriorityPolicy=42" default option
* AbstractTask - Removed unneded @JsonIgnore on priority
* Added priority to RealtimePlumber executors. All sub-executors (non query runners) get Thread.MIN_PRIORITY
* Add persistThreadPriority and mergeThreadPriority to realtime tuning config
2016-01-20 14:00:31 -08:00
Fangjin Yang
0c5f4b947c
Merge pull request #2301 from metamx/fix-2299_2
...
fix reference counting for segments
2016-01-20 11:21:35 -08:00
Nishant
61aca6f9cc
remove wrong checks
...
sink never have null hydrants and hydrants never have null adapters
2016-01-20 23:43:53 +05:30
Fangjin Yang
695f107870
Merge pull request #2302 from metamx/lowerCaseGranPathTest
...
Make GranularityPathSpecTest check with lower-case enums
2016-01-20 09:18:06 -08:00
Fangjin Yang
6f4dfbc8d1
Merge pull request #2296 from loganlinn/loganlinn-dataGranularity-type-doc
...
Update batch-ingestion.md
2016-01-20 08:58:49 -08:00
Nishant
ac6c90e657
Merge pull request #1953 from metamx/taskRunnerResourceManagement
...
Move resource managemnt to be the responsibility of the TaskRunner
2016-01-20 22:27:47 +05:30
Charles Allen
3c5ca3a5f2
Make GranularityPathSpecTest check with lower-case enums
2016-01-20 08:35:13 -08:00
Jaehong Choi
7132428bba
throw IAE when baseDir is null in LocalFireHose
2016-01-21 01:27:32 +09:00
Nishant
59ea186af7
fix reference counting for segments
2016-01-20 17:24:21 +05:30
Michael Schiff
50ceec78a2
null check in IncrementalIndexAdapter.getDimValueLookup()
2016-01-19 23:19:28 -08:00
Jonathan Wei
fc09929503
Merge pull request #2293 from jon-wei/index_merger_test_fix
...
Consolidate IndexMergerTest and IndexMergerV9Test
2016-01-19 17:42:11 -08:00
Logan Linn
c3bdaefe1f
Update batch-ingestion.md
...
Fix documented type of the `dataGranularity` config
2016-01-19 17:20:47 -08:00
binlijin
8e43e2c446
Do persist IncrementalIndex in another thread in IndexGeneratorReducer
2016-01-20 09:20:09 +08:00
jon-wei
bc1e9b27c8
Consolidate IndexMergerTest and IndexMergerV9Test
2016-01-19 16:28:35 -08:00
Fangjin Yang
cb8f714f82
Merge pull request #2006 from jon-wei/inherit_dim_order
...
Preserve dimension order across indexes during ingestion
2016-01-19 15:20:15 -08:00
jon-wei
747343e621
Preserve dimension order across indexes during ingestion
2016-01-19 13:34:11 -08:00
Jonathan Wei
df2906a91c
Merge pull request #2290 from gianm/index-merger-v9-stuff
...
Respect buildV9Directly in PlumberSchools, so it works on standalone realtime.
2016-01-19 13:04:00 -08:00
Fangjin Yang
0c31f007fc
Merge pull request #1728 from himanshug/aggregators_in_segment_metadata
...
Store AggregatorFactory[] in segment metadata
2016-01-19 12:55:49 -08:00
Himanshu
fe841fd961
Merge pull request #2118 from guobingkun/fix_segment_loading
...
Fix loading segment for historical
2016-01-19 14:25:48 -06:00
Himanshu Gupta
a99aef29a1
adding aggregators to segment metadata
2016-01-19 14:23:39 -06:00
Gian Merlino
1dcf22edb7
Respect buildV9Directly in PlumberSchools, so it works on standalone realtime nodes.
...
Also parameterize some tests to run with/without buildV9Directly:
- IndexGeneratorJobTest
- RealtimeIndexTaskTest
- RealtimePlumberSchoolTest
2016-01-19 12:15:06 -08:00
Fangjin Yang
1b359d6a47
Merge pull request #2285 from fjy/stringformat
...
fixed #1873 , add ability to express CONCAT as an extractionFn
2016-01-19 11:23:12 -08:00
Bingkun Guo
c4ad50f92c
Fix loading segment for historical
...
Historical will drop a segment that shouldn't be dropped in the following scenario:
Historical node tried to load segmentA, but failed with SegmentLoadingException,
then ZkCoordinator called removeSegment(segmentA, blah) to schedule a runnable that would drop segmentA by deleting its files. Now, before that runnable executed, another LOAD request was sent to this historical, this time historical actually succeeded on loading segmentA and announced it. But later on, the scheduled drop-of-segment runnable started executing and removed the segment files, while historical is still announcing segmentA.
2016-01-19 10:29:49 -06:00
Fangjin Yang
a2e327ed08
Merge pull request #2275 from himanshug/local_segment_killer_fix
...
kill unwanted parent directories when a segment is deleted from LocalDataSegmentKiller
2016-01-18 22:54:01 -08:00
Himanshu Gupta
52eb0f04a7
adding a new method getMergingFactory(..) to AggregatorFactory
2016-01-18 22:03:46 -06:00
Himanshu Gupta
77fc86c015
making AggregatorFactory abstract class
2016-01-18 22:03:46 -06:00
Himanshu Gupta
164b0aad7a
removing Map<String,Object> segmentMetadata from methods in Index[Maker/Merger] and using Metadata class
...
instead of a Map to store segment metadata
2016-01-18 22:03:46 -06:00
Himanshu Gupta
dcd3a24f59
adding log line for segment being killed in HdfsDataSegmentKiller
2016-01-18 21:51:04 -06:00
Himanshu Gupta
637d2605e7
kill unwanted parent directories when a segment is deleted from LocalDataSegmentKiller
2016-01-18 21:51:04 -06:00
zhxiaog
3459a202ce
fixed #1873 , add ability to express CONCAT as an extractionFn
2016-01-18 15:03:17 -08:00
Fangjin Yang
592638f6b3
Merge pull request #2209 from sirpkt/cascade_extraction_functions
...
support cascade execution of extraction filters in extraction dimension spec
2016-01-18 07:59:36 -08:00
Fangjin Yang
3f4cd22820
Merge pull request #2281 from gianm/zk-346
...
ZooKeeper 3.4.7 was unreleased, revert to 3.4.6.
2016-01-18 07:57:24 -08:00
Li Zhanhui
da5d7b9376
Reformat code according to Druid Java and Scala style
2016-01-18 16:37:03 +08:00
Gian Merlino
42ee473052
ZooKeeper 3.4.7 was unreleased, revert to 3.4.6.
2016-01-17 21:31:29 -08:00
Keuntae Park
238dd3be3c
support cascade execution of extraction filters in extraction dimension spec
2016-01-18 11:10:19 +09:00
Fangjin Yang
b3dcc04d9e
Merge pull request #2279 from gianm/multitenant-docs
...
Some more multitenancy docs
2016-01-17 17:57:41 -08:00
Gian Merlino
2d3f6e7705
Some more multitenancy docs
2016-01-17 17:47:49 -08:00
navis.ryu
f03f7fb625
Best effort to find locations for input splits
2016-01-18 08:31:05 +09:00
Harsh Jain
71f1cd5e34
Fix for issue 2021
2016-01-17 16:10:04 +05:30
Fangjin Yang
f6a1a4ae20
Merge pull request #2138 from KurtYoung/feature-build-v9
...
build v9 directly
2016-01-16 13:35:46 -06:00