Apache/druid
1
0
Fork 0
mirror of https://github.com/apache/druid.git synced 2025-02-09 03:24:55 +00:00
Code Issues Packages Projects Releases Wiki Activity
6,534 Commits 116 Branches 507 Tags
Commit Graph

6534 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Charles Allen
2e1d6aaf3d Use thread priorities. (aka set nice values for background-like tasks) 
* Defaults the thread priority to java.util.Thread.NORM_PRIORITY in io.druid.indexing.common.task.AbstractTask
 * Each exec service has its own Task Factory which is assigned a priority for spawned task. Therefore each priority class has a unique exec service
 * Added priority to tasks as taskPriority in the task context. <0 means low, 0 means take default, >0 means high. It is up to any particular implementation to determine how to handle these numbers
 * Add options to ForkingTaskRunner
    * Add "-XX:+UseThreadPriorities" default option
    * Add "-XX:ThreadPriorityPolicy=42" default option
 * AbstractTask - Removed unneded @JsonIgnore on priority
 * Added priority to RealtimePlumber executors. All sub-executors (non query runners) get Thread.MIN_PRIORITY
 * Add persistThreadPriority and mergeThreadPriority to realtime tuning config
 2016-01-20 14:00:31 -08:00
Fangjin Yang
695f107870 Merge pull request #2302 from metamx/lowerCaseGranPathTest 
Make GranularityPathSpecTest check with lower-case enums
 2016-01-20 09:18:06 -08:00
Fangjin Yang
6f4dfbc8d1 Merge pull request #2296 from loganlinn/loganlinn-dataGranularity-type-doc 
Update batch-ingestion.md
 2016-01-20 08:58:49 -08:00
Nishant
ac6c90e657 Merge pull request #1953 from metamx/taskRunnerResourceManagement 
Move resource managemnt to be the responsibility of the TaskRunner
 2016-01-20 22:27:47 +05:30
Charles Allen
3c5ca3a5f2 Make GranularityPathSpecTest check with lower-case enums 2016-01-20 08:35:13 -08:00
Jonathan Wei
fc09929503 Merge pull request #2293 from jon-wei/index_merger_test_fix 
Consolidate IndexMergerTest and IndexMergerV9Test
 2016-01-19 17:42:11 -08:00
Logan Linn
c3bdaefe1f Update batch-ingestion.md 
Fix documented type of the `dataGranularity` config
 2016-01-19 17:20:47 -08:00
jon-wei
bc1e9b27c8 Consolidate IndexMergerTest and IndexMergerV9Test 2016-01-19 16:28:35 -08:00
Fangjin Yang
cb8f714f82 Merge pull request #2006 from jon-wei/inherit_dim_order 
Preserve dimension order across indexes during ingestion
 2016-01-19 15:20:15 -08:00
jon-wei
747343e621 Preserve dimension order across indexes during ingestion 2016-01-19 13:34:11 -08:00
Jonathan Wei
df2906a91c Merge pull request #2290 from gianm/index-merger-v9-stuff 
Respect buildV9Directly in PlumberSchools, so it works on standalone realtime.
 2016-01-19 13:04:00 -08:00
Fangjin Yang
0c31f007fc Merge pull request #1728 from himanshug/aggregators_in_segment_metadata 
Store AggregatorFactory[] in segment metadata
 2016-01-19 12:55:49 -08:00
Himanshu
fe841fd961 Merge pull request #2118 from guobingkun/fix_segment_loading 
Fix loading segment for historical
 2016-01-19 14:25:48 -06:00
Himanshu Gupta
a99aef29a1 adding aggregators to segment metadata 2016-01-19 14:23:39 -06:00
Gian Merlino
1dcf22edb7 Respect buildV9Directly in PlumberSchools, so it works on standalone realtime nodes. 
Also parameterize some tests to run with/without buildV9Directly:

- IndexGeneratorJobTest
- RealtimeIndexTaskTest
- RealtimePlumberSchoolTest
 2016-01-19 12:15:06 -08:00
Fangjin Yang
1b359d6a47 Merge pull request #2285 from fjy/stringformat 
fixed #1873, add ability to express CONCAT as an extractionFn
 2016-01-19 11:23:12 -08:00
Bingkun Guo
c4ad50f92c Fix loading segment for historical 
Historical will drop a segment that shouldn't be dropped in the following scenario:
Historical node tried to load segmentA, but failed with SegmentLoadingException,
then ZkCoordinator called removeSegment(segmentA, blah) to schedule a runnable that would drop segmentA by deleting its files. Now, before that runnable executed, another LOAD request was sent to this historical, this time historical actually succeeded on loading segmentA and announced it. But later on, the scheduled drop-of-segment runnable started executing and removed the segment files, while historical is still announcing segmentA.
 2016-01-19 10:29:49 -06:00
Fangjin Yang
a2e327ed08 Merge pull request #2275 from himanshug/local_segment_killer_fix 
kill unwanted parent directories when a segment is deleted from LocalDataSegmentKiller
 2016-01-18 22:54:01 -08:00
Himanshu Gupta
52eb0f04a7 adding a new method getMergingFactory(..) to AggregatorFactory 2016-01-18 22:03:46 -06:00
Himanshu Gupta
77fc86c015 making AggregatorFactory abstract class 2016-01-18 22:03:46 -06:00
Himanshu Gupta
164b0aad7a removing Map<String,Object> segmentMetadata from methods in Index[Maker/Merger] and using Metadata class 
instead of a Map to store segment metadata
 2016-01-18 22:03:46 -06:00
Himanshu Gupta
dcd3a24f59 adding log line for segment being killed in HdfsDataSegmentKiller 2016-01-18 21:51:04 -06:00
Himanshu Gupta
637d2605e7 kill unwanted parent directories when a segment is deleted from LocalDataSegmentKiller 2016-01-18 21:51:04 -06:00
zhxiaog
3459a202ce fixed #1873, add ability to express CONCAT as an extractionFn 2016-01-18 15:03:17 -08:00
Fangjin Yang
592638f6b3 Merge pull request #2209 from sirpkt/cascade_extraction_functions 
support cascade execution of extraction filters in extraction dimension spec
 2016-01-18 07:59:36 -08:00
Fangjin Yang
3f4cd22820 Merge pull request #2281 from gianm/zk-346 
ZooKeeper 3.4.7 was unreleased, revert to 3.4.6.
 2016-01-18 07:57:24 -08:00
Gian Merlino
42ee473052 ZooKeeper 3.4.7 was unreleased, revert to 3.4.6. 2016-01-17 21:31:29 -08:00
Keuntae Park
238dd3be3c support cascade execution of extraction filters in extraction dimension spec 2016-01-18 11:10:19 +09:00
Fangjin Yang
b3dcc04d9e Merge pull request #2279 from gianm/multitenant-docs 
Some more multitenancy docs
 2016-01-17 17:57:41 -08:00
Gian Merlino
2d3f6e7705 Some more multitenancy docs 2016-01-17 17:47:49 -08:00
Fangjin Yang
f6a1a4ae20 Merge pull request #2138 from KurtYoung/feature-build-v9 
build v9 directly
 2016-01-16 13:35:46 -06:00
Kurt Young
82ff98c2bf add config for build v9 directly and update docs 2016-01-16 11:26:34 +08:00
Kurt Young
1f2168fae5 add IndexMergerV9 
add unit tests for IndexMergerV9 and fix some bugs

add more unit tests and fix bugs

handle null values and add more tests

minor changes & use LoggingProgressIndicator in IndexGeneratorReducer

make some static class public from IndexMerger

minor changes and add some comments

changes for comments
 2016-01-16 11:25:28 +08:00
Kurt Young
bb50d2a2b2 add some streaming writers 2016-01-16 11:25:26 +08:00
Fangjin Yang
2e54553a8f Merge pull request #1990 from himanshug/schedule_kill_task 
support periodic hard delete of segments
 2016-01-15 15:22:33 -06:00
David Lim
7c65880e55 Merge pull request #2270 from rasahner/warnOfChatHandlerNoop 
if chathandler is noop, log using warn instead of info
 2016-01-15 13:29:27 -07:00
David Lim
34cd8f8c72 Merge pull request #2258 from fjy/acl-zk 
acl for zookeeper is added
 2016-01-15 10:27:08 -07:00
Navis Ryu
e4d15f3729 Merge pull request #2133 from navis/log4j-shudown-callback 
callbacks registered to Log4jShutdown is not executed when stop is called
 2016-01-15 21:28:46 +09:00
Fangjin Yang
a54c726726 Merge pull request #2266 from anubhgup/fix-announce 
Fix loss in segment announcements when segments do not fit in zNode
 2016-01-14 17:10:43 -08:00
Fangjin Yang
e0932ba1c2 Merge pull request #2267 from himanshug/fix_topn_multi_val_filter 
Remap id's returned in XXXFilteredDimensionSpec.getRow() as per reduced cardinality
 2016-01-14 17:06:54 -08:00
Fangjin Yang
7704699b40 Merge pull request #2265 from navis/strlen-dimension-ignored 
Strlen sort spec ignores dimension
 2016-01-14 17:06:33 -08:00
Xavier Léauté
dc1a62c3d9 Merge pull request #2248 from metamx/druidNodeHashEquals 
Add hashCode and equals to DruidNode
 2016-01-14 16:04:58 -08:00
Anubhav Gupta
6d09ab839f Fix for loss in segment announcements when segments do not fit in the znodes during compress mode. 
Added unit test (from Navis).
 2016-01-14 14:44:23 -08:00
Robin
7361cd173f if chathandler is noop, log using warn instead of info 2016-01-14 10:12:42 -06:00
Himanshu Gupta
ae6a111444 fix XXXFilteredDimensionSpec to remap the dictionary encodings as per new cardinality 2016-01-13 22:25:02 -06:00
binlijin
a3140b2548 fix topN filtering on multi-valued dimension bug 2016-01-13 22:25:02 -06:00
navis.ryu
ea9fabdf2f Strlen sort spec ignores dimension 2016-01-14 11:05:44 +09:00
Nikita Geer
1908d63162 acl for zookeeper is added 2016-01-13 14:56:05 -08:00
Fangjin Yang
4c014c1574 Merge pull request #2228 from metamx/incremental-index-mem2 
Improve heap usage for IncrementalIndex
 2016-01-13 14:48:03 -08:00
Charles Allen
976d4c965b Move resource managemnt to be the responsibility of the TaskRunner 2016-01-13 10:38:22 -08:00