Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,974 Commits 111 Branches 503 Tags 721 MiB
Commit Graph

2656 Commits

Author SHA1 Message Date
Parag Jain b82b487f20 remove extra kill parameter 2016-02-24 17:16:18 -06:00
jon-wei c17ce02467 Allow IncrementalIndex to store Long/Float dimensions 2016-02-24 13:51:57 -08:00
Himanshu Gupta a3b37e9225 In persistAndMerge, increase the scope of try-catch block so that any exception while persisting hydrants is caught and consequently that sink is abandoned or the task will forever wait for handoff to happen. 2016-02-23 22:22:33 -06:00
Nishant 6c9e1a28ad Merge pull request #2519 from gianm/unparseable-handling 
Better handling of ParseExceptions.
 2016-02-24 04:46:29 +05:30
Fangjin Yang 93540c0631 Merge pull request #2503 from gianm/jetty-qos 
Add druid.indexer.server.maxChatRequests for QoS; deprecate separate ports.
 2016-02-23 10:35:53 -08:00
Gian Merlino 3534483433 Better handling of ParseExceptions. 
Two changes:
- Allow IncrementalIndex to suppress ParseExceptions on "aggregate".
- Add "reportParseExceptions" option to realtime tuning configs. By default this is "false".

Behavior of the counters should now be:

- processed: Number of rows indexed, including rows where some fields could be parsed and some could not.
- thrownAway: Number of rows thrown away due to rejection policy.
- unparseable: Number of rows thrown away due to being completely unparseable (no fields salvageable at all).

If "reportParseExceptions" is true then "unparseable" will always be zero (because a parse error would
cause an exception to be thrown). In addition, "processed" will only include fully parseable rows
(because even partial parse failures will cause exceptions to be thrown).

Fixes #2510.
 2016-02-23 10:11:43 -08:00
Fangjin Yang 0c984f9e32 Merge pull request #2109 from himanshug/segments_in_delta_ingestion 
idempotent batch delta ingestion
 2016-02-22 14:00:45 -08:00
Fangjin Yang 3bdd757024 Merge pull request #1773 from b-slim/log_details 
Adding downstream source when throwing QueryInterruptedException
 2016-02-22 10:16:07 -08:00
Himanshu Gupta 21b0b8a07d new coordinator endpoint to get list of used segment given a dataSource and list of intervals 2016-02-21 23:17:58 -06:00
Slim Bouguerra 77925cc061 adding downstream source of QueryInterruptedException 2016-02-20 13:05:14 -06:00
Gian Merlino 23c993c9e7 Add druid.indexer.server.maxChatRequests for QoS; deprecate separate ports. 
- Add druid.indexer.server.maxChatRequests, which sets up a QoSFilter on the main Jetty server.
- Deprecate druid.indexer.runner.separateIngestionEndpoint
- Deprecate druid.indexer.server.chathandler.*
 2016-02-19 13:36:09 -08:00
Gian Merlino 243ac5399b Harmonize realtime indexing loop across the task and standalone nodes. 
- Both now catch ParseExceptions on plumber.add (see https://groups.google.com/d/topic/druid-user/wmiRDvx2RvM/discussion)
- Standalone now treats IndexSizeExceededException as fatal (previously only the task did)
 2016-02-19 07:34:15 -08:00
Gian Merlino e0c049c0b0 Make startup properties logging optional. 
Off by default, but enabled in the example config files. See also #2452.
 2016-02-12 14:12:16 -08:00
Fangjin Yang 1430bc2c88 Merge pull request #2276 from harshjain2/feature-2021 
Fix for issue 2021.
 2016-02-10 17:04:45 -08:00
Gian Merlino fa92b77f5a Harmonize znode writing code in RTR and Worker. 
- Throw most exceptions rather than suppressing them, which should help
  detect problems. Continue suppressing exceptions that make sense to
  suppress.
- Handle payload length checks consistently, and improve error message.
- Remove unused WorkerCuratorCoordinator.announceTaskAnnouncement method.
- Max znode length should be int, not long.
- Add tests.
 2016-02-10 14:52:00 -08:00
Harsh Jain a3eb863c8e Fix for issue 2021 2016-02-10 22:19:12 +05:30
Himanshu Gupta d1cb17d3f7 at broker - only add segments from specific tiers to the timeline 2016-02-09 22:33:22 -06:00
Himanshu Gupta b40c342cd1 make Global stupid pool cache size configurable 2016-02-05 14:18:06 -06:00
Parag Jain 9002548eeb increase test time out and general clean up 2016-02-03 13:26:37 -06:00
Charles Allen 5111fd52f2 Add check for log4j-core in Log4jShutterDownerModule 2016-02-02 15:56:48 -08:00
Himanshu dc89cdd0f9 Merge pull request #2336 from himanshug/fix_2331 
limit size of X-Druid-Response-Context header to 7K
 2016-02-02 12:06:59 -06:00
navis.ryu c03918f89a AsyncQueryForwardingServletTest#testDeleteBroadcast sometimes fails by port conflict 2016-01-29 19:28:58 +09:00
Himanshu Gupta f6b4dbd697 bug fix and unit tests for DruidCoordinatorSegmentKiller 2016-01-28 14:10:17 -06:00
Himanshu Gupta ab3edfa8fc moving DruidCoordinatorSegmentKiller class out of DruidCoordinator 2016-01-28 14:03:56 -06:00
Nishant 3880f54b87 Merge pull request #2332 from himanshug/configurable_partial 
make populateUncoveredIntervals a configuration in query context
 2016-01-28 10:34:35 +05:30
Himanshu Gupta a7bde8f4da limit size of X-Druid-Response-Context header to 7K due to https://github.com/druid-io/druid/issues/2331 2016-01-27 15:18:08 -06:00
Xavier Léauté 5a3642bb93 Merge pull request #2247 from metamx/pedanticBuild 
Enable strict building in travis
 2016-01-27 10:27:03 -08:00
Charles Allen 508734c8b0 Long constant reformatting in tests `l` --> `L` 2016-01-27 08:59:19 -08:00
Nishant fd6bf3fe22 Use interval comparator instead of bucketMonthComparator 
fix when two segments have same interval

review comments
 2016-01-27 17:35:43 +05:30
Himanshu Gupta 3719b6e3c8 make populateUncoveredIntervals a configuration in query context 2016-01-26 15:13:45 -06:00
Harsh Jain 41730b96d4 Fix for issue 2021 2016-01-25 02:48:22 +05:30
Himanshu 7a6109f0ca Merge pull request #2321 from gianm/pending-index 
Replace two-column index on pendingSegments table with one-column index.
 2016-01-22 13:38:15 -06:00
Gian Merlino 0bd9bff075 Replace two-column index on pendingSegments table with one-column index. 
Fixes #2319.
 2016-01-22 10:50:21 -08:00
Himank Chaudhary 1a5d4e714c Adding custom mapper for json processing exception to return bad request instead of 500 2016-01-22 09:48:52 -08:00
Fangjin Yang 04d3054353 Merge pull request #2303 from CHOIJAEHONG1/localfirehouse-basedir-npe 
Throw an IAE when baseDir is null in LocalFireHose
 2016-01-21 07:58:52 -08:00
Nishant dcb7830330 Merge pull request #984 from drcrallen/thread-priority-rebase 
Use thread priorities. (aka set `nice` values for background-like tasks)
 2016-01-21 15:02:34 +05:30
Charles Allen 2e1d6aaf3d Use thread priorities. (aka set `nice` values for background-like tasks) 
* Defaults the thread priority to java.util.Thread.NORM_PRIORITY in io.druid.indexing.common.task.AbstractTask
 * Each exec service has its own Task Factory which is assigned a priority for spawned task. Therefore each priority class has a unique exec service
 * Added priority to tasks as taskPriority in the task context. <0 means low, 0 means take default, >0 means high. It is up to any particular implementation to determine how to handle these numbers
 * Add options to ForkingTaskRunner
    * Add "-XX:+UseThreadPriorities" default option
    * Add "-XX:ThreadPriorityPolicy=42" default option
 * AbstractTask - Removed unneded @JsonIgnore on priority
 * Added priority to RealtimePlumber executors. All sub-executors (non query runners) get Thread.MIN_PRIORITY
 * Add persistThreadPriority and mergeThreadPriority to realtime tuning config
 2016-01-20 14:00:31 -08:00
Nishant 61aca6f9cc remove wrong checks 
sink never have null hydrants and hydrants never have null adapters
 2016-01-20 23:43:53 +05:30
Jaehong Choi 7132428bba throw IAE when baseDir is null in LocalFireHose 2016-01-21 01:27:32 +09:00
Nishant 59ea186af7 fix reference counting for segments 2016-01-20 17:24:21 +05:30
jon-wei 747343e621 Preserve dimension order across indexes during ingestion 2016-01-19 13:34:11 -08:00
Jonathan Wei df2906a91c Merge pull request #2290 from gianm/index-merger-v9-stuff 
Respect buildV9Directly in PlumberSchools, so it works on standalone realtime.
 2016-01-19 13:04:00 -08:00
Fangjin Yang 0c31f007fc Merge pull request #1728 from himanshug/aggregators_in_segment_metadata 
Store AggregatorFactory[] in segment metadata
 2016-01-19 12:55:49 -08:00
Himanshu fe841fd961 Merge pull request #2118 from guobingkun/fix_segment_loading 
Fix loading segment for historical
 2016-01-19 14:25:48 -06:00
Himanshu Gupta a99aef29a1 adding aggregators to segment metadata 2016-01-19 14:23:39 -06:00
Gian Merlino 1dcf22edb7 Respect buildV9Directly in PlumberSchools, so it works on standalone realtime nodes. 
Also parameterize some tests to run with/without buildV9Directly:

- IndexGeneratorJobTest
- RealtimeIndexTaskTest
- RealtimePlumberSchoolTest
 2016-01-19 12:15:06 -08:00
Bingkun Guo c4ad50f92c Fix loading segment for historical 
Historical will drop a segment that shouldn't be dropped in the following scenario:
Historical node tried to load segmentA, but failed with SegmentLoadingException,
then ZkCoordinator called removeSegment(segmentA, blah) to schedule a runnable that would drop segmentA by deleting its files. Now, before that runnable executed, another LOAD request was sent to this historical, this time historical actually succeeded on loading segmentA and announced it. But later on, the scheduled drop-of-segment runnable started executing and removed the segment files, while historical is still announcing segmentA.
 2016-01-19 10:29:49 -06:00
Himanshu Gupta 164b0aad7a removing Map<String,Object> segmentMetadata from methods in Index[Maker/Merger] and using Metadata class 
instead of a Map to store segment metadata
 2016-01-18 22:03:46 -06:00
Himanshu Gupta 637d2605e7 kill unwanted parent directories when a segment is deleted from LocalDataSegmentKiller 2016-01-18 21:51:04 -06:00
Harsh Jain 71f1cd5e34 Fix for issue 2021 2016-01-17 16:10:04 +05:30
Fangjin Yang f6a1a4ae20 Merge pull request #2138 from KurtYoung/feature-build-v9 
build v9 directly
 2016-01-16 13:35:46 -06:00
Kurt Young 82ff98c2bf add config for build v9 directly and update docs 2016-01-16 11:26:34 +08:00
Fangjin Yang 2e54553a8f Merge pull request #1990 from himanshug/schedule_kill_task 
support periodic hard delete of segments
 2016-01-15 15:22:33 -06:00
David Lim 7c65880e55 Merge pull request #2270 from rasahner/warnOfChatHandlerNoop 
if chathandler is noop, log using warn instead of info
 2016-01-15 13:29:27 -07:00
Harsh Jain 6ec6835b5d Fix for issue 2021. 2016-01-16 00:58:33 +05:30
David Lim 34cd8f8c72 Merge pull request #2258 from fjy/acl-zk 
acl for zookeeper is added
 2016-01-15 10:27:08 -07:00
Fangjin Yang a54c726726 Merge pull request #2266 from anubhgup/fix-announce 
Fix loss in segment announcements when segments do not fit in zNode
 2016-01-14 17:10:43 -08:00
Xavier Léauté dc1a62c3d9 Merge pull request #2248 from metamx/druidNodeHashEquals 
Add hashCode and equals to DruidNode
 2016-01-14 16:04:58 -08:00
Anubhav Gupta 6d09ab839f Fix for loss in segment announcements when segments do not fit in the znodes during compress mode. 
Added unit test (from Navis).
 2016-01-14 14:44:23 -08:00
Robin 7361cd173f if chathandler is noop, log using warn instead of info 2016-01-14 10:12:42 -06:00
Nikita Geer 1908d63162 acl for zookeeper is added 2016-01-13 14:56:05 -08:00
navis.ryu 18479bb757 time-descending result of timeseries queries 2016-01-13 12:23:01 +09:00
Himanshu Gupta eb2d251ac8 support periodic hard delete of segments 2016-01-12 16:55:05 -06:00
Himanshu 01a0715ee2 Merge pull request #2161 from metamx/query-metrics-timeout 
Fix Query metrics for query timeout
 2016-01-12 09:50:40 -06:00
Himanshu Gupta ef3cddabe9 adding UTs for DruidCoordinatorConfig 2016-01-11 22:03:25 -06:00
Charles Allen b0e04a9162 Add hashCode and equals to DruidNode 2016-01-11 15:23:45 -08:00
Charles Allen ea623e43d2 Merge pull request #2240 from metamx/fix-load-rule 
Fix loadRule when one of the tiers had no available servers
 2016-01-11 10:05:31 -08:00
Nishant 32bc2f776e Fix loadRule when one of the tiers had no servers 
When one of the tiers have no servers, LoadRule should ignore that tier
and continue to load/drop segments in other available tiers.

the bug also causes whacky behavior with LoadRule with non existent
tier where the segment balancer keeps on moving segments to other nodes
in existing tiers but the extra segment copies are never dropped
eventually leading to all the tiers getting full .
 2016-01-11 15:53:14 +05:30
Himanshu d255f4baac Merge pull request #2234 from pjain1/emit_realtime_metrics 
emit handoff count metrics
 2016-01-08 14:24:16 -06:00
Parag Jain 9dba0f67e7 emit handoff count metrics 2016-01-08 12:36:13 -06:00
Fangjin Yang 15fc070232 Merge pull request #2213 from himanshug/fix_curtator_test_base 
[wip] trying/finding fix for announcer test failures
 2016-01-07 18:23:49 -08:00
Nishant 1bfb4e3988 Emit query/time for failed and timeout queries 
emit query/time metric also

add success flag

fix success flag for router metrics

review comments

formatting.
 2016-01-07 19:41:54 +05:30
Himanshu Gupta 7ab810f3eb do not ignore exceptions from curator cleanup in CuratorTestBase 2016-01-06 10:42:53 -06:00
Nishant 14989f272d Add metrics for ingest/bytes/received for EventReceiverFirehose 
review comments

review comments
 2016-01-05 20:06:09 +05:30
fjy 57d91d754d Comment out buggy unit tests, fix #2185 2016-01-03 09:50:16 -08:00
fjy 8424b2b456 Fix announcer test bad check 2016-01-01 19:55:22 -08:00
Himanshu Gupta fa5c3bb014 adding decorate(DimensionSelector) to DimensionSpec to enable support for arbitrary filtering/transformations to returned dimension values 2015-12-30 15:06:24 -06:00
Bingkun Guo 3c107c5757 Merge pull request #2150 from himanshug/emit_query_bytes 
emit query/bytes metric
 2015-12-30 13:44:19 -06:00
Fangjin Yang 7ffa706655 Merge pull request #2152 from metamx/add-taskId 
Add taskId to realtimeMetrics
 2015-12-29 10:33:40 -08:00
Fangjin Yang b1261035a7 Merge pull request #1861 from guobingkun/insert_segment_tool 
insert-segment tool
 2015-12-29 10:06:07 -08:00
fjy 38b0f1fbc2 fix transient failures in unit tests 2015-12-28 20:03:30 -08:00
Fangjin Yang d94821998f Merge pull request #2153 from navis/fix-testfail-DruidCoordinatorTest 
Another try to fix DruidCoordinatorTest#testCoordinatorRun
 2015-12-27 17:03:46 -08:00
navis.ryu 10c19fe9f8 Another try to fix DruidCoordinatorTest#testCoordinatorRun 2015-12-24 03:02:37 +09:00
navis.ryu 2c3c4a3f8f Another try to fix xxServerViewTests 2015-12-24 02:13:40 +09:00
Fangjin Yang 508629916d Merge pull request #2137 from guobingkun/fix_serverview_transient_error2 
[Don't merge]Another effort on fixing the transient error for XXXServerViewTest
 2015-12-23 07:33:54 -08:00
Nishant 978a3fd8ae Add taskId to realtimeMetrics 
Add task Id to Realtime Metrics
 2015-12-23 18:05:25 +05:30
Himanshu Gupta 1a8546a682 emit query/bytes metric 2015-12-23 00:29:44 -06:00
Bingkun Guo 89e2a9a6d2 build at Tue Dec 22 21:30:00 CST 2015 2015-12-22 21:30:00 -06:00
Bingkun Guo a110eafff7 build at Tue Dec 22 17:30:00 CST 2015 2015-12-22 17:30:00 -06:00
Bingkun Guo 4b93528334 build at Tue Dec 22 17:00:01 CST 2015 2015-12-22 17:00:01 -06:00
Bingkun Guo b8d9860dbc build at Tue Dec 22 16:00:00 CST 2015 2015-12-22 16:00:00 -06:00
Bingkun Guo ffdb9407e5 build at Tue Dec 22 15:30:00 CST 2015 2015-12-22 15:30:00 -06:00
Bingkun Guo 6f54c1b60d build at Tue Dec 22 15:00:00 CST 2015 2015-12-22 15:00:01 -06:00
Bingkun Guo 4e61261c68 build at Tue Dec 22 14:30:00 CST 2015 2015-12-22 14:30:00 -06:00
Bingkun Guo 4961b8c0bd build at Tue Dec 22 14:00:00 CST 2015 2015-12-22 14:00:00 -06:00
Bingkun Guo 0a635a11d6 build at Tue Dec 22 13:40:00 CST 2015 2015-12-22 13:40:00 -06:00
Charles Allen 05c9e1b598 Reorder Before/After in JDBCExtractionNamespaceTest 
* Fixes https://github.com/druid-io/druid/issues/2120
 2015-12-22 11:39:46 -08:00
Bingkun Guo 849ab2d958 build at 2015-12-22 13:08:29 -06:00
Bingkun Guo 455980d659 address another node exists failure 2015-12-22 13:06:20 -06:00
Bingkun Guo 2ffeda5d25 build number 13 2015-12-22 10:19:39 -06:00