Apache
/
druid
mirror of https://github.com/apache/druid.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity
6,853 Commits 111 Branches 503 Tags 728 MiB
Commit Graph

2617 Commits

Author SHA1 Message Date
Nishant 59ea186af7 fix reference counting for segments 2016-01-20 17:24:21 +05:30
jon-wei 747343e621 Preserve dimension order across indexes during ingestion 2016-01-19 13:34:11 -08:00
Jonathan Wei df2906a91c Merge pull request #2290 from gianm/index-merger-v9-stuff 
Respect buildV9Directly in PlumberSchools, so it works on standalone realtime.
 2016-01-19 13:04:00 -08:00
Fangjin Yang 0c31f007fc Merge pull request #1728 from himanshug/aggregators_in_segment_metadata 
Store AggregatorFactory[] in segment metadata
 2016-01-19 12:55:49 -08:00
Himanshu fe841fd961 Merge pull request #2118 from guobingkun/fix_segment_loading 
Fix loading segment for historical
 2016-01-19 14:25:48 -06:00
Himanshu Gupta a99aef29a1 adding aggregators to segment metadata 2016-01-19 14:23:39 -06:00
Gian Merlino 1dcf22edb7 Respect buildV9Directly in PlumberSchools, so it works on standalone realtime nodes. 
Also parameterize some tests to run with/without buildV9Directly:

- IndexGeneratorJobTest
- RealtimeIndexTaskTest
- RealtimePlumberSchoolTest
 2016-01-19 12:15:06 -08:00
Bingkun Guo c4ad50f92c Fix loading segment for historical 
Historical will drop a segment that shouldn't be dropped in the following scenario:
Historical node tried to load segmentA, but failed with SegmentLoadingException,
then ZkCoordinator called removeSegment(segmentA, blah) to schedule a runnable that would drop segmentA by deleting its files. Now, before that runnable executed, another LOAD request was sent to this historical, this time historical actually succeeded on loading segmentA and announced it. But later on, the scheduled drop-of-segment runnable started executing and removed the segment files, while historical is still announcing segmentA.
 2016-01-19 10:29:49 -06:00
Himanshu Gupta 164b0aad7a removing Map<String,Object> segmentMetadata from methods in Index[Maker/Merger] and using Metadata class 
instead of a Map to store segment metadata
 2016-01-18 22:03:46 -06:00
Himanshu Gupta 637d2605e7 kill unwanted parent directories when a segment is deleted from LocalDataSegmentKiller 2016-01-18 21:51:04 -06:00
Harsh Jain 71f1cd5e34 Fix for issue 2021 2016-01-17 16:10:04 +05:30
Fangjin Yang f6a1a4ae20 Merge pull request #2138 from KurtYoung/feature-build-v9 
build v9 directly
 2016-01-16 13:35:46 -06:00
Kurt Young 82ff98c2bf add config for build v9 directly and update docs 2016-01-16 11:26:34 +08:00
Fangjin Yang 2e54553a8f Merge pull request #1990 from himanshug/schedule_kill_task 
support periodic hard delete of segments
 2016-01-15 15:22:33 -06:00
David Lim 7c65880e55 Merge pull request #2270 from rasahner/warnOfChatHandlerNoop 
if chathandler is noop, log using warn instead of info
 2016-01-15 13:29:27 -07:00
Harsh Jain 6ec6835b5d Fix for issue 2021. 2016-01-16 00:58:33 +05:30
David Lim 34cd8f8c72 Merge pull request #2258 from fjy/acl-zk 
acl for zookeeper is added
 2016-01-15 10:27:08 -07:00
Fangjin Yang a54c726726 Merge pull request #2266 from anubhgup/fix-announce 
Fix loss in segment announcements when segments do not fit in zNode
 2016-01-14 17:10:43 -08:00
Xavier Léauté dc1a62c3d9 Merge pull request #2248 from metamx/druidNodeHashEquals 
Add hashCode and equals to DruidNode
 2016-01-14 16:04:58 -08:00
Anubhav Gupta 6d09ab839f Fix for loss in segment announcements when segments do not fit in the znodes during compress mode. 
Added unit test (from Navis).
 2016-01-14 14:44:23 -08:00
Robin 7361cd173f if chathandler is noop, log using warn instead of info 2016-01-14 10:12:42 -06:00
Nikita Geer 1908d63162 acl for zookeeper is added 2016-01-13 14:56:05 -08:00
navis.ryu 18479bb757 time-descending result of timeseries queries 2016-01-13 12:23:01 +09:00
Himanshu Gupta eb2d251ac8 support periodic hard delete of segments 2016-01-12 16:55:05 -06:00
Himanshu 01a0715ee2 Merge pull request #2161 from metamx/query-metrics-timeout 
Fix Query metrics for query timeout
 2016-01-12 09:50:40 -06:00
Himanshu Gupta ef3cddabe9 adding UTs for DruidCoordinatorConfig 2016-01-11 22:03:25 -06:00
Charles Allen b0e04a9162 Add hashCode and equals to DruidNode 2016-01-11 15:23:45 -08:00
Charles Allen ea623e43d2 Merge pull request #2240 from metamx/fix-load-rule 
Fix loadRule when one of the tiers had no available servers
 2016-01-11 10:05:31 -08:00
Nishant 32bc2f776e Fix loadRule when one of the tiers had no servers 
When one of the tiers have no servers, LoadRule should ignore that tier
and continue to load/drop segments in other available tiers.

the bug also causes whacky behavior with LoadRule with non existent
tier where the segment balancer keeps on moving segments to other nodes
in existing tiers but the extra segment copies are never dropped
eventually leading to all the tiers getting full .
 2016-01-11 15:53:14 +05:30
Himanshu d255f4baac Merge pull request #2234 from pjain1/emit_realtime_metrics 
emit handoff count metrics
 2016-01-08 14:24:16 -06:00
Parag Jain 9dba0f67e7 emit handoff count metrics 2016-01-08 12:36:13 -06:00
Fangjin Yang 15fc070232 Merge pull request #2213 from himanshug/fix_curtator_test_base 
[wip] trying/finding fix for announcer test failures
 2016-01-07 18:23:49 -08:00
Nishant 1bfb4e3988 Emit query/time for failed and timeout queries 
emit query/time metric also

add success flag

fix success flag for router metrics

review comments

formatting.
 2016-01-07 19:41:54 +05:30
Himanshu Gupta 7ab810f3eb do not ignore exceptions from curator cleanup in CuratorTestBase 2016-01-06 10:42:53 -06:00
Nishant 14989f272d Add metrics for ingest/bytes/received for EventReceiverFirehose 
review comments

review comments
 2016-01-05 20:06:09 +05:30
fjy 57d91d754d Comment out buggy unit tests, fix #2185 2016-01-03 09:50:16 -08:00
fjy 8424b2b456 Fix announcer test bad check 2016-01-01 19:55:22 -08:00
Himanshu Gupta fa5c3bb014 adding decorate(DimensionSelector) to DimensionSpec to enable support for arbitrary filtering/transformations to returned dimension values 2015-12-30 15:06:24 -06:00
Bingkun Guo 3c107c5757 Merge pull request #2150 from himanshug/emit_query_bytes 
emit query/bytes metric
 2015-12-30 13:44:19 -06:00
Fangjin Yang 7ffa706655 Merge pull request #2152 from metamx/add-taskId 
Add taskId to realtimeMetrics
 2015-12-29 10:33:40 -08:00
Fangjin Yang b1261035a7 Merge pull request #1861 from guobingkun/insert_segment_tool 
insert-segment tool
 2015-12-29 10:06:07 -08:00
fjy 38b0f1fbc2 fix transient failures in unit tests 2015-12-28 20:03:30 -08:00
Fangjin Yang d94821998f Merge pull request #2153 from navis/fix-testfail-DruidCoordinatorTest 
Another try to fix DruidCoordinatorTest#testCoordinatorRun
 2015-12-27 17:03:46 -08:00
navis.ryu 10c19fe9f8 Another try to fix DruidCoordinatorTest#testCoordinatorRun 2015-12-24 03:02:37 +09:00
navis.ryu 2c3c4a3f8f Another try to fix xxServerViewTests 2015-12-24 02:13:40 +09:00
Fangjin Yang 508629916d Merge pull request #2137 from guobingkun/fix_serverview_transient_error2 
[Don't merge]Another effort on fixing the transient error for XXXServerViewTest
 2015-12-23 07:33:54 -08:00
Nishant 978a3fd8ae Add taskId to realtimeMetrics 
Add task Id to Realtime Metrics
 2015-12-23 18:05:25 +05:30
Himanshu Gupta 1a8546a682 emit query/bytes metric 2015-12-23 00:29:44 -06:00
Bingkun Guo 89e2a9a6d2 build at Tue Dec 22 21:30:00 CST 2015 2015-12-22 21:30:00 -06:00
Bingkun Guo a110eafff7 build at Tue Dec 22 17:30:00 CST 2015 2015-12-22 17:30:00 -06:00
Bingkun Guo 4b93528334 build at Tue Dec 22 17:00:01 CST 2015 2015-12-22 17:00:01 -06:00
Bingkun Guo b8d9860dbc build at Tue Dec 22 16:00:00 CST 2015 2015-12-22 16:00:00 -06:00
Bingkun Guo ffdb9407e5 build at Tue Dec 22 15:30:00 CST 2015 2015-12-22 15:30:00 -06:00
Bingkun Guo 6f54c1b60d build at Tue Dec 22 15:00:00 CST 2015 2015-12-22 15:00:01 -06:00
Bingkun Guo 4e61261c68 build at Tue Dec 22 14:30:00 CST 2015 2015-12-22 14:30:00 -06:00
Bingkun Guo 4961b8c0bd build at Tue Dec 22 14:00:00 CST 2015 2015-12-22 14:00:00 -06:00
Bingkun Guo 0a635a11d6 build at Tue Dec 22 13:40:00 CST 2015 2015-12-22 13:40:00 -06:00
Charles Allen 05c9e1b598 Reorder Before/After in JDBCExtractionNamespaceTest 
* Fixes https://github.com/druid-io/druid/issues/2120
 2015-12-22 11:39:46 -08:00
Bingkun Guo 849ab2d958 build at 2015-12-22 13:08:29 -06:00
Bingkun Guo 455980d659 address another node exists failure 2015-12-22 13:06:20 -06:00
Bingkun Guo 2ffeda5d25 build number 13 2015-12-22 10:19:39 -06:00
Charles Allen 7e64d5179f Merge pull request #2139 from himanshug/emit_query_node_bytes 
emit query/node/bytes metric
 2015-12-22 08:07:04 -08:00
Bingkun Guo bc573484f3 build number 12 2015-12-22 00:52:58 -06:00
Himanshu 7b5fd76058 Merge pull request #2119 from navis/fix-sporadic-testfail2 
Fix sporadic fail of DruidCoordinatorTest#testCoordinatorRun
 2015-12-21 23:28:32 -06:00
Himanshu Gupta b96f560255 emit query/node/bytes metric 2015-12-21 23:23:20 -06:00
Bingkun Guo 06c220ae7b build number 11 2015-12-21 19:40:47 -06:00
Bingkun Guo 93a91e5633 build number 10 2015-12-21 18:21:15 -06:00
Bingkun Guo 471dd7e164 build number 9 2015-12-21 17:35:18 -06:00
Bingkun Guo 627c311923 build #8 2015-12-21 17:06:37 -06:00
Bingkun Guo 293f2ad65d build #7 2015-12-21 16:09:19 -06:00
Bingkun Guo 9ed843ea21 build #6 2015-12-21 15:41:23 -06:00
Bingkun Guo f8342ca240 build #5 2015-12-21 15:10:37 -06:00
Bingkun Guo 52261c5539 build #4 2015-12-21 14:41:38 -06:00
Bingkun Guo 47a8f2d3f1 build #3 2015-12-21 13:56:43 -06:00
Bingkun Guo 5cc56cf9e7 build #2 2015-12-21 13:05:38 -06:00
Bingkun Guo f12cfa99fa another effort on fixing the transient error for Broker/CoordinatorServerViewTest happend on Travis build 2015-12-21 12:31:36 -06:00
Charles Allen c2a2d19d62 Merge pull request #2126 from guobingkun/remove_single_announcer 
Remove SingleDataSegmentAnnouncer in favor of BatchDataSegmentAnnouncer
 2015-12-21 08:55:07 -08:00
navis.ryu 7b24d1f355 Fix sporadic fail of DruidCoordinatorTest#testCoordinatorRun 2015-12-21 17:00:52 +09:00
Bingkun Guo 951a4e9b35 Remove SingleDataSegmentAnnouncer in favor of BatchDataSegmentAnnouncer 2015-12-21 00:05:53 -06:00
Bingkun Guo 89b477970f DataSegmentFinder tool 
`insert-segment-to-db` is a tool that can insert segments into Druid metadata storage. It is intended to be used
to update the segment table in metadata storage after people manually migrate segments from one place to another.
It can also be used to insert missing segment into Druid, or even recover metadata storage by telling it where the
segments are stored.

Note: This tool expects users to have Druid cluster running in a "safe" mode, where there are no active tasks to interfere
the segments being inserted. Users can optionally bring down the cluster to make 100% sure nothing is interfering.
 2015-12-21 00:02:04 -06:00
navis.ryu ad3312171b Fix sporadic fail of io.druid.client.CoordinatorServerViewTest 2015-12-21 11:39:32 +09:00
Fangjin Yang 7019d3c421 Merge pull request #2107 from jon-wei/fix_smq 
More efficient SegmentMetadataQuery
 2015-12-18 16:40:47 -08:00
Fangjin Yang e52c9d2b3c Merge pull request #2095 from guobingkun/fix_client_merge_task_and_serde 
Fix getType() and Json serialization in ClientMergeQuery + serde tests
 2015-12-18 11:40:13 -08:00
jon-wei 356b07c6c3 More efficient SegmentMetadataQuery 2015-12-17 12:46:23 -08:00
Bingkun Guo 1e5aa2f3ac fix getType() and Json serialization in ClientMergeQuery and add serde tests 2015-12-15 12:08:43 -06:00
Gian Merlino e6c2db89da Merge pull request #1921 from jon-wei/flat_json 
Add docs and benchmark for JSON flattening parser
 2015-12-14 20:53:54 -08:00
Xavier Léauté a2d0bea223 Merge pull request #2074 from guobingkun/add_comment 
Add comment to BrokerServerViewTest
 2015-12-09 16:19:27 -08:00
jon-wei c53bf85d83 Add docs and benchmark for JSON flattening parser 2015-12-09 16:13:30 -08:00
Bingkun Guo 31502a920c Add comment to BrokerServerViewTest 
Explain why we need to do explicit checks on existences of zNodes before we create them. Address issue: https://github.com/druid-io/druid/issues/1512
 2015-12-09 14:01:07 -06:00
Fangjin Yang ce79d707dd Merge pull request #2072 from guobingkun/fix_transient_error_brokerserverviewtest 
An effort to fix transient error in BrokerServerViewTest
 2015-12-09 11:12:35 -08:00
Bingkun Guo 161c4ce288 An effort to fix transient error in BrokerServerViewTest 2015-12-09 10:50:13 -06:00
Bingkun Guo c4de26ef09 fix typo in InitializationTest.java 2015-12-09 00:11:03 -06:00
Charles Allen 5bc6d61a25 Fix InitializationTest 2015-12-08 17:37:56 -08:00
Xavier Léauté dcd15735cd Merge pull request #2015 from metamx/handoff-notifier-coordinator 
Remove ServerView from RealtimeIndexTasks and use coordinator http endpoint for handoff information
 2015-12-08 14:06:39 -08:00
Nishant a32906c7fd Remove FilteredServerView 2015-12-09 01:54:12 +05:30
Nishant 9491e8de3b Remove ServerView from RealtimeIndexTasks and use coordinator http endpoint for handoffs 
- fixes #1970
- extracted out segment handoff callbacks in SegmentHandoffNotifier
which is responsible for tracking segment handoffs and doing callbacks
when handoff is complete.
- Coordinator now maintains a view of segments in the cluster, this
will affect the jam heap requirements for the overlord for large
clusters.
realtime index task and nodes now use HTTP end points exposed by the
coordinator to get serverView

review comment

fix realtime node guide injection

review comments

make test not rely on scheduled exec

fix compilation

fix import

review comment

introduce immutableSegmentLoadInfo

fix son reading

remove unnecessary logging
 2015-12-09 01:54:09 +05:30
Fangjin Yang 56fd48c0e4 Merge pull request #2056 from guobingkun/fix_server_selector2 
Bug fix: stale segment in ServerSelector
 2015-12-07 16:57:33 -08:00
Fangjin Yang ba980dcadb Merge pull request #2058 from cheddar/report-uncovered-segments 
Add "uncoveredIntervals" to responseContext
 2015-12-07 16:42:42 -08:00
Eric Tschetter a36185926e Add "uncoveredIntervals" to responseContext 
This change will cause the CachingClusteredClient to populate the "uncoveredIntervals"
key in the responseContext map.  The value will be any intervals that were requested in
the query but are not actually covered by the segments underlying the data source.

For unit testing, CachingClisteredClientTest is testing the caching behavior of the
object and it is pretty hard to adjust it to only test this new behavior, so I created
a new, parallel "CachingClusteredClientFunctionalityTest" to simplify testing just
basic functionality.
 2015-12-07 16:16:57 -08:00
Bingkun Guo 970b9cfcaa fix stale segment in ServerSelector 2015-12-07 14:04:34 -06:00