29d29ba98d
Merge pull request #2263 from jon-wei/flex_dims3
...
Allow IncrementalIndex to store Long/Float dimensions
2016-02-25 17:23:02 -08:00
b331fb4a83
Fix parsing of druid.indexer.server.maxChatRequests.
2016-02-25 14:47:15 -08:00
b82b487f20
remove extra kill parameter
2016-02-24 17:16:18 -06:00
c17ce02467
Allow IncrementalIndex to store Long/Float dimensions
2016-02-24 13:51:57 -08:00
a3b37e9225
In persistAndMerge, increase the scope of try-catch block so that any exception while persisting hydrants is caught and consequently that sink is abandoned or the task will forever wait for handoff to happen.
2016-02-23 22:22:33 -06:00
6c9e1a28ad
Merge pull request #2519 from gianm/unparseable-handling
...
Better handling of ParseExceptions.
2016-02-24 04:46:29 +05:30
93540c0631
Merge pull request #2503 from gianm/jetty-qos
...
Add druid.indexer.server.maxChatRequests for QoS; deprecate separate ports.
2016-02-23 10:35:53 -08:00
3534483433
Better handling of ParseExceptions.
...
Two changes:
- Allow IncrementalIndex to suppress ParseExceptions on "aggregate".
- Add "reportParseExceptions" option to realtime tuning configs. By default this is "false".
Behavior of the counters should now be:
- processed: Number of rows indexed, including rows where some fields could be parsed and some could not.
- thrownAway: Number of rows thrown away due to rejection policy.
- unparseable: Number of rows thrown away due to being completely unparseable (no fields salvageable at all).
If "reportParseExceptions" is true then "unparseable" will always be zero (because a parse error would
cause an exception to be thrown). In addition, "processed" will only include fully parseable rows
(because even partial parse failures will cause exceptions to be thrown).
Fixes #2510 .
2016-02-23 10:11:43 -08:00
0c984f9e32
Merge pull request #2109 from himanshug/segments_in_delta_ingestion
...
idempotent batch delta ingestion
2016-02-22 14:00:45 -08:00
3bdd757024
Merge pull request #1773 from b-slim/log_details
...
Adding downstream source when throwing QueryInterruptedException
2016-02-22 10:16:07 -08:00
21b0b8a07d
new coordinator endpoint to get list of used segment given a dataSource and list of intervals
2016-02-21 23:17:58 -06:00
77925cc061
adding downstream source of QueryInterruptedException
2016-02-20 13:05:14 -06:00
23c993c9e7
Add druid.indexer.server.maxChatRequests for QoS; deprecate separate ports.
...
- Add druid.indexer.server.maxChatRequests, which sets up a QoSFilter on the main Jetty server.
- Deprecate druid.indexer.runner.separateIngestionEndpoint
- Deprecate druid.indexer.server.chathandler.*
2016-02-19 13:36:09 -08:00
243ac5399b
Harmonize realtime indexing loop across the task and standalone nodes.
...
- Both now catch ParseExceptions on plumber.add (see https://groups.google.com/d/topic/druid-user/wmiRDvx2RvM/discussion )
- Standalone now treats IndexSizeExceededException as fatal (previously only the task did)
2016-02-19 07:34:15 -08:00
e0c049c0b0
Make startup properties logging optional.
...
Off by default, but enabled in the example config files. See also #2452 .
2016-02-12 14:12:16 -08:00
1430bc2c88
Merge pull request #2276 from harshjain2/feature-2021
...
Fix for issue 2021.
2016-02-10 17:04:45 -08:00
fa92b77f5a
Harmonize znode writing code in RTR and Worker.
...
- Throw most exceptions rather than suppressing them, which should help
detect problems. Continue suppressing exceptions that make sense to
suppress.
- Handle payload length checks consistently, and improve error message.
- Remove unused WorkerCuratorCoordinator.announceTaskAnnouncement method.
- Max znode length should be int, not long.
- Add tests.
2016-02-10 14:52:00 -08:00
a3eb863c8e
Fix for issue 2021
2016-02-10 22:19:12 +05:30
d1cb17d3f7
at broker - only add segments from specific tiers to the timeline
2016-02-09 22:33:22 -06:00
b40c342cd1
make Global stupid pool cache size configurable
2016-02-05 14:18:06 -06:00
9002548eeb
increase test time out and general clean up
2016-02-03 13:26:37 -06:00
5111fd52f2
Add check for log4j-core in Log4jShutterDownerModule
2016-02-02 15:56:48 -08:00
dc89cdd0f9
Merge pull request #2336 from himanshug/fix_2331
...
limit size of X-Druid-Response-Context header to 7K
2016-02-02 12:06:59 -06:00
c03918f89a
AsyncQueryForwardingServletTest#testDeleteBroadcast sometimes fails by port conflict
2016-01-29 19:28:58 +09:00
f6b4dbd697
bug fix and unit tests for DruidCoordinatorSegmentKiller
2016-01-28 14:10:17 -06:00
ab3edfa8fc
moving DruidCoordinatorSegmentKiller class out of DruidCoordinator
2016-01-28 14:03:56 -06:00
3880f54b87
Merge pull request #2332 from himanshug/configurable_partial
...
make populateUncoveredIntervals a configuration in query context
2016-01-28 10:34:35 +05:30
a7bde8f4da
limit size of X-Druid-Response-Context header to 7K due to https://github.com/druid-io/druid/issues/2331
2016-01-27 15:18:08 -06:00
5a3642bb93
Merge pull request #2247 from metamx/pedanticBuild
...
Enable strict building in travis
2016-01-27 10:27:03 -08:00
508734c8b0
Long constant reformatting in tests `l` --> `L`
2016-01-27 08:59:19 -08:00
fd6bf3fe22
Use interval comparator instead of bucketMonthComparator
...
fix when two segments have same interval
review comments
2016-01-27 17:35:43 +05:30
3719b6e3c8
make populateUncoveredIntervals a configuration in query context
2016-01-26 15:13:45 -06:00
41730b96d4
Fix for issue 2021
2016-01-25 02:48:22 +05:30
7a6109f0ca
Merge pull request #2321 from gianm/pending-index
...
Replace two-column index on pendingSegments table with one-column index.
2016-01-22 13:38:15 -06:00
0bd9bff075
Replace two-column index on pendingSegments table with one-column index.
...
Fixes #2319 .
2016-01-22 10:50:21 -08:00
1a5d4e714c
Adding custom mapper for json processing exception to return bad request instead of 500
2016-01-22 09:48:52 -08:00
04d3054353
Merge pull request #2303 from CHOIJAEHONG1/localfirehouse-basedir-npe
...
Throw an IAE when baseDir is null in LocalFireHose
2016-01-21 07:58:52 -08:00
dcb7830330
Merge pull request #984 from drcrallen/thread-priority-rebase
...
Use thread priorities. (aka set `nice` values for background-like tasks)
2016-01-21 15:02:34 +05:30
2e1d6aaf3d
Use thread priorities. (aka set `nice` values for background-like tasks)
...
* Defaults the thread priority to java.util.Thread.NORM_PRIORITY in io.druid.indexing.common.task.AbstractTask
* Each exec service has its own Task Factory which is assigned a priority for spawned task. Therefore each priority class has a unique exec service
* Added priority to tasks as taskPriority in the task context. <0 means low, 0 means take default, >0 means high. It is up to any particular implementation to determine how to handle these numbers
* Add options to ForkingTaskRunner
* Add "-XX:+UseThreadPriorities" default option
* Add "-XX:ThreadPriorityPolicy=42" default option
* AbstractTask - Removed unneded @JsonIgnore on priority
* Added priority to RealtimePlumber executors. All sub-executors (non query runners) get Thread.MIN_PRIORITY
* Add persistThreadPriority and mergeThreadPriority to realtime tuning config
2016-01-20 14:00:31 -08:00
61aca6f9cc
remove wrong checks
...
sink never have null hydrants and hydrants never have null adapters
2016-01-20 23:43:53 +05:30
7132428bba
throw IAE when baseDir is null in LocalFireHose
2016-01-21 01:27:32 +09:00
59ea186af7
fix reference counting for segments
2016-01-20 17:24:21 +05:30
747343e621
Preserve dimension order across indexes during ingestion
2016-01-19 13:34:11 -08:00
df2906a91c
Merge pull request #2290 from gianm/index-merger-v9-stuff
...
Respect buildV9Directly in PlumberSchools, so it works on standalone realtime.
2016-01-19 13:04:00 -08:00
0c31f007fc
Merge pull request #1728 from himanshug/aggregators_in_segment_metadata
...
Store AggregatorFactory[] in segment metadata
2016-01-19 12:55:49 -08:00
fe841fd961
Merge pull request #2118 from guobingkun/fix_segment_loading
...
Fix loading segment for historical
2016-01-19 14:25:48 -06:00
a99aef29a1
adding aggregators to segment metadata
2016-01-19 14:23:39 -06:00
1dcf22edb7
Respect buildV9Directly in PlumberSchools, so it works on standalone realtime nodes.
...
Also parameterize some tests to run with/without buildV9Directly:
- IndexGeneratorJobTest
- RealtimeIndexTaskTest
- RealtimePlumberSchoolTest
2016-01-19 12:15:06 -08:00
c4ad50f92c
Fix loading segment for historical
...
Historical will drop a segment that shouldn't be dropped in the following scenario:
Historical node tried to load segmentA, but failed with SegmentLoadingException,
then ZkCoordinator called removeSegment(segmentA, blah) to schedule a runnable that would drop segmentA by deleting its files. Now, before that runnable executed, another LOAD request was sent to this historical, this time historical actually succeeded on loading segmentA and announced it. But later on, the scheduled drop-of-segment runnable started executing and removed the segment files, while historical is still announcing segmentA.
2016-01-19 10:29:49 -06:00
164b0aad7a
removing Map<String,Object> segmentMetadata from methods in Index[Maker/Merger] and using Metadata class
...
instead of a Map to store segment metadata
2016-01-18 22:03:46 -06:00
637d2605e7
kill unwanted parent directories when a segment is deleted from LocalDataSegmentKiller
2016-01-18 21:51:04 -06:00
71f1cd5e34
Fix for issue 2021
2016-01-17 16:10:04 +05:30
f6a1a4ae20
Merge pull request #2138 from KurtYoung/feature-build-v9
...
build v9 directly
2016-01-16 13:35:46 -06:00
82ff98c2bf
add config for build v9 directly and update docs
2016-01-16 11:26:34 +08:00
2e54553a8f
Merge pull request #1990 from himanshug/schedule_kill_task
...
support periodic hard delete of segments
2016-01-15 15:22:33 -06:00
7c65880e55
Merge pull request #2270 from rasahner/warnOfChatHandlerNoop
...
if chathandler is noop, log using warn instead of info
2016-01-15 13:29:27 -07:00
6ec6835b5d
Fix for issue 2021.
2016-01-16 00:58:33 +05:30
34cd8f8c72
Merge pull request #2258 from fjy/acl-zk
...
acl for zookeeper is added
2016-01-15 10:27:08 -07:00
a54c726726
Merge pull request #2266 from anubhgup/fix-announce
...
Fix loss in segment announcements when segments do not fit in zNode
2016-01-14 17:10:43 -08:00
dc1a62c3d9
Merge pull request #2248 from metamx/druidNodeHashEquals
...
Add hashCode and equals to DruidNode
2016-01-14 16:04:58 -08:00
6d09ab839f
Fix for loss in segment announcements when segments do not fit in the znodes during compress mode.
...
Added unit test (from Navis).
2016-01-14 14:44:23 -08:00
7361cd173f
if chathandler is noop, log using warn instead of info
2016-01-14 10:12:42 -06:00
1908d63162
acl for zookeeper is added
2016-01-13 14:56:05 -08:00
18479bb757
time-descending result of timeseries queries
2016-01-13 12:23:01 +09:00
eb2d251ac8
support periodic hard delete of segments
2016-01-12 16:55:05 -06:00
01a0715ee2
Merge pull request #2161 from metamx/query-metrics-timeout
...
Fix Query metrics for query timeout
2016-01-12 09:50:40 -06:00
ef3cddabe9
adding UTs for DruidCoordinatorConfig
2016-01-11 22:03:25 -06:00
b0e04a9162
Add hashCode and equals to DruidNode
2016-01-11 15:23:45 -08:00
ea623e43d2
Merge pull request #2240 from metamx/fix-load-rule
...
Fix loadRule when one of the tiers had no available servers
2016-01-11 10:05:31 -08:00
32bc2f776e
Fix loadRule when one of the tiers had no servers
...
When one of the tiers have no servers, LoadRule should ignore that tier
and continue to load/drop segments in other available tiers.
the bug also causes whacky behavior with LoadRule with non existent
tier where the segment balancer keeps on moving segments to other nodes
in existing tiers but the extra segment copies are never dropped
eventually leading to all the tiers getting full .
2016-01-11 15:53:14 +05:30
d255f4baac
Merge pull request #2234 from pjain1/emit_realtime_metrics
...
emit handoff count metrics
2016-01-08 14:24:16 -06:00
9dba0f67e7
emit handoff count metrics
2016-01-08 12:36:13 -06:00
15fc070232
Merge pull request #2213 from himanshug/fix_curtator_test_base
...
[wip] trying/finding fix for announcer test failures
2016-01-07 18:23:49 -08:00
1bfb4e3988
Emit query/time for failed and timeout queries
...
emit query/time metric also
add success flag
fix success flag for router metrics
review comments
formatting.
2016-01-07 19:41:54 +05:30
7ab810f3eb
do not ignore exceptions from curator cleanup in CuratorTestBase
2016-01-06 10:42:53 -06:00
14989f272d
Add metrics for ingest/bytes/received for EventReceiverFirehose
...
review comments
review comments
2016-01-05 20:06:09 +05:30
57d91d754d
Comment out buggy unit tests, fix #2185
2016-01-03 09:50:16 -08:00
8424b2b456
Fix announcer test bad check
2016-01-01 19:55:22 -08:00
fa5c3bb014
adding decorate(DimensionSelector) to DimensionSpec to enable support for arbitrary filtering/transformations to returned dimension values
2015-12-30 15:06:24 -06:00
3c107c5757
Merge pull request #2150 from himanshug/emit_query_bytes
...
emit query/bytes metric
2015-12-30 13:44:19 -06:00
7ffa706655
Merge pull request #2152 from metamx/add-taskId
...
Add taskId to realtimeMetrics
2015-12-29 10:33:40 -08:00
b1261035a7
Merge pull request #1861 from guobingkun/insert_segment_tool
...
insert-segment tool
2015-12-29 10:06:07 -08:00
38b0f1fbc2
fix transient failures in unit tests
2015-12-28 20:03:30 -08:00
d94821998f
Merge pull request #2153 from navis/fix-testfail-DruidCoordinatorTest
...
Another try to fix DruidCoordinatorTest#testCoordinatorRun
2015-12-27 17:03:46 -08:00
10c19fe9f8
Another try to fix DruidCoordinatorTest#testCoordinatorRun
2015-12-24 03:02:37 +09:00
2c3c4a3f8f
Another try to fix xxServerViewTests
2015-12-24 02:13:40 +09:00
508629916d
Merge pull request #2137 from guobingkun/fix_serverview_transient_error2
...
[Don't merge]Another effort on fixing the transient error for XXXServerViewTest
2015-12-23 07:33:54 -08:00
978a3fd8ae
Add taskId to realtimeMetrics
...
Add task Id to Realtime Metrics
2015-12-23 18:05:25 +05:30
1a8546a682
emit query/bytes metric
2015-12-23 00:29:44 -06:00
89e2a9a6d2
build at Tue Dec 22 21:30:00 CST 2015
2015-12-22 21:30:00 -06:00
a110eafff7
build at Tue Dec 22 17:30:00 CST 2015
2015-12-22 17:30:00 -06:00
4b93528334
build at Tue Dec 22 17:00:01 CST 2015
2015-12-22 17:00:01 -06:00
b8d9860dbc
build at Tue Dec 22 16:00:00 CST 2015
2015-12-22 16:00:00 -06:00
ffdb9407e5
build at Tue Dec 22 15:30:00 CST 2015
2015-12-22 15:30:00 -06:00
6f54c1b60d
build at Tue Dec 22 15:00:00 CST 2015
2015-12-22 15:00:01 -06:00
4e61261c68
build at Tue Dec 22 14:30:00 CST 2015
2015-12-22 14:30:00 -06:00
4961b8c0bd
build at Tue Dec 22 14:00:00 CST 2015
2015-12-22 14:00:00 -06:00
0a635a11d6
build at Tue Dec 22 13:40:00 CST 2015
2015-12-22 13:40:00 -06:00
05c9e1b598
Reorder Before/After in JDBCExtractionNamespaceTest
...
* Fixes https://github.com/druid-io/druid/issues/2120
2015-12-22 11:39:46 -08:00
849ab2d958
build at
2015-12-22 13:08:29 -06:00
455980d659
address another node exists failure
2015-12-22 13:06:20 -06:00
2ffeda5d25
build number 13
2015-12-22 10:19:39 -06:00
7e64d5179f
Merge pull request #2139 from himanshug/emit_query_node_bytes
...
emit query/node/bytes metric
2015-12-22 08:07:04 -08:00
bc573484f3
build number 12
2015-12-22 00:52:58 -06:00
7b5fd76058
Merge pull request #2119 from navis/fix-sporadic-testfail2
...
Fix sporadic fail of DruidCoordinatorTest#testCoordinatorRun
2015-12-21 23:28:32 -06:00
b96f560255
emit query/node/bytes metric
2015-12-21 23:23:20 -06:00
06c220ae7b
build number 11
2015-12-21 19:40:47 -06:00
93a91e5633
build number 10
2015-12-21 18:21:15 -06:00
471dd7e164
build number 9
2015-12-21 17:35:18 -06:00
627c311923
build #8
2015-12-21 17:06:37 -06:00
293f2ad65d
build #7
2015-12-21 16:09:19 -06:00
9ed843ea21
build #6
2015-12-21 15:41:23 -06:00
f8342ca240
build #5
2015-12-21 15:10:37 -06:00
52261c5539
build #4
2015-12-21 14:41:38 -06:00
47a8f2d3f1
build #3
2015-12-21 13:56:43 -06:00
5cc56cf9e7
build #2
2015-12-21 13:05:38 -06:00
f12cfa99fa
another effort on fixing the transient error for Broker/CoordinatorServerViewTest happend on Travis build
2015-12-21 12:31:36 -06:00
c2a2d19d62
Merge pull request #2126 from guobingkun/remove_single_announcer
...
Remove SingleDataSegmentAnnouncer in favor of BatchDataSegmentAnnouncer
2015-12-21 08:55:07 -08:00
7b24d1f355
Fix sporadic fail of DruidCoordinatorTest#testCoordinatorRun
2015-12-21 17:00:52 +09:00
951a4e9b35
Remove SingleDataSegmentAnnouncer in favor of BatchDataSegmentAnnouncer
2015-12-21 00:05:53 -06:00
89b477970f
DataSegmentFinder tool
...
`insert-segment-to-db` is a tool that can insert segments into Druid metadata storage. It is intended to be used
to update the segment table in metadata storage after people manually migrate segments from one place to another.
It can also be used to insert missing segment into Druid, or even recover metadata storage by telling it where the
segments are stored.
Note: This tool expects users to have Druid cluster running in a "safe" mode, where there are no active tasks to interfere
the segments being inserted. Users can optionally bring down the cluster to make 100% sure nothing is interfering.
2015-12-21 00:02:04 -06:00
ad3312171b
Fix sporadic fail of io.druid.client.CoordinatorServerViewTest
2015-12-21 11:39:32 +09:00
7019d3c421
Merge pull request #2107 from jon-wei/fix_smq
...
More efficient SegmentMetadataQuery
2015-12-18 16:40:47 -08:00
e52c9d2b3c
Merge pull request #2095 from guobingkun/fix_client_merge_task_and_serde
...
Fix getType() and Json serialization in ClientMergeQuery + serde tests
2015-12-18 11:40:13 -08:00
356b07c6c3
More efficient SegmentMetadataQuery
2015-12-17 12:46:23 -08:00
1e5aa2f3ac
fix getType() and Json serialization in ClientMergeQuery and add serde tests
2015-12-15 12:08:43 -06:00
e6c2db89da
Merge pull request #1921 from jon-wei/flat_json
...
Add docs and benchmark for JSON flattening parser
2015-12-14 20:53:54 -08:00
a2d0bea223
Merge pull request #2074 from guobingkun/add_comment
...
Add comment to BrokerServerViewTest
2015-12-09 16:19:27 -08:00
c53bf85d83
Add docs and benchmark for JSON flattening parser
2015-12-09 16:13:30 -08:00
31502a920c
Add comment to BrokerServerViewTest
...
Explain why we need to do explicit checks on existences of zNodes before we create them. Address issue: https://github.com/druid-io/druid/issues/1512
2015-12-09 14:01:07 -06:00
ce79d707dd
Merge pull request #2072 from guobingkun/fix_transient_error_brokerserverviewtest
...
An effort to fix transient error in BrokerServerViewTest
2015-12-09 11:12:35 -08:00
161c4ce288
An effort to fix transient error in BrokerServerViewTest
2015-12-09 10:50:13 -06:00
c4de26ef09
fix typo in InitializationTest.java
2015-12-09 00:11:03 -06:00
5bc6d61a25
Fix InitializationTest
2015-12-08 17:37:56 -08:00
dcd15735cd
Merge pull request #2015 from metamx/handoff-notifier-coordinator
...
Remove ServerView from RealtimeIndexTasks and use coordinator http endpoint for handoff information
2015-12-08 14:06:39 -08:00
a32906c7fd
Remove FilteredServerView
2015-12-09 01:54:12 +05:30
9491e8de3b
Remove ServerView from RealtimeIndexTasks and use coordinator http endpoint for handoffs
...
- fixes #1970
- extracted out segment handoff callbacks in SegmentHandoffNotifier
which is responsible for tracking segment handoffs and doing callbacks
when handoff is complete.
- Coordinator now maintains a view of segments in the cluster, this
will affect the jam heap requirements for the overlord for large
clusters.
realtime index task and nodes now use HTTP end points exposed by the
coordinator to get serverView
review comment
fix realtime node guide injection
review comments
make test not rely on scheduled exec
fix compilation
fix import
review comment
introduce immutableSegmentLoadInfo
fix son reading
remove unnecessary logging
2015-12-09 01:54:09 +05:30
56fd48c0e4
Merge pull request #2056 from guobingkun/fix_server_selector2
...
Bug fix: stale segment in ServerSelector
2015-12-07 16:57:33 -08:00
ba980dcadb
Merge pull request #2058 from cheddar/report-uncovered-segments
...
Add "uncoveredIntervals" to responseContext
2015-12-07 16:42:42 -08:00
a36185926e
Add "uncoveredIntervals" to responseContext
...
This change will cause the CachingClusteredClient to populate the "uncoveredIntervals"
key in the responseContext map. The value will be any intervals that were requested in
the query but are not actually covered by the segments underlying the data source.
For unit testing, CachingClisteredClientTest is testing the caching behavior of the
object and it is pretty hard to adjust it to only test this new behavior, so I created
a new, parallel "CachingClusteredClientFunctionalityTest" to simplify testing just
basic functionality.
2015-12-07 16:16:57 -08:00
970b9cfcaa
fix stale segment in ServerSelector
2015-12-07 14:04:34 -06:00
00416e26ff
Merge pull request #2012 from himanshug/add_comment
...
adding more comments to why drops do not happen
2015-12-07 11:50:06 -08:00
62ba9ade37
unifying license header in all java files
2015-12-05 22:16:23 -06:00
3781aa380f
Added an injection test for ZkCoordinator.
2015-12-05 11:56:22 +08:00
221fb95d07
add support for getting used segments for multiple interval in IndexerMetadataStorageCoordinator
2015-12-03 21:28:04 -06:00
1eb8211346
Add datasource and taskId to metrics emitted by peons
...
This PR adds the datasource and taskId to the jvm and sys metrics
emitted by the peons.
fix spelling
review comment
review comment
2015-12-01 23:20:59 +05:30
7c91219130
adding more comments to why drops do not happen when there are no segments in db
2015-11-25 16:26:51 -06:00
8e83d800d6
Merge pull request #1881 from gianm/restartable-tasks
...
Restorable indexing tasks
2015-11-23 21:14:37 -08:00
13af260102
Merge pull request #1957 from metamx/skip-incremental-segment
...
Ability to skip Incremental Index during query using query context
2015-11-23 17:32:57 -08:00
501dcb43fa
Some changes that make it possible to restart tasks on the same hardware.
...
This is done by killing and respawning the jvms rather than reconnecting to existing
jvms, for a couple reasons. One is that it lets you restore tasks after server reboots
too, and another is that it lets you upgrade all the software on a box at once by just
restarting everything.
The main changes are,
1) Add "canRestore" and "stopGracefully" methods to Tasks that say if a task can
stop gracefully, and actually do a graceful stop. RealtimeIndexTask is the only
one that currently implements this.
2) Add "stop" method to TaskRunners that attempts to do an orderly shutdown.
ThreadPoolTaskRunner- call stopGracefully on restorable tasks, wait for exit
ForkingTaskRunner- close output stream to restorable tasks, wait for exit
RemoteTaskRunner- do nothing special, we actually don't want to shutdown
3) Add "restore" method to TaskRunners that attempts to bootstrap tasks from last run.
Only ForkingTaskRunner does anything here. It maintains a "restore.json" file with
a list of restorable tasks.
4) Have the CliPeon's ExecutorLifecycle lock the task base directory to avoid a restored
task and a zombie old task from stomping on each other.
2015-11-23 11:22:08 -08:00
36569094ce
EventReceiverFirehose: Drain buffer when closed, until empty.
2015-11-20 19:12:59 -08:00
21c84b5ff7
Merge pull request #1896 from gianm/allocate-segment
...
SegmentAllocateAction (fixes #1515 )
2015-11-18 21:05:46 -08:00
60f649dab1
Ability to skip Incremental Index during query using query context
...
This PR adds adds the ability to skip incremental index when querying
results from realtime nodes. default behaviour is to include
incrementalIndex in queries.
review comment
2015-11-19 00:29:41 +05:30
8fcf2403e3
Merge pull request #1943 from metamx/realtime-caching
...
Enable caching on intermediate realtime persists
2015-11-17 15:06:43 -08:00
8fa34ee671
Merge pull request #1973 from metamx/cachemonitor-optional-injection
...
CacheMonitor - make cache injection optional
2015-11-17 13:02:06 -08:00
9118b010c0
Merge pull request #1976 from noddi/feature/count-parameter-rules-history
...
Add count parameter to rules history endpoint.
2015-11-17 14:41:50 -06:00
dbe201aeed
Merge pull request #1929 from pjain1/jetty_threads
...
separate ingestion and query thread pool
2015-11-17 12:14:25 -08:00
6c498b7d4a
separate ingestion and query thread pool
2015-11-17 13:42:41 -06:00
b0a444eafc
Add count parameter to rules history endpoint.
2015-11-17 20:14:12 +01:00
d7eb2f717e
enable query caching on intermediate realtime persists
2015-11-17 10:58:00 -08:00
71376efd57
make cache injection optional
...
allows enabling the CacheMonitor for all index tasks, even if some don't
bind any cache instance.
2015-11-16 22:40:01 -08:00
4f46d457f1
Merge pull request #1947 from noddi/feature/count-parameter-history-endpoints
...
Add count parameter to history endpoints
2015-11-12 10:23:44 -08:00
465cbcf9a7
Merge pull request #1956 from metamx/remove-unused-imports
...
Cleanup + remove unused imports
2015-11-11 17:36:47 -08:00
e4e5f0375b
SegmentAllocateAction ( fixes #1515 )
...
This is a feature meant to allow realtime tasks to work without being told upfront
what shardSpec they should use (so we can potentially publish a variable number
of segments per interval).
The idea is that there is a "pendingSegments" table in the metadata store that
tracks allocated segments. Each one has a segment id (the same segment id we know
and love) and is also part of a sequence.
The sequences are an idea from @cheddar that offers a way of doing replication.
If there are N tasks reading exactly the same data with exactly the same logic
(think Kafka tasks reading a fixed range of offsets) then you can place them
in the same sequence, and they will generate the same sequence of segments.
2015-11-11 16:54:35 -08:00
dc4ae59aaf
Merge pull request #1865 from noddi/bugfix/issue-560
...
Fix #560 - datasource segments summary shouldn't include replication.
2015-11-11 15:29:08 -08:00
6e5d2c6745
Add count parameter to history endpoints.
2015-11-11 23:03:57 +01:00
8e743b70c6
SegmentIdentifier, like a pre-DataSegment.
2015-11-11 12:25:43 -08:00
e26f7fca36
cleanup cache interface
2015-11-11 12:25:22 -08:00
fa6142e217
cleanup and remove unused imports
2015-11-11 12:25:21 -08:00
f2c271e500
Better error message when LocalDataSegmentPusher cannot create its directory.
2015-11-10 19:49:01 -08:00
cf779946ef
Merge pull request #1791 from guobingkun/event_receiver_firehose_monitor
...
EventReceiverFirehoseMonitor
2015-11-10 11:09:42 -08:00
a57cbfd2c3
Merge pull request #1387 from metamx/enableShutdownLogging
...
Add special handler to allow logger messages during shutdown
2015-11-09 17:20:09 -08:00
1df4baf489
Move Jackson Guice adapters into io.druid
...
* Removes access to protected methods in com.fasterxml
* Eliminates druid-common's use of foreign package com.fasterxml
2015-11-09 10:50:45 -08:00
fe6efb6367
Merge pull request #1932 from gianm/data-schema-exclude-metric-names
...
DataSchema: Exclude metric names from dimension list.
2015-11-07 15:22:33 -08:00
8b5de492cc
Fix #560 - datasource segments summary shouldn't include replication.
2015-11-07 21:43:47 +01:00
6b908a5061
DataSchema: Exclude metric names from dimension list.
...
Otherwise we could end up with a metric and a dimension with the same column name.
2015-11-07 12:25:36 -08:00
e9533db987
Merge pull request #1850 from metamx/friendlyBardCache
...
Allow setting upper limit on the number of cache segments a broker will try to fetch.
2015-11-06 10:25:49 -08:00
c896818241
Update curator to 2.9.1
...
Lots of bugfixes since 2.8.0
- https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314425&version=12333324
- https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314425&version=12332392
2015-11-05 15:53:01 -08:00
4efc90dda3
fix zkcoordinator inject
2015-11-04 17:30:17 -08:00
8f231fd3e3
cleanup druid codebase
2015-11-04 13:59:53 -08:00
3b3d88bd03
Merge pull request #1857 from noddi/feature/intervals-route
...
Queryable intervals
2015-11-03 13:19:04 -08:00
8b67417ac8
make methods in Index[Merger,Maker,IO] non-static so that they can have
...
appropriate ObjectMapper injected instead of creating one statically
2015-11-02 23:24:26 -06:00
f90ddfdb89
Merge pull request #1745 from himanshug/numbered_to_elastic
...
making NumberedShardSpec elastic
2015-10-30 16:01:03 -07:00
c3b6fcce9d
Add EventReceiverFirehoseMonitor
...
add an EventReceiverFirehoseMonitor so that we can monitor how many
events have been queued in the EventReceiverFirehose and get a sense
about whether the firehose is under too much pressure.
2015-10-30 11:40:02 -05:00
6de51e022d
Coordinator - add intervals route.
2015-10-30 11:51:42 +01:00
149333d77d
forward cancellation request to all brokers, fixes #1802
2015-10-29 16:58:38 -07:00
dfce14ed17
Allow setting upper limit on the number of cache segments a broker will try to fetch.
2015-10-29 11:50:00 -07:00
59872bd0cd
Merge pull request #1809 from metamx/fifoPriorityExecutorService
...
Make PrioritizedExecutorService optionally FIFO
2015-10-27 15:19:32 -07:00
ea2267e08c
Merge pull request #1868 from gianm/fix-announcements
...
Historical and MiddleManager server announcements should not remove parents.
2015-10-27 14:50:05 -07:00
7df7370935
Merge pull request #1862 from metamx/indexingServiceMMGone
...
Add timeout to shutdown request to middle manager for indexing service
2015-10-27 14:38:01 -07:00
7a2ceef690
Add special handler to allow logger messages during shutdown
...
* Adds a special PropertyChecker interface which is ONLY for setting string properties at the very start of psvm
2015-10-27 14:33:36 -07:00
ecdafa87c5
Make PrioritizedExecutorService optionally FIFO
2015-10-27 14:16:22 -07:00
67d58f6d8d
Update CoordinatorRuleManager for http-client 1.0.4
2015-10-27 14:08:45 -07:00
4b92752deb
Historical and MiddleManager server announcements should not remove parents.
...
Removing parent paths causes watchers of the "announcements" path to get stuck
and stop seeing new updates.
2015-10-27 08:06:11 -07:00
5a082b2f5e
Merge pull request #1824 from metamx/UniformGranularitySpecHashEquals
...
Add hashCode and equals to UniformGranularitySpec
2015-10-26 09:34:01 -07:00
5f23703216
Merge pull request #1638 from guobingkun/remove_maven_client_code
...
Remove Maven client at runtime + Provide a way to load Druid extensions through local file system
2015-10-26 09:30:05 -07:00
72c408cf2d
Merge pull request #1770 from metamx/merge-time
...
Add segment merge time as a metric
2015-10-22 22:03:41 -07:00
853f95f64c
Merge pull request #1845 from metamx/moreVerboseRuleAuditLogging
...
Add more verbose logging to SQLMetadataRuleManager
2015-10-22 16:12:46 -05:00
7cecc55045
Add segment merge time as a metric
...
Add merge and persist cpu time
Fix typo
review comment
move cpu time measuring to VMUtils
review comments.
2015-10-22 12:28:03 +05:30
6ceab03fc4
Add debug logging to start of segment manager polling
2015-10-21 16:17:49 -07:00
Fangjin Yang
Gian Merlino
Parag Jain
jon-wei
Himanshu Gupta
Nishant
Slim Bouguerra
Harsh Jain
Charles Allen
navis.ryu
Xavier Léauté
Himank Chaudhary
Charles Allen
Jaehong Choi
Jonathan Wei
Bingkun Guo
Kurt Young
David Lim
Anubhav Gupta
Robin
Nikita Geer
Eric Tschetter
刘麒贇
Bartosz Ługowski
Xavier Léauté