81cab8a7bb
Make lookups more idempotent on update requests. ( #2954 )
...
* No longer fails if an update fails but it shouldn't have replaced it
2016-05-11 11:22:35 -07:00
f2510cf125
Remove DataSchema equals() and hashcode()
2016-05-10 16:09:28 -07:00
6332bd70f4
Add smile provider ( #2951 )
2016-05-10 16:03:39 -07:00
0c04650e69
Lookup Announcer eager starting ( #2944 )
2016-05-10 12:21:47 +05:30
454bb034f1
Nicer toString on ListneingAnnouncerConfig ( #2936 )
...
* Helps with debugging
2016-05-10 12:21:06 +05:30
2cfd337378
Merge pull request #2933 from dclim/SQLMetadataSupervisorManagerTest-fix
...
add uuid to primary key for supervisor table
2016-05-09 10:41:32 -06:00
a2dd57cf65
Optimize CostBalancerStrategy ( #2910 )
...
* Optimize CostBalancerStrategy
Ignore benchmark test in normal run
fix test
review comments
fix compilation
fix test
* review comments
* review comment
2016-05-05 08:29:08 -07:00
b489f63698
Supervisor for KafkaIndexTask ( #2656 )
...
* supervisor for kafka indexing tasks
* cr changes
2016-05-04 23:13:13 -07:00
841be5c61f
periodically emit metric segment/scan/pending ( #2854 )
2016-05-02 22:38:13 -07:00
6c5bf91f9a
publish metrics numJettyConns to see how number of active jetty connections change over time ( #2839 )
...
this can be compared with numer of active queries to see if requests are waiting in jetty queue
2016-05-02 14:08:25 -07:00
6b957aa072
[QTL] Make URI Exctraction Namespace take more sane arguments ( #2738 )
...
* Make URI Exctraction Namespace take more sane arguments
* Fixes https://github.com/druid-io/druid/issues/2669
* Update docs
* Rename error message
* Undo overzealous deletion of docs
* Explain caching mechanism a bit more in docs
2016-05-02 12:54:34 -07:00
5f0a9ccc57
fix ClassCastException in FiniteAppenderatorDriver ( #2896 )
2016-04-28 18:39:24 -07:00
0d745ee120
Basic authorization support in Druid ( #2424 )
...
- Introduce `AuthorizationInfo` interface, specific implementations of which would be provided by extensions
- If the `druid.auth.enabled` is set to `true` then the `isAuthorized` method of `AuthorizationInfo` will be called to perform authorization checks
- `AuthorizationInfo` object will be created in the servlet filters of specific extension and will be passed as a request attribute with attribute name as `AuthConfig.DRUID_AUTH_TOKEN`
- As per the scope of this PR, all resources that needs to be secured are divided into 3 types - `DATASOURCE`, `CONFIG` and `STATE`. For any type of resource, possible actions are - `READ` or `WRITE`
- Specific ResourceFilters are used to perform auth checks for all endpoints that corresponds to a specific resource type. This prevents duplication of logic and need to inject HttpServletRequest inside each endpoint. For example
- `DatasourceResourceFilter` is used for endpoints where the datasource information is present after "datasources" segment in the request Path such as `/druid/coordinator/v1/datasources/`, `/druid/coordinator/v1/metadata/datasources/`, `/druid/v2/datasources/`
- `RulesResourceFilter` is used where the datasource information is present after "rules" segment in the request Path such as `/druid/coordinator/v1/rules/`
- `TaskResourceFilter` is used for endpoints is used where the datasource information is present after "task" segment in the request Path such as `druid/indexer/v1/task`
- `ConfigResourceFilter` is used for endpoints like `/druid/coordinator/v1/config`, `/druid/indexer/v1/worker`, `/druid/worker/v1` etc
- `StateResourceFilter` is used for endpoints like `/druid/broker/v1/loadstatus`, `/druid/coordinator/v1/leader`, `/druid/coordinator/v1/loadqueue`, `/druid/coordinator/v1/rules` etc
- For endpoints where a list of resources is returned like `/druid/coordinator/v1/datasources`, `/druid/indexer/v1/completeTasks` etc. the list is filtered to return only the resources to which the requested user has access. In these cases, `HttpServletRequest` instance needs to be injected in the endpoint method.
Note -
JAX-RS specification provides an interface called `SecurityContext`. However, we did not use this but provided our own interface `AuthorizationInfo` mainly because it provides more flexibility. For example, `SecurityContext` has a method called `isUserInRole(String role)` which would be used for auth checks and if used then the mapping of what roles can access what resource needs to be modeled inside Druid either using some convention or some other means which is not very flexible as Druid has dynamic resources like datasources. Fixes #2355 with PR #2424
2016-04-28 16:50:28 -07:00
c29cb7d711
add pending task based resource management strategy ( #2086 )
2016-04-27 10:40:53 -07:00
984a518c9f
Merge pull request #2734 from b-slim/LookupIntrospection2
...
[QTL][Lookup] adding introspection endpoint
2016-04-21 12:15:57 -05:00
c74391e54c
JavaScript: Ability to disable. ( #2853 )
...
Fixes #2852 .
2016-04-21 09:43:15 -05:00
5938d9085b
Stream segments from database ( #2859 )
...
* Avoids fetching all segment records into heap by JDBC driver
* Set connection to read-only to help database optimize queries
* Update JDBC drivers (MySQL has fixes for streaming results)
2016-04-21 05:40:07 +08:00
fc91120b54
Merge pull request #2857 from metamx/upgrade-zk
...
upgrade zookeeper client dependency to 3.4.8
2016-04-20 10:36:07 +05:30
dbf63f738f
Add ability to filter segments for specific dataSources on broker without creating tiers ( #2848 )
...
* Add back FilteredServerView removed in a32906c7fd
2016-04-19 10:10:06 -07:00
08c784fbf6
KafkaIndexTask: Use a separate sequence per Kafka partition in order to make ( #2844 )
...
segment creation deterministic.
This means that each segment will contain data from just one Kafka
partition. So, users will probably not want to have a super high number
of Kafka partitions...
Fixes #2703 .
2016-04-18 22:29:52 -07:00
7b65ca7889
refactor ClientQuerySegmentWalker ( #2837 )
...
* refactor ClientQuerySegmentWalker
* add header to FluentQueryRunnerBuilder
* refactor QueryRunnerTestHelper
2016-04-18 14:00:47 -07:00
c1e690288c
Improve some log ( #2807 )
2016-04-15 09:34:26 -07:00
632b21472b
fix test failure ( #2818 )
...
formatting changes
2016-04-14 21:40:19 -07:00
886ee4e30d
Merge pull request #2821 from metamx/review-comments-2784
...
handle review comments for PR 2784
2016-04-12 10:20:43 -07:00
b486eff6b7
Merge pull request #2805 from metamx/query-time-start
...
request log should reflect time the query was received
2016-04-12 09:44:42 -07:00
deb6ecf919
handle review comments for PR 2784
...
https://github.com/druid-io/druid/pull/2784#discussion_r59062021
2016-04-12 21:52:00 +05:30
aa6a230c90
remove DruidSQL.g4, its failing with newer version of ANTLR, will bring it back and fix if needed later
2016-04-08 11:46:21 -05:00
d4d1d615c1
request log should reflect time the query was received, as opposed to processed
2016-04-07 12:39:34 -07:00
edd74f2b67
Allow Lite DataSegment Announcements
...
separate config for each skipping dimensions, metrics and loadSpec
Add test
fix test comment
Add docs
2016-04-07 18:24:12 +05:30
0e481d6f93
Allow filters to use extraction functions
2016-04-05 13:24:56 -07:00
728da75224
remove unused code
2016-04-01 13:10:35 -07:00
9cb197adec
Merge pull request #2722 from himanshug/fix_hadoop_jar_upload
...
config to explicitly specify classpath for hadoop container during hadoop ingestion
2016-03-28 14:49:03 -07:00
7a84c267f7
Merge pull request #2743 from metamx/fixlookuptest
...
Fix LookupCoordinatorManager and Test for alerts
2016-03-28 14:41:37 -07:00
89a8277ae2
Merge pull request #2712 from guobingkun/make_runnables_pluggable
...
make Coordinator IndexingService helpers pluggable
2016-03-28 12:18:18 -05:00
05151bc325
Fix LookupCoordinatorManagerTest for alerts
...
* Also fixes bad alerting on missing nodes
2016-03-28 09:41:47 -07:00
3c4691aa5a
Merge pull request #2741 from gianm/examples-wiki
...
Downgrade geoip2, exclude com.google.http-client.
2016-03-25 23:08:38 -07:00
01f3221a62
Merge pull request #2665 from jisookim0513/remove-druid-server-serialization
...
remove serialization of DruidServer
2016-03-25 15:54:05 -07:00
977e867ad8
Downgrade geoip2, exclude com.google.http-client.
...
Reverts "Update com.maxmind.geoip2 to 2.6.0" and exclude the google http client
from com.maxmind.geoip2. This should satisfy the original need from #2646 (wanting
to run Druid along with an upgraded com.google.http-client) while preventing
Jackson conflicts pointed out in #2717 .
Fixes #2717 .
This reverts commit 21b7572533
2016-03-25 14:43:22 -07:00
0d3c5a3b6c
remove serialization of Druid Server and add tests for ServersResource
2016-03-25 12:27:27 -07:00
0872448ff0
make Coordinator IndexingService helpers pluggable
...
Fixes #2682
IndexingService helpers are added according to the settings in runtime.properties.
Rather than having all the config.isXXX checks there, it makes sense to have a pluggable
approach for allowing the dynamic configuration to bring in implementations for helpers
without having to have hard-coded sets of available helpers. Plus, it will also make it possible for extensions to plug helpers in.
With https://github.com/druid-io/druid-api/pull/76 , we could conditionally bind a helper to Coordinator's runlist.
The condition is driven by the value set in the runtime.properties.
2016-03-25 11:48:54 -05:00
e78a469fb7
UTs for ExtensionsConfig
2016-03-25 10:51:28 -05:00
004b00bb96
config to explicitly specify classpath for hadoop container during hadoop ingestion
2016-03-25 10:51:28 -05:00
713062053c
Filters: Add filter.toFilter method, use that instead of the instanceof chain in Filters.
...
I believe that the instanceof chain in Filters exists because in the past, Filter
and DimFilter were in different packages (DimFilter was in druid-client and Filter
was in druid-processing). And since druid-client didn't depend on druid-processing,
DimFilter couldn't have a toFilter method. But now it can.
2016-03-23 17:03:49 -07:00
f25b2ed6f8
Duplicate statement in ReservoirSegmentSamplerTest.java
2016-03-22 22:14:36 -04:00
826b371259
Merge pull request #2697 from guobingkun/remove_duplicate_version_converter
...
remove duplicated DruidCoordinatorVersionConverter
2016-03-22 15:48:09 -07:00
a6e9ff48ec
Merge pull request #2688 from pjain1/props_cli
...
do not inject properties directly in module
2016-03-22 15:27:19 -05:00
3778adf1f4
remove duplicated DruidCoordinatorVersionConverter
2016-03-22 14:45:52 -05:00
7b93195dc6
do not inject properties directly in module
2016-03-22 14:30:10 -05:00
00d7021291
Merge pull request #2607 from jon-wei/dim_schema
...
Support use of DimensionSchema class in DimensionsSpec
2016-03-22 11:53:46 -05:00
3220b109ad
Merge pull request #2570 from binlijin/single_dimension_partitioning
...
Single dimension hash-based partitioning
2016-03-22 11:51:06 -05:00
bce600f5d5
Single dimension hash-based partitioning
2016-03-22 13:15:33 +08:00
a59c9ee1b1
Support use of DimensionSchema class in DimensionsSpec
2016-03-21 13:12:04 -07:00
25967d0ed8
fix servlet startup sequence, fixes #2681
2016-03-18 15:06:15 -07:00
5da9a280b6
Query Time Lookup - Dynamic Configuration
2016-03-18 09:45:05 -07:00
45c413af7e
Merge pull request #2674 from metamx/fix-broadcast-lockup
...
separate HTTP client pool for cancellation requests
2016-03-17 15:23:42 -07:00
1718a7224b
separate HTTP pool for cancellation requests
...
* reduces contention between queries and cancellation requests
* more aggressive timeouts for cancellation requests
2016-03-17 12:11:18 -07:00
c716af5b04
Merge pull request #2678 from metamx/fixImports
...
Fix some google related imports
2016-03-17 11:53:16 -07:00
a52c6d3bee
Fix some google related imports
2016-03-17 11:03:29 -07:00
738dcd8cd9
Update version to 0.9.1-SNAPSHOT.
...
Fixes #2462
2016-03-17 10:34:20 -07:00
948b19a088
do not silently ingnore rows
2016-03-16 09:30:19 -05:00
ec949d76e3
Merge pull request #2655 from navis/hint-coordinator-client
...
Add hint message for missing `druid.selectors.coordinator.serviceName`
2016-03-14 20:57:40 -07:00
5ec5ac92c6
Merge pull request #2382 from himanshug/broker_segment_tier_selection
...
at broker, if configured, only add segments from specific tiers to the timeline
2016-03-14 16:53:06 -07:00
83e1d5d7bf
Add hint message for missing `druid.selectors.coordinator.serviceName`
2016-03-15 08:39:07 +09:00
06813b510a
Merge pull request #2571 from himanshug/gp_by_avoid_sort
...
avoid sort while doing groupBy merging when possible
2016-03-14 14:46:51 -07:00
773d6fe86c
Merge pull request #2646 from atomx/update-maxmind
...
Update com.maxmind.geoip2 to 2.6.0
2016-03-14 11:20:48 -07:00
d51a0a0cf4
Merge pull request #2220 from gianm/appenderator-kafka
...
Appenderators, DataSource metadata, KafkaIndexTask
2016-03-14 13:14:36 -05:00
2861e854f0
Merge pull request #2540 from pjain1/remove_kill
...
Remove extra parameter from deleteDataSourceSpecificInterval endpoint and correct exception message for invalid interval
2016-03-14 11:16:23 -05:00
21b7572533
Update com.maxmind.geoip2 to 2.6.0
...
com.maxmind.geoip2 2.6.0 depends on com.google.http-client 1.15.0-rc (3 years old).
When trying to include other libraries in Druid that require an up to date version of com.google.http-client this causes a problem.
2016-03-12 09:44:00 +00:00
187569e702
DataSource metadata.
...
Geared towards supporting transactional inserts of new segments. This involves an
interface "DataSourceMetadata" that allows combining of partially specified metadata
(useful for partitioned ingestion).
DataSource metadata is stored in a new "dataSource" table.
2016-03-10 17:41:50 -08:00
3d2214377d
Appenderatoring.
...
Appenderators are a way of getting more control over the ingestion process
than a Plumber allows. The idea is that existing Plumbers could be implemented
using Appenderators, but you could also implement things that Plumbers can't do.
FiniteAppenderatorDrivers help simplify indexing a finite stream of data.
Also:
- Sink: Ability to consider itself "finished" vs "still writable".
- Sink: Ability to return the number of rows contained within the sink.
2016-03-10 17:41:50 -08:00
92c828f904
Make SegmentHandoffNotifier Closeable.
2016-03-10 16:50:37 -08:00
ad5ffdf483
Nix Committers.supplierOf; Suppliers.ofInstance is good enough.
2016-03-10 16:50:37 -08:00
8a11161b20
Plumbers: Move plumber.add out of try/catch for ParseException.
...
The incremental indexes handle that now so it's not necessary.
Also, add debug logging and more detailed exceptions to the incremental
indexes for the case where there are parse exceptions during aggregation.
2016-03-10 16:39:26 -08:00
02dfd5cd80
update IncrementalIndex to support unsorted facts map that can be used in groupBy merging to improve performance
2016-03-10 16:11:48 -06:00
708bc674fa
Make specifying query context booleans more consistent.
...
Before, some needed to be strings and some needed to be real booleans. Now
they can all be either one.
2016-03-08 19:38:26 -08:00
9c2420a1bc
Merge pull request #2599 from himanshug/datasource_isolation
...
make coordinator db polling for list of segments more robust
2016-03-08 12:43:49 -08:00
e7018f524f
Merge pull request #2598 from himanshug/handoff_timeout
...
optional ability to configure handoff wait timeout on realtime tasks
2016-03-08 12:43:36 -08:00
c72438ead0
override metric name
2016-03-08 10:58:12 -06:00
1288784bde
in coordinator db polling for available segments, ignore corrupted entries in segments table so that coordinator continues to load new segments even if there are few corrupted segment entries
2016-03-07 15:13:10 -06:00
0402636598
configurable handoffConditionTimeout in realtime tasks for segment handoff wait
2016-03-05 10:14:54 -06:00
d06c1c5c85
Merge pull request #2583 from guobingkun/fix_multiple_specs_2
...
update querySegmentSpec when passing query to getQueryRunner
2016-03-02 18:05:34 -08:00
9e74772d6b
Merge pull request #2574 from gianm/allostuff
...
Make first few allocatePendingSegment retries quiet.
2016-03-02 16:16:53 -07:00
cfe2dbf1eb
Merge pull request #2580 from gianm/rtc-basePersist
...
RealtimeTuningConfig: Use different default basePersistDirectory per instance.
2016-03-02 16:56:49 -06:00
4a58462fc7
update querySegmentSpec when passing query to getQueryRunner
...
After finding the FireChief for a specific partition, Druid will need to find the specific queryRunner for each segment being queried by passing the query to FireChief. Currently Druid is passing the original query that contains all the segments need to be queried, it's possible that fireChief.getQueryRunner(query) returns more than 1 queryRunner because query.getIntervals() is not specific to a single segment.
In this patch, for each segment being queried, Druid will update the query with its corresponding SpecificSegmentSpec.
2016-03-02 16:44:56 -06:00
e65e6a49a5
RealtimeTuningConfig: Use different default basePersistDirectory per instance.
2016-03-02 13:57:53 -08:00
004028b887
Make first few allocatePendingSegment retries quiet.
...
Some light retrying can happen during normal operation (SELECT -> INSERT races) and the
ensuing log messages would be scary for users.
2016-03-02 13:40:29 -08:00
612e327426
Merge pull request #2581 from gianm/fix-deadlock
...
CliPeon: Fix deadlock on startup by eagerly creating ExecutorLifecycle, ChatHandlerResource.
2016-03-02 11:37:49 -08:00
7557eb2800
CliPeon: Fix deadlock on startup by eagerly creating ExecutorLifecycle, ChatHandlerResource.
...
See stack traces here, from current master: https://gist.github.com/gianm/bd9a66c826995f97fc8f
1. The thread "qtp925672150-62" holds the lock on InternalInjectorCreator.class,
used by Scopes.SINGLETON, and wants the lock on "handlers" in Lifecycle.addMaybeStartHandler
called by DiscoveryModule.getServiceAnnouncer.
2. The main thread holds the lock on "handlers" in Lifecycle.addMaybeStartHandler, which it
took because it's trying to add the ExecutorLifecycle to the lifecycle. main is trying
to get the InternalInjectorCreator.class lock because it's running ExecutorLifecycle.start,
which does some Jackson deserialization, and Jackson needs that lock in order to inject
stuff into the Task it's deserializing.
This patch eagerly instantiates ChatHandlerResource (which I believe is what's trying to
create the ServiceAnnouncer in the qtp925672150-62 jetty thread) and the ExecutorLifecycle.
2016-03-02 10:53:42 -08:00
102fc92120
SQLMetadataConnector: Fix overzealous retries that were preventing EntryExistsException from making it out.
2016-03-01 17:20:33 -08:00
9340cae985
Merge pull request #2457 from bjozet/docs/fixes
...
Default value for maxRowsInMemory
2016-03-01 07:43:26 -08:00
2462c82c0e
New defaults for maxRowsInMemory rowFlushBoundary
...
To bring consistency to docs and source this commit changes the default
values for maxRowsInMemory and rowFlushBoundary to 75000 after
discussion in PR https://github.com/druid-io/druid/pull/2457 .
The previous default was 500000 and it's lower now on the grounds that
it's better for a default to be somewhat less efficient, and work,
than to reach for the stars and possibly result in
"OutOfMemoryError: java heap space" errors.
2016-03-01 13:50:28 +01:00
4edcb1b861
Refactor FireChief + UTs for RealtimeManagerTest
...
Add tests that verify whether RealtimeManager is querying the correct FireChief for a specific partition
make FireChief static and package private, add latches in the UT
2016-02-29 14:41:10 -06:00
68631d89e9
Allow realtime nodes to have multiple shards of the same datasource
2016-02-29 12:30:25 -06:00
6b3c96c63a
better exception for invalid interval
2016-02-26 10:02:38 -06:00
29d29ba98d
Merge pull request #2263 from jon-wei/flex_dims3
...
Allow IncrementalIndex to store Long/Float dimensions
2016-02-25 17:23:02 -08:00
b331fb4a83
Fix parsing of druid.indexer.server.maxChatRequests.
2016-02-25 14:47:15 -08:00
b82b487f20
remove extra kill parameter
2016-02-24 17:16:18 -06:00
c17ce02467
Allow IncrementalIndex to store Long/Float dimensions
2016-02-24 13:51:57 -08:00
a3b37e9225
In persistAndMerge, increase the scope of try-catch block so that any exception while persisting hydrants is caught and consequently that sink is abandoned or the task will forever wait for handoff to happen.
2016-02-23 22:22:33 -06:00
6c9e1a28ad
Merge pull request #2519 from gianm/unparseable-handling
...
Better handling of ParseExceptions.
2016-02-24 04:46:29 +05:30
93540c0631
Merge pull request #2503 from gianm/jetty-qos
...
Add druid.indexer.server.maxChatRequests for QoS; deprecate separate ports.
2016-02-23 10:35:53 -08:00
3534483433
Better handling of ParseExceptions.
...
Two changes:
- Allow IncrementalIndex to suppress ParseExceptions on "aggregate".
- Add "reportParseExceptions" option to realtime tuning configs. By default this is "false".
Behavior of the counters should now be:
- processed: Number of rows indexed, including rows where some fields could be parsed and some could not.
- thrownAway: Number of rows thrown away due to rejection policy.
- unparseable: Number of rows thrown away due to being completely unparseable (no fields salvageable at all).
If "reportParseExceptions" is true then "unparseable" will always be zero (because a parse error would
cause an exception to be thrown). In addition, "processed" will only include fully parseable rows
(because even partial parse failures will cause exceptions to be thrown).
Fixes #2510 .
2016-02-23 10:11:43 -08:00
0c984f9e32
Merge pull request #2109 from himanshug/segments_in_delta_ingestion
...
idempotent batch delta ingestion
2016-02-22 14:00:45 -08:00
3bdd757024
Merge pull request #1773 from b-slim/log_details
...
Adding downstream source when throwing QueryInterruptedException
2016-02-22 10:16:07 -08:00
21b0b8a07d
new coordinator endpoint to get list of used segment given a dataSource and list of intervals
2016-02-21 23:17:58 -06:00
77925cc061
adding downstream source of QueryInterruptedException
2016-02-20 13:05:14 -06:00
23c993c9e7
Add druid.indexer.server.maxChatRequests for QoS; deprecate separate ports.
...
- Add druid.indexer.server.maxChatRequests, which sets up a QoSFilter on the main Jetty server.
- Deprecate druid.indexer.runner.separateIngestionEndpoint
- Deprecate druid.indexer.server.chathandler.*
2016-02-19 13:36:09 -08:00
243ac5399b
Harmonize realtime indexing loop across the task and standalone nodes.
...
- Both now catch ParseExceptions on plumber.add (see https://groups.google.com/d/topic/druid-user/wmiRDvx2RvM/discussion )
- Standalone now treats IndexSizeExceededException as fatal (previously only the task did)
2016-02-19 07:34:15 -08:00
e0c049c0b0
Make startup properties logging optional.
...
Off by default, but enabled in the example config files. See also #2452 .
2016-02-12 14:12:16 -08:00
1430bc2c88
Merge pull request #2276 from harshjain2/feature-2021
...
Fix for issue 2021.
2016-02-10 17:04:45 -08:00
fa92b77f5a
Harmonize znode writing code in RTR and Worker.
...
- Throw most exceptions rather than suppressing them, which should help
detect problems. Continue suppressing exceptions that make sense to
suppress.
- Handle payload length checks consistently, and improve error message.
- Remove unused WorkerCuratorCoordinator.announceTaskAnnouncement method.
- Max znode length should be int, not long.
- Add tests.
2016-02-10 14:52:00 -08:00
a3eb863c8e
Fix for issue 2021
2016-02-10 22:19:12 +05:30
d1cb17d3f7
at broker - only add segments from specific tiers to the timeline
2016-02-09 22:33:22 -06:00
b40c342cd1
make Global stupid pool cache size configurable
2016-02-05 14:18:06 -06:00
9002548eeb
increase test time out and general clean up
2016-02-03 13:26:37 -06:00
5111fd52f2
Add check for log4j-core in Log4jShutterDownerModule
2016-02-02 15:56:48 -08:00
dc89cdd0f9
Merge pull request #2336 from himanshug/fix_2331
...
limit size of X-Druid-Response-Context header to 7K
2016-02-02 12:06:59 -06:00
c03918f89a
AsyncQueryForwardingServletTest#testDeleteBroadcast sometimes fails by port conflict
2016-01-29 19:28:58 +09:00
f6b4dbd697
bug fix and unit tests for DruidCoordinatorSegmentKiller
2016-01-28 14:10:17 -06:00
ab3edfa8fc
moving DruidCoordinatorSegmentKiller class out of DruidCoordinator
2016-01-28 14:03:56 -06:00
3880f54b87
Merge pull request #2332 from himanshug/configurable_partial
...
make populateUncoveredIntervals a configuration in query context
2016-01-28 10:34:35 +05:30
a7bde8f4da
limit size of X-Druid-Response-Context header to 7K due to https://github.com/druid-io/druid/issues/2331
2016-01-27 15:18:08 -06:00
5a3642bb93
Merge pull request #2247 from metamx/pedanticBuild
...
Enable strict building in travis
2016-01-27 10:27:03 -08:00
508734c8b0
Long constant reformatting in tests `l` --> `L`
2016-01-27 08:59:19 -08:00
fd6bf3fe22
Use interval comparator instead of bucketMonthComparator
...
fix when two segments have same interval
review comments
2016-01-27 17:35:43 +05:30
3719b6e3c8
make populateUncoveredIntervals a configuration in query context
2016-01-26 15:13:45 -06:00
41730b96d4
Fix for issue 2021
2016-01-25 02:48:22 +05:30
7a6109f0ca
Merge pull request #2321 from gianm/pending-index
...
Replace two-column index on pendingSegments table with one-column index.
2016-01-22 13:38:15 -06:00
0bd9bff075
Replace two-column index on pendingSegments table with one-column index.
...
Fixes #2319 .
2016-01-22 10:50:21 -08:00
1a5d4e714c
Adding custom mapper for json processing exception to return bad request instead of 500
2016-01-22 09:48:52 -08:00
04d3054353
Merge pull request #2303 from CHOIJAEHONG1/localfirehouse-basedir-npe
...
Throw an IAE when baseDir is null in LocalFireHose
2016-01-21 07:58:52 -08:00
dcb7830330
Merge pull request #984 from drcrallen/thread-priority-rebase
...
Use thread priorities. (aka set `nice` values for background-like tasks)
2016-01-21 15:02:34 +05:30
2e1d6aaf3d
Use thread priorities. (aka set `nice` values for background-like tasks)
...
* Defaults the thread priority to java.util.Thread.NORM_PRIORITY in io.druid.indexing.common.task.AbstractTask
* Each exec service has its own Task Factory which is assigned a priority for spawned task. Therefore each priority class has a unique exec service
* Added priority to tasks as taskPriority in the task context. <0 means low, 0 means take default, >0 means high. It is up to any particular implementation to determine how to handle these numbers
* Add options to ForkingTaskRunner
* Add "-XX:+UseThreadPriorities" default option
* Add "-XX:ThreadPriorityPolicy=42" default option
* AbstractTask - Removed unneded @JsonIgnore on priority
* Added priority to RealtimePlumber executors. All sub-executors (non query runners) get Thread.MIN_PRIORITY
* Add persistThreadPriority and mergeThreadPriority to realtime tuning config
2016-01-20 14:00:31 -08:00
61aca6f9cc
remove wrong checks
...
sink never have null hydrants and hydrants never have null adapters
2016-01-20 23:43:53 +05:30
7132428bba
throw IAE when baseDir is null in LocalFireHose
2016-01-21 01:27:32 +09:00
59ea186af7
fix reference counting for segments
2016-01-20 17:24:21 +05:30
747343e621
Preserve dimension order across indexes during ingestion
2016-01-19 13:34:11 -08:00
df2906a91c
Merge pull request #2290 from gianm/index-merger-v9-stuff
...
Respect buildV9Directly in PlumberSchools, so it works on standalone realtime.
2016-01-19 13:04:00 -08:00
0c31f007fc
Merge pull request #1728 from himanshug/aggregators_in_segment_metadata
...
Store AggregatorFactory[] in segment metadata
2016-01-19 12:55:49 -08:00
fe841fd961
Merge pull request #2118 from guobingkun/fix_segment_loading
...
Fix loading segment for historical
2016-01-19 14:25:48 -06:00
a99aef29a1
adding aggregators to segment metadata
2016-01-19 14:23:39 -06:00
1dcf22edb7
Respect buildV9Directly in PlumberSchools, so it works on standalone realtime nodes.
...
Also parameterize some tests to run with/without buildV9Directly:
- IndexGeneratorJobTest
- RealtimeIndexTaskTest
- RealtimePlumberSchoolTest
2016-01-19 12:15:06 -08:00
c4ad50f92c
Fix loading segment for historical
...
Historical will drop a segment that shouldn't be dropped in the following scenario:
Historical node tried to load segmentA, but failed with SegmentLoadingException,
then ZkCoordinator called removeSegment(segmentA, blah) to schedule a runnable that would drop segmentA by deleting its files. Now, before that runnable executed, another LOAD request was sent to this historical, this time historical actually succeeded on loading segmentA and announced it. But later on, the scheduled drop-of-segment runnable started executing and removed the segment files, while historical is still announcing segmentA.
2016-01-19 10:29:49 -06:00
164b0aad7a
removing Map<String,Object> segmentMetadata from methods in Index[Maker/Merger] and using Metadata class
...
instead of a Map to store segment metadata
2016-01-18 22:03:46 -06:00
637d2605e7
kill unwanted parent directories when a segment is deleted from LocalDataSegmentKiller
2016-01-18 21:51:04 -06:00
71f1cd5e34
Fix for issue 2021
2016-01-17 16:10:04 +05:30
f6a1a4ae20
Merge pull request #2138 from KurtYoung/feature-build-v9
...
build v9 directly
2016-01-16 13:35:46 -06:00
82ff98c2bf
add config for build v9 directly and update docs
2016-01-16 11:26:34 +08:00
2e54553a8f
Merge pull request #1990 from himanshug/schedule_kill_task
...
support periodic hard delete of segments
2016-01-15 15:22:33 -06:00
7c65880e55
Merge pull request #2270 from rasahner/warnOfChatHandlerNoop
...
if chathandler is noop, log using warn instead of info
2016-01-15 13:29:27 -07:00
6ec6835b5d
Fix for issue 2021.
2016-01-16 00:58:33 +05:30
34cd8f8c72
Merge pull request #2258 from fjy/acl-zk
...
acl for zookeeper is added
2016-01-15 10:27:08 -07:00
a54c726726
Merge pull request #2266 from anubhgup/fix-announce
...
Fix loss in segment announcements when segments do not fit in zNode
2016-01-14 17:10:43 -08:00
dc1a62c3d9
Merge pull request #2248 from metamx/druidNodeHashEquals
...
Add hashCode and equals to DruidNode
2016-01-14 16:04:58 -08:00
6d09ab839f
Fix for loss in segment announcements when segments do not fit in the znodes during compress mode.
...
Added unit test (from Navis).
2016-01-14 14:44:23 -08:00
7361cd173f
if chathandler is noop, log using warn instead of info
2016-01-14 10:12:42 -06:00
1908d63162
acl for zookeeper is added
2016-01-13 14:56:05 -08:00
18479bb757
time-descending result of timeseries queries
2016-01-13 12:23:01 +09:00
eb2d251ac8
support periodic hard delete of segments
2016-01-12 16:55:05 -06:00
01a0715ee2
Merge pull request #2161 from metamx/query-metrics-timeout
...
Fix Query metrics for query timeout
2016-01-12 09:50:40 -06:00
ef3cddabe9
adding UTs for DruidCoordinatorConfig
2016-01-11 22:03:25 -06:00
b0e04a9162
Add hashCode and equals to DruidNode
2016-01-11 15:23:45 -08:00
ea623e43d2
Merge pull request #2240 from metamx/fix-load-rule
...
Fix loadRule when one of the tiers had no available servers
2016-01-11 10:05:31 -08:00
32bc2f776e
Fix loadRule when one of the tiers had no servers
...
When one of the tiers have no servers, LoadRule should ignore that tier
and continue to load/drop segments in other available tiers.
the bug also causes whacky behavior with LoadRule with non existent
tier where the segment balancer keeps on moving segments to other nodes
in existing tiers but the extra segment copies are never dropped
eventually leading to all the tiers getting full .
2016-01-11 15:53:14 +05:30
d255f4baac
Merge pull request #2234 from pjain1/emit_realtime_metrics
...
emit handoff count metrics
2016-01-08 14:24:16 -06:00
9dba0f67e7
emit handoff count metrics
2016-01-08 12:36:13 -06:00
15fc070232
Merge pull request #2213 from himanshug/fix_curtator_test_base
...
[wip] trying/finding fix for announcer test failures
2016-01-07 18:23:49 -08:00
1bfb4e3988
Emit query/time for failed and timeout queries
...
emit query/time metric also
add success flag
fix success flag for router metrics
review comments
formatting.
2016-01-07 19:41:54 +05:30
7ab810f3eb
do not ignore exceptions from curator cleanup in CuratorTestBase
2016-01-06 10:42:53 -06:00
14989f272d
Add metrics for ingest/bytes/received for EventReceiverFirehose
...
review comments
review comments
2016-01-05 20:06:09 +05:30
57d91d754d
Comment out buggy unit tests, fix #2185
2016-01-03 09:50:16 -08:00
8424b2b456
Fix announcer test bad check
2016-01-01 19:55:22 -08:00
fa5c3bb014
adding decorate(DimensionSelector) to DimensionSpec to enable support for arbitrary filtering/transformations to returned dimension values
2015-12-30 15:06:24 -06:00
3c107c5757
Merge pull request #2150 from himanshug/emit_query_bytes
...
emit query/bytes metric
2015-12-30 13:44:19 -06:00
7ffa706655
Merge pull request #2152 from metamx/add-taskId
...
Add taskId to realtimeMetrics
2015-12-29 10:33:40 -08:00
b1261035a7
Merge pull request #1861 from guobingkun/insert_segment_tool
...
insert-segment tool
2015-12-29 10:06:07 -08:00
38b0f1fbc2
fix transient failures in unit tests
2015-12-28 20:03:30 -08:00
d94821998f
Merge pull request #2153 from navis/fix-testfail-DruidCoordinatorTest
...
Another try to fix DruidCoordinatorTest#testCoordinatorRun
2015-12-27 17:03:46 -08:00
10c19fe9f8
Another try to fix DruidCoordinatorTest#testCoordinatorRun
2015-12-24 03:02:37 +09:00
2c3c4a3f8f
Another try to fix xxServerViewTests
2015-12-24 02:13:40 +09:00
508629916d
Merge pull request #2137 from guobingkun/fix_serverview_transient_error2
...
[Don't merge]Another effort on fixing the transient error for XXXServerViewTest
2015-12-23 07:33:54 -08:00
978a3fd8ae
Add taskId to realtimeMetrics
...
Add task Id to Realtime Metrics
2015-12-23 18:05:25 +05:30
1a8546a682
emit query/bytes metric
2015-12-23 00:29:44 -06:00
89e2a9a6d2
build at Tue Dec 22 21:30:00 CST 2015
2015-12-22 21:30:00 -06:00
a110eafff7
build at Tue Dec 22 17:30:00 CST 2015
2015-12-22 17:30:00 -06:00
4b93528334
build at Tue Dec 22 17:00:01 CST 2015
2015-12-22 17:00:01 -06:00
b8d9860dbc
build at Tue Dec 22 16:00:00 CST 2015
2015-12-22 16:00:00 -06:00
ffdb9407e5
build at Tue Dec 22 15:30:00 CST 2015
2015-12-22 15:30:00 -06:00
6f54c1b60d
build at Tue Dec 22 15:00:00 CST 2015
2015-12-22 15:00:01 -06:00
4e61261c68
build at Tue Dec 22 14:30:00 CST 2015
2015-12-22 14:30:00 -06:00
4961b8c0bd
build at Tue Dec 22 14:00:00 CST 2015
2015-12-22 14:00:00 -06:00
0a635a11d6
build at Tue Dec 22 13:40:00 CST 2015
2015-12-22 13:40:00 -06:00
05c9e1b598
Reorder Before/After in JDBCExtractionNamespaceTest
...
* Fixes https://github.com/druid-io/druid/issues/2120
2015-12-22 11:39:46 -08:00
849ab2d958
build at
2015-12-22 13:08:29 -06:00
455980d659
address another node exists failure
2015-12-22 13:06:20 -06:00
2ffeda5d25
build number 13
2015-12-22 10:19:39 -06:00
7e64d5179f
Merge pull request #2139 from himanshug/emit_query_node_bytes
...
emit query/node/bytes metric
2015-12-22 08:07:04 -08:00
bc573484f3
build number 12
2015-12-22 00:52:58 -06:00
7b5fd76058
Merge pull request #2119 from navis/fix-sporadic-testfail2
...
Fix sporadic fail of DruidCoordinatorTest#testCoordinatorRun
2015-12-21 23:28:32 -06:00
b96f560255
emit query/node/bytes metric
2015-12-21 23:23:20 -06:00
06c220ae7b
build number 11
2015-12-21 19:40:47 -06:00
93a91e5633
build number 10
2015-12-21 18:21:15 -06:00
471dd7e164
build number 9
2015-12-21 17:35:18 -06:00
627c311923
build #8
2015-12-21 17:06:37 -06:00
293f2ad65d
build #7
2015-12-21 16:09:19 -06:00
9ed843ea21
build #6
2015-12-21 15:41:23 -06:00
f8342ca240
build #5
2015-12-21 15:10:37 -06:00
52261c5539
build #4
2015-12-21 14:41:38 -06:00
47a8f2d3f1
build #3
2015-12-21 13:56:43 -06:00
5cc56cf9e7
build #2
2015-12-21 13:05:38 -06:00
f12cfa99fa
another effort on fixing the transient error for Broker/CoordinatorServerViewTest happend on Travis build
2015-12-21 12:31:36 -06:00
c2a2d19d62
Merge pull request #2126 from guobingkun/remove_single_announcer
...
Remove SingleDataSegmentAnnouncer in favor of BatchDataSegmentAnnouncer
2015-12-21 08:55:07 -08:00
7b24d1f355
Fix sporadic fail of DruidCoordinatorTest#testCoordinatorRun
2015-12-21 17:00:52 +09:00
951a4e9b35
Remove SingleDataSegmentAnnouncer in favor of BatchDataSegmentAnnouncer
2015-12-21 00:05:53 -06:00
89b477970f
DataSegmentFinder tool
...
`insert-segment-to-db` is a tool that can insert segments into Druid metadata storage. It is intended to be used
to update the segment table in metadata storage after people manually migrate segments from one place to another.
It can also be used to insert missing segment into Druid, or even recover metadata storage by telling it where the
segments are stored.
Note: This tool expects users to have Druid cluster running in a "safe" mode, where there are no active tasks to interfere
the segments being inserted. Users can optionally bring down the cluster to make 100% sure nothing is interfering.
2015-12-21 00:02:04 -06:00
ad3312171b
Fix sporadic fail of io.druid.client.CoordinatorServerViewTest
2015-12-21 11:39:32 +09:00
7019d3c421
Merge pull request #2107 from jon-wei/fix_smq
...
More efficient SegmentMetadataQuery
2015-12-18 16:40:47 -08:00
e52c9d2b3c
Merge pull request #2095 from guobingkun/fix_client_merge_task_and_serde
...
Fix getType() and Json serialization in ClientMergeQuery + serde tests
2015-12-18 11:40:13 -08:00
356b07c6c3
More efficient SegmentMetadataQuery
2015-12-17 12:46:23 -08:00
1e5aa2f3ac
fix getType() and Json serialization in ClientMergeQuery and add serde tests
2015-12-15 12:08:43 -06:00
e6c2db89da
Merge pull request #1921 from jon-wei/flat_json
...
Add docs and benchmark for JSON flattening parser
2015-12-14 20:53:54 -08:00
a2d0bea223
Merge pull request #2074 from guobingkun/add_comment
...
Add comment to BrokerServerViewTest
2015-12-09 16:19:27 -08:00
c53bf85d83
Add docs and benchmark for JSON flattening parser
2015-12-09 16:13:30 -08:00
31502a920c
Add comment to BrokerServerViewTest
...
Explain why we need to do explicit checks on existences of zNodes before we create them. Address issue: https://github.com/druid-io/druid/issues/1512
2015-12-09 14:01:07 -06:00
ce79d707dd
Merge pull request #2072 from guobingkun/fix_transient_error_brokerserverviewtest
...
An effort to fix transient error in BrokerServerViewTest
2015-12-09 11:12:35 -08:00
161c4ce288
An effort to fix transient error in BrokerServerViewTest
2015-12-09 10:50:13 -06:00
c4de26ef09
fix typo in InitializationTest.java
2015-12-09 00:11:03 -06:00
5bc6d61a25
Fix InitializationTest
2015-12-08 17:37:56 -08:00
dcd15735cd
Merge pull request #2015 from metamx/handoff-notifier-coordinator
...
Remove ServerView from RealtimeIndexTasks and use coordinator http endpoint for handoff information
2015-12-08 14:06:39 -08:00
a32906c7fd
Remove FilteredServerView
2015-12-09 01:54:12 +05:30
9491e8de3b
Remove ServerView from RealtimeIndexTasks and use coordinator http endpoint for handoffs
...
- fixes #1970
- extracted out segment handoff callbacks in SegmentHandoffNotifier
which is responsible for tracking segment handoffs and doing callbacks
when handoff is complete.
- Coordinator now maintains a view of segments in the cluster, this
will affect the jam heap requirements for the overlord for large
clusters.
realtime index task and nodes now use HTTP end points exposed by the
coordinator to get serverView
review comment
fix realtime node guide injection
review comments
make test not rely on scheduled exec
fix compilation
fix import
review comment
introduce immutableSegmentLoadInfo
fix son reading
remove unnecessary logging
2015-12-09 01:54:09 +05:30
56fd48c0e4
Merge pull request #2056 from guobingkun/fix_server_selector2
...
Bug fix: stale segment in ServerSelector
2015-12-07 16:57:33 -08:00
ba980dcadb
Merge pull request #2058 from cheddar/report-uncovered-segments
...
Add "uncoveredIntervals" to responseContext
2015-12-07 16:42:42 -08:00
a36185926e
Add "uncoveredIntervals" to responseContext
...
This change will cause the CachingClusteredClient to populate the "uncoveredIntervals"
key in the responseContext map. The value will be any intervals that were requested in
the query but are not actually covered by the segments underlying the data source.
For unit testing, CachingClisteredClientTest is testing the caching behavior of the
object and it is pretty hard to adjust it to only test this new behavior, so I created
a new, parallel "CachingClusteredClientFunctionalityTest" to simplify testing just
basic functionality.
2015-12-07 16:16:57 -08:00
970b9cfcaa
fix stale segment in ServerSelector
2015-12-07 14:04:34 -06:00
00416e26ff
Merge pull request #2012 from himanshug/add_comment
...
adding more comments to why drops do not happen
2015-12-07 11:50:06 -08:00
62ba9ade37
unifying license header in all java files
2015-12-05 22:16:23 -06:00
3781aa380f
Added an injection test for ZkCoordinator.
2015-12-05 11:56:22 +08:00
221fb95d07
add support for getting used segments for multiple interval in IndexerMetadataStorageCoordinator
2015-12-03 21:28:04 -06:00
1eb8211346
Add datasource and taskId to metrics emitted by peons
...
This PR adds the datasource and taskId to the jvm and sys metrics
emitted by the peons.
fix spelling
review comment
review comment
2015-12-01 23:20:59 +05:30
7c91219130
adding more comments to why drops do not happen when there are no segments in db
2015-11-25 16:26:51 -06:00
8e83d800d6
Merge pull request #1881 from gianm/restartable-tasks
...
Restorable indexing tasks
2015-11-23 21:14:37 -08:00
13af260102
Merge pull request #1957 from metamx/skip-incremental-segment
...
Ability to skip Incremental Index during query using query context
2015-11-23 17:32:57 -08:00
501dcb43fa
Some changes that make it possible to restart tasks on the same hardware.
...
This is done by killing and respawning the jvms rather than reconnecting to existing
jvms, for a couple reasons. One is that it lets you restore tasks after server reboots
too, and another is that it lets you upgrade all the software on a box at once by just
restarting everything.
The main changes are,
1) Add "canRestore" and "stopGracefully" methods to Tasks that say if a task can
stop gracefully, and actually do a graceful stop. RealtimeIndexTask is the only
one that currently implements this.
2) Add "stop" method to TaskRunners that attempts to do an orderly shutdown.
ThreadPoolTaskRunner- call stopGracefully on restorable tasks, wait for exit
ForkingTaskRunner- close output stream to restorable tasks, wait for exit
RemoteTaskRunner- do nothing special, we actually don't want to shutdown
3) Add "restore" method to TaskRunners that attempts to bootstrap tasks from last run.
Only ForkingTaskRunner does anything here. It maintains a "restore.json" file with
a list of restorable tasks.
4) Have the CliPeon's ExecutorLifecycle lock the task base directory to avoid a restored
task and a zombie old task from stomping on each other.
2015-11-23 11:22:08 -08:00
36569094ce
EventReceiverFirehose: Drain buffer when closed, until empty.
2015-11-20 19:12:59 -08:00
21c84b5ff7
Merge pull request #1896 from gianm/allocate-segment
...
SegmentAllocateAction (fixes #1515 )
2015-11-18 21:05:46 -08:00
60f649dab1
Ability to skip Incremental Index during query using query context
...
This PR adds adds the ability to skip incremental index when querying
results from realtime nodes. default behaviour is to include
incrementalIndex in queries.
review comment
2015-11-19 00:29:41 +05:30
8fcf2403e3
Merge pull request #1943 from metamx/realtime-caching
...
Enable caching on intermediate realtime persists
2015-11-17 15:06:43 -08:00
8fa34ee671
Merge pull request #1973 from metamx/cachemonitor-optional-injection
...
CacheMonitor - make cache injection optional
2015-11-17 13:02:06 -08:00
9118b010c0
Merge pull request #1976 from noddi/feature/count-parameter-rules-history
...
Add count parameter to rules history endpoint.
2015-11-17 14:41:50 -06:00
dbe201aeed
Merge pull request #1929 from pjain1/jetty_threads
...
separate ingestion and query thread pool
2015-11-17 12:14:25 -08:00
6c498b7d4a
separate ingestion and query thread pool
2015-11-17 13:42:41 -06:00
b0a444eafc
Add count parameter to rules history endpoint.
2015-11-17 20:14:12 +01:00
d7eb2f717e
enable query caching on intermediate realtime persists
2015-11-17 10:58:00 -08:00
71376efd57
make cache injection optional
...
allows enabling the CacheMonitor for all index tasks, even if some don't
bind any cache instance.
2015-11-16 22:40:01 -08:00
4f46d457f1
Merge pull request #1947 from noddi/feature/count-parameter-history-endpoints
...
Add count parameter to history endpoints
2015-11-12 10:23:44 -08:00
465cbcf9a7
Merge pull request #1956 from metamx/remove-unused-imports
...
Cleanup + remove unused imports
2015-11-11 17:36:47 -08:00
e4e5f0375b
SegmentAllocateAction ( fixes #1515 )
...
This is a feature meant to allow realtime tasks to work without being told upfront
what shardSpec they should use (so we can potentially publish a variable number
of segments per interval).
The idea is that there is a "pendingSegments" table in the metadata store that
tracks allocated segments. Each one has a segment id (the same segment id we know
and love) and is also part of a sequence.
The sequences are an idea from @cheddar that offers a way of doing replication.
If there are N tasks reading exactly the same data with exactly the same logic
(think Kafka tasks reading a fixed range of offsets) then you can place them
in the same sequence, and they will generate the same sequence of segments.
2015-11-11 16:54:35 -08:00
dc4ae59aaf
Merge pull request #1865 from noddi/bugfix/issue-560
...
Fix #560 - datasource segments summary shouldn't include replication.
2015-11-11 15:29:08 -08:00
6e5d2c6745
Add count parameter to history endpoints.
2015-11-11 23:03:57 +01:00
8e743b70c6
SegmentIdentifier, like a pre-DataSegment.
2015-11-11 12:25:43 -08:00
e26f7fca36
cleanup cache interface
2015-11-11 12:25:22 -08:00
fa6142e217
cleanup and remove unused imports
2015-11-11 12:25:21 -08:00
f2c271e500
Better error message when LocalDataSegmentPusher cannot create its directory.
2015-11-10 19:49:01 -08:00
cf779946ef
Merge pull request #1791 from guobingkun/event_receiver_firehose_monitor
...
EventReceiverFirehoseMonitor
2015-11-10 11:09:42 -08:00
a57cbfd2c3
Merge pull request #1387 from metamx/enableShutdownLogging
...
Add special handler to allow logger messages during shutdown
2015-11-09 17:20:09 -08:00
1df4baf489
Move Jackson Guice adapters into io.druid
...
* Removes access to protected methods in com.fasterxml
* Eliminates druid-common's use of foreign package com.fasterxml
2015-11-09 10:50:45 -08:00
fe6efb6367
Merge pull request #1932 from gianm/data-schema-exclude-metric-names
...
DataSchema: Exclude metric names from dimension list.
2015-11-07 15:22:33 -08:00
8b5de492cc
Fix #560 - datasource segments summary shouldn't include replication.
2015-11-07 21:43:47 +01:00
6b908a5061
DataSchema: Exclude metric names from dimension list.
...
Otherwise we could end up with a metric and a dimension with the same column name.
2015-11-07 12:25:36 -08:00
e9533db987
Merge pull request #1850 from metamx/friendlyBardCache
...
Allow setting upper limit on the number of cache segments a broker will try to fetch.
2015-11-06 10:25:49 -08:00
c896818241
Update curator to 2.9.1
...
Lots of bugfixes since 2.8.0
- https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314425&version=12333324
- https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12314425&version=12332392
2015-11-05 15:53:01 -08:00
4efc90dda3
fix zkcoordinator inject
2015-11-04 17:30:17 -08:00
8f231fd3e3
cleanup druid codebase
2015-11-04 13:59:53 -08:00
3b3d88bd03
Merge pull request #1857 from noddi/feature/intervals-route
...
Queryable intervals
2015-11-03 13:19:04 -08:00
8b67417ac8
make methods in Index[Merger,Maker,IO] non-static so that they can have
...
appropriate ObjectMapper injected instead of creating one statically
2015-11-02 23:24:26 -06:00
f90ddfdb89
Merge pull request #1745 from himanshug/numbered_to_elastic
...
making NumberedShardSpec elastic
2015-10-30 16:01:03 -07:00
c3b6fcce9d
Add EventReceiverFirehoseMonitor
...
add an EventReceiverFirehoseMonitor so that we can monitor how many
events have been queued in the EventReceiverFirehose and get a sense
about whether the firehose is under too much pressure.
2015-10-30 11:40:02 -05:00
6de51e022d
Coordinator - add intervals route.
2015-10-30 11:51:42 +01:00
149333d77d
forward cancellation request to all brokers, fixes #1802
2015-10-29 16:58:38 -07:00
dfce14ed17
Allow setting upper limit on the number of cache segments a broker will try to fetch.
2015-10-29 11:50:00 -07:00
59872bd0cd
Merge pull request #1809 from metamx/fifoPriorityExecutorService
...
Make PrioritizedExecutorService optionally FIFO
2015-10-27 15:19:32 -07:00
ea2267e08c
Merge pull request #1868 from gianm/fix-announcements
...
Historical and MiddleManager server announcements should not remove parents.
2015-10-27 14:50:05 -07:00
7df7370935
Merge pull request #1862 from metamx/indexingServiceMMGone
...
Add timeout to shutdown request to middle manager for indexing service
2015-10-27 14:38:01 -07:00
7a2ceef690
Add special handler to allow logger messages during shutdown
...
* Adds a special PropertyChecker interface which is ONLY for setting string properties at the very start of psvm
2015-10-27 14:33:36 -07:00
ecdafa87c5
Make PrioritizedExecutorService optionally FIFO
2015-10-27 14:16:22 -07:00
67d58f6d8d
Update CoordinatorRuleManager for http-client 1.0.4
2015-10-27 14:08:45 -07:00
4b92752deb
Historical and MiddleManager server announcements should not remove parents.
...
Removing parent paths causes watchers of the "announcements" path to get stuck
and stop seeing new updates.
2015-10-27 08:06:11 -07:00
5a082b2f5e
Merge pull request #1824 from metamx/UniformGranularitySpecHashEquals
...
Add hashCode and equals to UniformGranularitySpec
2015-10-26 09:34:01 -07:00
5f23703216
Merge pull request #1638 from guobingkun/remove_maven_client_code
...
Remove Maven client at runtime + Provide a way to load Druid extensions through local file system
2015-10-26 09:30:05 -07:00
72c408cf2d
Merge pull request #1770 from metamx/merge-time
...
Add segment merge time as a metric
2015-10-22 22:03:41 -07:00
853f95f64c
Merge pull request #1845 from metamx/moreVerboseRuleAuditLogging
...
Add more verbose logging to SQLMetadataRuleManager
2015-10-22 16:12:46 -05:00
7cecc55045
Add segment merge time as a metric
...
Add merge and persist cpu time
Fix typo
review comment
move cpu time measuring to VMUtils
review comments.
2015-10-22 12:28:03 +05:30
6ceab03fc4
Add debug logging to start of segment manager polling
2015-10-21 16:17:49 -07:00
1cad571354
Add more verbose logging to SQLMetadataRuleManager
2015-10-21 16:11:40 -07:00
4914925d65
New extension loading mechanism
...
1) Remove maven client from downloading extensions at runtime.
2) Provide a way to load Druid extensions and hadoop dependencies through file system.
3) Refactor pull-deps so that it can download extensions into extension directories.
4) Add documents on how to use this new extension loading mechanism.
5) Change the way how Druid tarball is generated. Now all the extensions + hadoop-client 2.3.0
are packaged within the Druid tarball.
2015-10-21 14:22:36 -05:00
e4ac78e43d
bump next snapshot to 0.9.0
2015-10-20 13:46:13 -07:00
0292db648d
Merge pull request #1833 from metamx/improve-log
...
Improve alert message in CachingClusteredClient - Add Datasource info.
2015-10-20 15:40:57 -05:00
bf214f23b2
Improve message for alert - Add Datasource info.
2015-10-16 19:43:21 +05:30
4c2c7a2c37
update version to 0.8.3
2015-10-14 21:40:55 -07:00
f432b8e3f9
Add hashCode and equals to UniformGranularitySpec
...
* Also add hashCode != 0 to AllGranularity and NoneGranularity
2015-10-13 16:42:21 -07:00
bf11723a52
Update usages of io.druid.client.selector.Server to build URL or URI directly instead of using String.format
2015-10-12 12:30:56 -07:00
e450877a78
Make ServerDiscoverySelector more IPv6 friendly
2015-10-08 10:03:50 -07:00
693e14a14d
Add unit tests for ipv6 in ServerDiscoverySelectorTest
2015-10-08 10:03:22 -07:00
020a706ac4
Merge pull request #1800 from metamx/hybrid-cache
...
Hybrid L1/L2 cache
2015-10-06 15:18:07 -07:00
b11c0859e8
hybrid l1/l2 cache to combine local and remote cache
2015-10-06 14:32:08 -07:00
b5737a233a
Merge pull request #1801 from guobingkun/clientInfoResource
...
Fix ClientInfoResource
2015-10-06 07:53:44 -07:00
2bc4aed1e5
Add CPUTimeMetricQueryRunner to ClientQuerySegmentWalker
2015-10-05 10:09:22 -07:00
57c8e56451
Fix ClientInfoResource
...
Fix ClientInfoResource so that it doesn't return empty list for overshadowed segments
2015-10-02 14:12:55 -05:00
166c4fcf46
Merge pull request #1795 from metamx/announcerTestTransactions
...
Try and make AnnouncerTest a bit more predictable
2015-10-02 13:04:33 -05:00
7d635a2ce2
Try and make AnnouncerTest a bit more predictable
2015-10-02 09:19:40 -07:00
5db4c5a089
move cache bindings into a dedicated module
2015-10-02 00:29:05 -04:00
b9740a727b
unit test for IngestSegmentFirehose
2015-10-01 16:06:04 -05:00
2d847ad654
Merge pull request #1730 from metamx/union-queries-fix
...
fix #1727 - Union bySegment queries fix
2015-09-29 12:23:25 -07:00
573aa96bd6
fix #1727 - Union bySegment queries fix
...
Fixes #1727 .
revert to doing merging for results for union queries on broker.
revert unrelated changes
Add test for union query runner
Add test
remove unused imports
fix imports
fix renamed file
fix test
update docs.
2015-09-29 23:32:36 +05:30
348172203f
OverlordRedirectInfo: Fix ability to detect that there is no leader.
2015-09-25 09:30:09 -07:00
906e99d631
RedirectFilter: User agents are more likely to preserve method on 307.
2015-09-25 08:54:23 -07:00
3aba401ee0
SQLMetadataConnector: Retry table creation, in case something goes wrong.
...
Also rejigger table creation methods to not take a DBI. It's already available
inside the connector, and everyone was just using that one anyway.
2015-09-24 21:39:36 -07:00
4308aa93e2
Add meter metrics for memcached
2015-09-22 12:50:54 -07:00
46ba60f10f
Enable logging for memcached in factory
2015-09-22 11:13:47 -07:00
1c86eac9e1
Merge pull request #1760 from metamx/memcached082Updates
...
Update memcached client for better concurrency in metrics.
2015-09-22 10:59:43 -07:00
35caa753aa
Merge pull request #1761 from gianm/local-path-for-hadoop
...
LocalDataSegmentPusher: Fix for Hadoop + relative paths.
2015-09-22 08:57:44 -07:00
4efbe649a3
LocalDataSegmentPusher: Fix for Hadoop + relative paths.
2015-09-22 07:44:12 -07:00
e6f07a832d
Update memcached client for better concurrency in metrics. Also fixes another injection problem
2015-09-22 07:40:24 -07:00
34a8fbcd68
Merge pull request #1755 from metamx/update-druid-api
...
update druid-api for timestamp parsing speedup
2015-09-21 19:43:13 -05:00
490b1aea2a
Merge pull request #1758 from metamx/fixMemcachedInjection
...
Fix memcached cache provider injection and add test
2015-09-21 15:36:29 -07:00
37eb83a1d0
Fix memcached cache provider injection and add test
2015-09-21 15:00:22 -07:00
8f489c1f66
fix type cast warnings
2015-09-21 09:57:29 -07:00
af86c0e6ea
update druid-api + java-util for timstamp parsing speedup
2015-09-21 09:57:29 -07:00
117a6754b5
improving msging when indexing service is not found
2015-09-18 23:07:44 -05:00
6605a60ef0
Merge pull request #1747 from jon-wei/realtime_corrupt_sink_fix
...
Account for potential gaps in hydrants in sink initialization, hydrant swapping (e.g. h0, h1, h4)
2015-09-18 15:22:50 -07:00
9f6bb03ef4
Account for potential gaps in hydrants in sink initialization, hydrant swapping (e.g. h0, h1, h4)
2015-09-18 14:53:37 -07:00
df4c8a3aa5
balance memcached cache across multiple connections
2015-09-18 09:52:16 -07:00
a7b1cacbbf
make NumberedShardSpec "elastic" and UTs
2015-09-17 08:38:48 -05:00
9705c5139b
Merge pull request #1732 from jon-wei/segmentmeta
...
Add support for a configurable default segment history period for segmentMetadata queries and GET /datasources/<datasourceName> lookups
2015-09-16 12:36:25 -07:00
74f4572bd4
Lazily deserialize "parser" to InputRowParser in DataSchema
...
so that user hadoop related InputRowParsers are created only when needed
this allows overlord to accept a HadoopIndexTask with a hadoopy InputRowParser
and not fail because hadoopy InputRowParser might need hadoop libraries
2015-09-16 10:58:13 -05:00
b464da438c
Merge pull request #1688 from metamx/moreMemcachedMetrics
...
More memcached metrics
2015-09-15 15:33:51 -07:00
193fb4fdfc
Add support for a configurable default segment history period for segmentMetadata queries and GET /datasources/<datasourceName> lookups
2015-09-14 19:41:42 -07:00
5ff92664f8
Merge pull request #1696 from metamx/cpuTimeReporting
...
Add CPU time to metrics for segment scanning.
2015-09-14 10:53:55 -05:00
ebdb612933
composing emitter module to use multiple emitters together
2015-09-09 16:45:50 -05:00
5813856819
More memcached metrics
2015-09-08 13:34:58 -07:00
fcf5cae81d
Add CPU time to metrics for segment scanning.
2015-09-08 13:34:19 -07:00
062a47fba4
Modify Plumbers in these ways,
...
1) Persist using Committer instead of Runnable. (Although the metadata object
is ignored in this patch)
2) Remove the getSink method.
3) Plumbers are now responsible for time-based and hydrant-full-based periodic
committing. (FireChief, RealtimeIndexTask, and IndexTask used to do this)
2015-09-03 11:13:06 -07:00
e79572b4bc
Merge pull request #1637 from metamx/peonPlaysNiceOnShutdown
...
Make jetty scheduler threads daemon thread
2015-09-03 00:03:20 +05:30
0096e6a0a0
Merge pull request #1658 from metamx/cleanupJDBCExtractionNamespaceTest
...
Hopefully add better timeouts and ordering to JDBCExtractionNamespaceTest
2015-09-02 23:49:49 +05:30
0835e12f2a
add endpoint to fetch rule history for all datasources.
...
review comments
Review comment
fix compilation
2015-08-31 14:05:52 +05:30
dfe3be0f6d
Merge pull request #1689 from metamx/lz4TranscoderUpdate
...
Update LZ4Transcoder to match Compressed strategy factory type.
2015-08-29 12:31:59 -07:00
413c0d1c9e
Update LZ4Transcoder to match Compressed strategy factory type.
2015-08-28 19:19:48 -07:00
940e1aa3eb
Replace funky imports with standard ones.
...
1) Lots of Guava imports were not coming from the actual Guava
2) junit.framework.Assert should be org.junit.Assert
2015-08-28 18:02:05 -07:00
ceaa49ec4f
Merge pull request #1681 from gianm/ingest-segment-overlapping
...
Fix overlapping segments in IngestSegmentFirehose, DatasourceInputFormat
2015-08-28 11:12:51 -05:00
e575ef2b20
Remove unused Indexer interface.
2015-08-28 08:40:47 -07:00
414a6fb477
Fix overlapping segments in IngestSegmentFirehose, DatasourceInputFormat.
...
Fixes #1678 . IngestSegmentFirehose (and its users) need to remember which
windows of which segments should actually be read, based on a timeline.
2015-08-28 07:32:41 -07:00
2e0dd1d792
adding UTs and addressing review comments to
...
firehoseV2 addition to Realtime[Manager|Plumber],
essential segment metadata persist support,
kafka-simple-consumer-firehose extension patch
2015-08-27 20:50:46 -05:00
2237a8cf0f
kafka 8 simple consumer firehose
2015-08-27 20:50:46 -05:00
10af233b37
Merge pull request #1622 from metamx/smile-eventreceiverfirehose
...
Support for JSON Smile format for EventReceiverFirehoseFactory
2015-08-27 10:42:25 -07:00
ac8e32b58e
Hopefully add better timeouts and ordering to JDBCExtractionNamespaceTest
2015-08-26 23:05:51 -07:00
5b67ec6cda
Merge pull request #1672 from metamx/derbyRule
...
Migrate TestDerbyConnector to a JUnit @Rule
2015-08-26 22:56:18 -07:00
e38cf54bc8
Migrate TestDerbyConnector to a JUnit @Rule
2015-08-26 21:47:40 -07:00
e9237611c5
Merge pull request #1668 from metamx/cachingClusteredClientCodeCleanup1
...
Code cleanup for CachingClusteredClientTest
2015-08-26 17:13:55 -07:00
c325b194c0
Merge pull request #1661 from pjain1/poll_rules
...
Poll rules immediately after change
2015-08-25 13:46:55 -07:00
85aeda834c
Code cleanup for CachingClusteredClientTest
2015-08-25 12:49:55 -07:00
2bf9a70bfa
Consolidate SQL retrying by moving logic into the connectors.
...
Also change boolean removeLock to void addLock in MetadataStorageActionHandler.
2015-08-25 12:42:29 -07:00
073c32f01b
Poll rules immediately after change
2015-08-24 16:08:25 -05:00
e3964cd9f6
Merge pull request #1653 from metamx/fix-success-flag
...
fix success flag emitted in router query metrics
2015-08-21 07:56:57 -07:00
6d85a62ed2
fix success flag emitted in router query metrics
2015-08-21 12:25:57 +05:30
e6b20db62d
Support for JSON Smile format for EventReceiverFirehoseFactory
...
Support for Smile format for EventReceiverFireshoseFactory.
would help in decreasing network load on middle managers ingesting high
data volumes.
review comments
2015-08-20 19:52:45 +05:30
6a0389297c
Merge pull request #1634 from metamx/IrcFirehoseFixes
...
Allow IrcFirehoseFactory to shutdown cleanly
2015-08-18 21:06:34 -07:00
3b2e41e42a
update for next release
2015-08-18 17:16:46 -07:00
13773865cf
Make jetty scheduler threads daemon thread
2015-08-18 13:08:19 -07:00
cd2c3776f8
Allow IrcFirehoseFactory to shutdown cleanly
2015-08-17 13:24:41 -07:00
4d4aa8bfc6
refactor IngestSegmentFirehoseFactory so that IngestSegmentFirehose becomes reusable
...
Conflicts:
indexing-service/src/main/java/io/druid/indexing/firehose/IngestSegmentFirehoseFactory.java
2015-08-14 14:44:22 -05:00
db19d2d547
Revert "Update to guice 4.0"
2015-08-14 09:26:07 -07:00
be89105621
Merge pull request #1602 from metamx/more-code-cleanup
...
Some perf Improvements in Broker
2015-08-11 13:51:49 -07:00
7e61216287
Update to guice 4.0
...
- Mark a lot of `@Provides` methods as final since guice 4.0 disallows overriding them
2015-08-10 13:57:18 -07:00
b33c644da0
avoid unnecessary call to MultipleSpecificSegmentSpec.getIntervals()
...
avoid unnecessary call to MultipleSpecificSegmentSpec.getIntervals()
profiling shows it took upto 6-12% of cpu time in
JodaUtils.condenseIntervals
2015-08-06 13:51:17 +05:30
9dd18de1a5
Merge pull request #1596 from metamx/console-artifact
...
make console a maven dependency instead of filedump
2015-08-04 15:28:45 -07:00
07ee1acca2
make console a maven dependency instead of filedump
2015-08-04 09:49:34 -07:00
b05be0a104
Merge pull request #1593 from metamx/run-tests-offline
...
Fix test so that it can run offline
2015-08-04 09:23:40 -07:00
f490547faf
Fix test so that it can run offline too
...
Test was getting stuck for me when I tried to run it offline while
resolving the host. creating unresolved SocketAddress works around
this.
2015-08-03 23:37:04 +05:30
479b09c2dd
fix transient error in BrokerServerViewTest
2015-07-31 13:33:55 -05:00
22567946cf
Merge pull request #1259 from metamx/queryTimeLookup
...
Query Time Lookup
2015-07-28 11:43:05 -10:00
ba59f8afc4
Merge pull request #1550 from himanshug/optionally_log_all_requests
...
print *all* HTTP requests to log if configured
2015-07-28 11:19:36 -07:00
86ede702b1
Add namespaced lookups as extensions
...
* Adds kafka, URI, and JDBC namespace defintions
* Add ability to explicitly rename using a "namespace" which is a particular data collection that is loaded on all realtime, historic nodes, and brokers. If any of these nodes has the namespace extension, ALL nodes have the namespace extension.
* Add namespace caching and populating (can be on heap or off heap)
* Add NamespaceExtractionCacheManager for handling caches
* Added ExtractionNamespace for handling metadata on the extraction namespaces
* Added ExtractionNamespaceUpdate for handling metadata related to updates
* Add extension which caches renames from a kafka stream (requires kafka8)
* Added README.md for the namespace kafka extension
* Added docs
* Added namespace/size, namespace/count, namespace/deltaTasksStarted metrics
Add static config for namespaces via `druid.query.extraction.namespace`
* This is a rebase of https://github.com/b-slim/druid/tree/static_config_only
2015-07-28 11:14:14 -07:00
90b475975e
Log all the requests to debug level in logger io.druid.jetty.RequestLog
2015-07-28 12:56:04 -05:00
efb6e0649e
simplify merging of results on broker.
...
add benchmark
formatting and review comments
organize imports
2015-07-27 11:46:11 -07:00
6e8da5de86
Merge pull request #1441 from metamx/queryResourceFlusher
...
Add flush prior to close on output stream in QueryResource
2015-07-17 16:32:52 -07:00
e21195f987
Merge pull request #1469 from guobingkun/table_config
...
Inconsistent property names for "druid.metadata.storage.tables.xxx"
2015-07-17 07:43:19 -07:00
456ad9ffba
Merge pull request #1529 from metamx/update-versions
...
inrement version
2015-07-15 13:25:31 -07:00
4cfb00bc8a
inrement version
2015-07-15 13:09:05 -07:00
a4cb0332fb
Merge pull request #1524 from metamx/peonShutdown
...
Make CliPeon run shutdown hooks properly
2015-07-15 10:09:33 -07:00
92d5c250ab
Remove comment
2015-07-14 16:20:36 -07:00
5eadd395e2
Move lots of executor service creation to Execs
2015-07-14 15:38:49 -07:00
2b87ad307d
Make CliPeon run shutdown hooks properly
...
* Also make jetty threads daemon
2015-07-14 15:15:26 -07:00
da8e86146e
Merge pull request #1505 from metamx/improve-test
...
Modify test to check for multiple shards for same interval
2015-07-14 08:50:47 -07:00
3f7ba58227
Merge pull request #1504 from metamx/fix-1447
...
fix for #1447
2015-07-14 08:50:08 -07:00
1affdf7bec
Merge pull request #1502 from samjhecht/poll-alerting
...
add alert on errors polling for rules
2015-07-13 18:37:37 -05:00
fc438e5a72
add alert on errors polling for rules
...
reset retrystarttime after alerting
2015-07-13 09:03:51 -07:00
725086cc89
Merge pull request #1506 from gianm/realtime-plumber-nulls
...
Consider null inputRows and parse errors as unparseable during realtime ingestion.
2015-07-13 10:12:12 -05:00
9068bcd062
Consider null inputRows and parse errors as unparseable during realtime ingestion.
...
Also, harmonize exception handling between the RealtimeIndexTask and the RealtimeManager.
Conditions other than null inputRows and parse errors bubble up in both.
2015-07-11 20:40:03 -07:00
e9b627bf55
Merge pull request #1476 from pjain1/improved_tests
...
all the review comments have been taken care of, so merging.
2015-07-09 16:25:30 -05:00
1766341648
Modify test to check for multiple shards for same interval
...
Modify test to check for multiple shards for same interval.
2015-07-09 20:35:31 +05:30
58cc3954ff
improved coordinator and datasourceresource tests
2015-07-09 09:46:54 -05:00
5fe27fe4ad
fix for #1447
...
fixes #1447
2015-07-09 19:05:48 +05:30
184b12bee8
fix groupBy caching to work with renamed aggregators
...
Issue - while storing results in cache we store the event map which
contains aggregator names mapped to values. Now when someone fire same
query after renaming aggs, the cache key will be same but the event
will contain metric values mapped to older names which leads to wrong
results.
Fix - modify cache to not store raw event but the actual list of values
only.
review comments + fix dimension renaming
review comment
2015-07-09 11:48:26 +05:30
fe36f7f852
emit alert for DB polling error
2015-07-07 19:51:36 +05:30
d846f3ced6
Merge pull request #1498 from rasahner/doc_localFirehose
...
say explicitly that local firehose searches dirs recursively for files
2015-07-05 15:34:36 -07:00
acd20e8c00
say explicitly that local firehose searches directories recursively for files
2015-07-05 14:46:44 -05:00
f2919b98b1
Merge pull request #1471 from metamx/direct-client-metrics
...
add query/node/time metrics to DirectDruidClient
2015-06-30 22:06:44 -07:00
4a0ae7d8d5
Fix inconsistent druid property names for "druid.metadata.storage.tables.xxx" between document and code
2015-06-29 10:12:30 -05:00
8172bdff1c
Merge pull request #1468 from guobingkun/fix_npe_for_storage_action_handler
...
Fix npe thrown from SQLMetadataStorageActionHandler
2015-06-29 09:57:51 -05:00
28fa1642b9
add node time metrics to DirectDruidClient
2015-06-26 17:57:44 -07:00
fbcac10e00
Remove metrics emitting from caching clustered client
2015-06-26 10:49:13 -07:00
e4e18b17b5
fix npe thrown from getEntry() and getStatus() SQLMetadataStorageActionHandler due to a non-exist entryId
2015-06-26 10:40:56 -05:00
0c85c8c60a
Merge pull request #1462 from pjain1/validate_rules
...
tieredReplicants can be null
2015-06-24 13:29:27 -07:00
ff738ace05
converting null tieredeplicants to default and not allowing empty ones
2015-06-24 14:55:12 -05:00
0c13fd8018
Merge pull request #1450 from metamx/cache-key-distribution
...
Fix bad distribution of cache keys across cache nodes
2015-06-22 17:58:15 -07:00
34c8f4c3da
Merge pull request #1437 from metamx/limitUsedSegmentInterval
...
Add start and end bounds to getUsedSegmentsForInterval
2015-06-22 14:11:38 -05:00
8d91c8d399
Add start and end bounds to getUsedSegmentsForInterval
...
* Add unit tests
* Add javadoc for methods in IndexerMetadataStorageCoordinator
2015-06-22 11:46:53 -07:00
f1951b253c
Fix bad distribution of cache keys across nodes
...
With the existing hash function some nodes could end up with 3 times the
number of keys as others. The following changes improve that to roughly
less than 5% differences across nodes.
- switch from fnv-1a to murmur3_128 hash
- increase repetitions for ketama algorithm
- test to analyze distribution
Also updates spymemcached for recent bugfixes
2015-06-19 15:30:35 -07:00
0a5bb909a2
[maven-release-plugin] prepare for next development iteration
2015-06-18 17:35:19 -07:00
59c6b2b279
[maven-release-plugin] prepare release druid-0.8.0-rc1
2015-06-18 17:35:14 -07:00
d276d2c90b
Merge pull request #1444 from druid-io/logging-improvement
...
Separate bootstrap threads from loading threads on historical startup
2015-06-17 15:24:07 -07:00
06c97b6d7d
Separate out the "bootstrapping" threads from the "loading" threads.
...
It is highly recommended to keep loading threads single threaded. There can be benefits to having multiple bootstrapping threads, though. Those bootstrapping threads shouldn't be kept alive for the whole life of the process, however.
Also, adjust some logging and do some code cleanup.
2015-06-17 14:58:28 -07:00
0539e0073d
Merge pull request #1442 from guobingkun/broker_server_view_test
...
Unit test for BrokerServerView
2015-06-16 19:45:20 -07:00
1aad7ce521
Add a bit more information to the mapping logging logic.
2015-06-16 19:33:38 -07:00
446d01cc53
Unit test for BrokerServerView
2015-06-16 15:20:32 -05:00
69bbe21512
Add flush prior to close on output stream in QueryResource
2015-06-16 11:40:56 -07:00
9cf8662aeb
Merge pull request #1427 from guobingkun/fix_issue_1016
...
Fix duplicate extension loading issue described in #1016
2015-06-10 14:42:24 -07:00
bdf4d541e3
Fix issue #1016 that if user specifies extension coordinates, the local extensions will be loaded twice,
...
which could cause Guice duplicate binding errors.
Add unit test to replicate duplicate extension issue. Update documents. Add an package accessible getter for loadersMap for testing only.
Notice that extensions explicitly specified in druid.extensions.coordinates have a higher priority than ones included in the classpath.
Extension modules that don't have a canonical class name will be ignored.
2015-06-10 15:48:03 -05:00
e5937482bc
Merge pull request #1431 from pjain1/coordinator_ut
...
Druid Coordinator test
2015-06-10 07:24:39 -07:00
b72e6fd51a
Druid Coordinator Unit test
2015-06-09 13:51:01 -05:00
22607d19be
Make AbstractHttpClientProvider contents lazily instantiated
2015-06-08 09:53:26 -05:00
1646f650b6
Unit tests for LoadQueuePeon and SegmentLoaderLocalCacheManager, JavaDoc for LoadPeonCallback
2015-06-04 17:28:22 -05:00
f81a0d9210
Merge pull request #1425 from guobingkun/guice_fix
...
Anyways, @guobingkun , if possible, can you please test proposed change and send another PR to fix #1016 . but that doesn't necessarily hold this PR, so I'm merging this.
2015-06-04 14:43:26 -05:00
f44f7f07de
Merge pull request #1424 from pjain1/validate_rules
...
Check lower bound of replicant values for rules
2015-06-04 14:16:50 -05:00
1030899cdf
Check lower bound of replicant values for rules
2015-06-04 13:47:43 -05:00
dc26350480
Separate overwrite of Guice modules into 2 steps so that extension modules can overwrite cli modules
2015-06-04 09:57:56 -05:00
21864a9407
reduce test runtime
...
fix check
fix test
2015-06-04 08:35:12 +05:30
1c4d42bc15
Replace "timeout" with QueryContextKeys.TIMEOUT
2015-06-02 14:49:21 -07:00
185c99deec
Add schema check for overlap between dimension and metric names
2015-05-28 22:18:08 -07:00
be2a35188e
Additional schema validations and better logs for common extensions
2015-05-27 16:25:02 -07:00
051c3ccede
Add futures to SQLMetadataSegmentManager and SQLMetadataRuleManager
2015-05-15 16:08:22 -07:00
3c3db7229c
Merge pull request #1355 from himanshug/long_max_min_aggregators
...
Long max/min aggregators
2015-05-13 12:08:11 -07:00
d0ec945129
adding aliases doubleMax and doubleMin for max and min respectively
...
renamed all [Max/Min]*.java to [DoubleMax/DoubleMin]*.java and created [Max/Min]AggregatorFactory.java which can be removed when we dont need the min/max aggregator type backward compatibility
2015-05-13 09:25:41 -05:00
7a6acf5c1b
update pom to 0.8
2015-05-11 19:41:58 -06:00
a2dc58cd2d
Merge pull request #1345 from pjain1/unit_test_warn_fix
...
fix warn msg and some unit tests
2015-05-08 08:06:20 -07:00
01448d264c
Fix warn msg and added some unit tests
2015-05-07 17:10:05 -05:00
7608bf5799
Merge pull request #1342 from guobingkun/unit_test
...
Fix a potential issue in DeterminePartitionsJob, and two unit tests, reformat code
2015-05-05 09:24:23 -07:00
1ee550dd91
Fix a potential issue in DeterminePartitionsJob by making HadoopDruidIndexerConfig non-static, and two unit tests for DeterminPartitionsJob and LocalDataSegmentKiller
2015-05-04 20:00:29 -07:00
e69d82a2b4
Realtime: Delay firehose connection until job is started.
...
Some firehoses (like the Kafka firehose) acquire input resources when they
connect, so it helps to delay this until after plumber.startJob() runs.
2015-05-04 10:54:07 -07:00
8eb441ec08
RealtimeManager: Include partitionNum in chief thread name.
2015-05-04 10:53:48 -07:00
27e25fe513
fix IRCFirehoseFactory json name/nick mismatch
...
* Now it is nick as per wikipedia example
2015-05-01 12:35:44 -07:00
15ed308a4b
Merge pull request #1322 from himanshug/fix_audit_tbl_idx
...
fixing the typo for audit table (type, created_date) index creation
2015-04-30 13:22:04 -07:00
a7f9bb1a1e
Additional coordinator logs to figure out when racey things might be happening
2015-04-30 08:26:52 -07:00
4aa9381f6a
fixing the typo for audit table (type, created_date) index creation
2015-04-29 17:09:35 -05:00
6e84caf3d5
Proxy Requests from coordinator to overlord
...
Add unit test and support https scheme
review comments
2015-04-29 20:58:26 +05:30
062a413ccd
Fix DruidNodeTest for ipv6 only hosts
2015-04-28 13:11:47 -07:00
721505c017
Merge pull request #1208 from druid-io/rework-metrics
...
Schemaless metrics + additional metrics for things we care about
2015-04-27 15:04:54 -07:00
963e5765bf
Schemaless metrics + additional metrics for things we care about
2015-04-27 13:39:40 -07:00
958b34e0c2
fix exception during timeboundary
...
Fix Exception during time boundary query where results can be
represented as integers
2015-04-23 20:32:35 +05:30
1addbe163a
Merge pull request #1295 from anubhgup/master
...
Fix type of values in sql update.
2015-04-17 20:26:17 -07:00
4c240da4e9
Fix type of values in sql update.
2015-04-17 18:44:49 -07:00
d260515a43
update druid-api version
2015-04-17 14:58:35 -07:00
ea5572d001
Merge pull request #1271 from metamx/strictErrorChecking
...
Add stricter checking for potential coding errors
2015-04-15 15:21:41 -07:00
abdeaa0746
Add stricter checking for potential coding errors
...
Can use via `mvn clean compile test-compile -P strict'
2015-04-15 14:52:25 -07:00
3a3046ccf3
add support for dimension compression
...
- compression for single-value dimensions using CompressedVSizeIntsIndexedSupplier
- makes dimension compression configurable via IndexSpec
- IndexSpec also enables configuring bitmap and metric compression
2015-04-14 10:44:18 -07:00
44e23a8267
Merge pull request #1224 from metamx/alert-improve
...
skip raising false alert when the coordinator looses leadership
2015-04-07 10:11:34 -07:00
208e307915
Merge pull request #1251 from metamx/uriSegmentLoaders
...
Revert "Revert "Overhaul of SegmentPullers to add consistency and retries""
2015-03-30 17:43:51 -07:00
aea7f9d192
[maven-release-plugin] prepare for next development iteration
2015-03-30 16:35:24 -07:00
060d7aef03
[maven-release-plugin] prepare release druid-0.7.1
2015-03-30 16:35:20 -07:00
1c6cbea89c
Revert "Revert "Overhaul of SegmentPullers to add consistency and retries""
...
This reverts commit f904bc7858
2015-03-30 13:40:04 -07:00
f904bc7858
Revert "Overhaul of SegmentPullers to add consistency and retries"
2015-03-30 13:15:50 -07:00
6d407e8677
Add URI handling to SegmentPullers
...
* Requires https://github.com/druid-io/druid-api/pull/37
* Requires https://github.com/metamx/java-util/pull/22
* Moves the puller logic to use a more standard workflow going through java-util helpers instead of re-writing the handlers for each impl
* General workflow goes like this: 1) LoadSpec makes sure the correct Puller is called with the correct parameters. 2) The Puller sets up general information like how to make an InputStream, how to find a file name (for .gz files for example), and when to retry. 3) CompressionUtils does most of the heavy lifting when it can
2015-03-30 12:33:23 -07:00
e5653f0752
Merge pull request #1190 from vigiglobe/master
...
Fix NPE when partionNumber 0 does not exist.
2015-03-26 13:25:39 -07:00
2c58e355fd
Merge pull request #1239 from metamx/log4j2forRealz
...
Exclude log4j from curator dependencies in favor of log4j-1.2-api
2015-03-26 13:21:13 -07:00
9cd6c08e96
Exclude log4j from curator dependencies in favor of log4j-1.2-api
2015-03-26 13:05:12 -07:00
47cc11829d
Merge pull request #1240 from himanshug/async_query_forwarding_servlet_router_fix
...
adding router numMaxThread configuration for ProxyServlet HttpClient executor
2015-03-26 10:15:44 -07:00
942c17a2aa
Remove timeline chunk count assumptions.
...
* Replace with generic iterables
2015-03-24 22:40:49 +01:00
b131a4fd3b
adding router numMaxThread configuration for ProxyServlet HttpClient executor
2015-03-23 21:58:01 -05:00
3b894f9fb9
add ut for TierSelectorStrategies
...
fix test
2015-03-24 08:18:28 +05:30
23528299c9
Merge pull request #1235 from druid-io/comments-update
...
Minor update to zk comments to make it more clear what is happening
2015-03-23 14:02:08 -07:00
a0ce1caaf8
minor update to zk comments to make it more clear what is happening
2015-03-23 11:47:43 -07:00
b5b32dbcb4
in test explicitly setting maxThreads for ProxyServlet to work around https://tickets.puppetlabs.com/browse/TK-152
2015-03-23 12:27:19 -05:00
b389cfe404
[maven-release-plugin] prepare for next development iteration
2015-03-19 12:38:17 -07:00
60e7d543cc
[maven-release-plugin] prepare release druid-0.7.1-rc1
2015-03-19 12:38:13 -07:00
21cd75bcf6
improve alert
...
skip doing false alert when the coordinator looses leadership
2015-03-18 09:42:22 +05:30
8d0dfd9af1
Merge pull request #1221 from metamx/AnnounceIdUUID
...
Change announcement ID to a UUID instead of Timestamp
2015-03-17 15:38:53 -07:00
79b1443cc3
Change announcement ID to a UUID instead of Timestamp
...
* Also add a UUIDUtils to the common package
2015-03-17 15:34:04 -07:00
f9821d242f
also log author ip address in audit log
2015-03-17 23:15:15 +05:30
526a386f50
Merge pull request #1193 from metamx/reduce-test-verbosity
...
move test output to file for cleaner build logs
2015-03-12 15:42:34 -07:00
3ec4a30ab5
initial commit
...
review comments
more refactoring and cleaning of redundant code
add UT + docs + more refactoring
fixes + review comments
more cleanup
end points to fetch history
review comments
remove unnecessary changes
review comments rename header name
review comments + add test for MetadataRulesManager
review comments docs
2015-03-12 22:50:29 +05:30
fc613771d2
move test output to file for cleaner build logs
...
- removes the need for special test log4j2.xml
2015-03-11 17:56:19 -07:00
9f242ed1ba
Merge pull request #1182 from druid-io/better-docs
...
Rework the druid docs and fix many mistakes
2015-03-10 14:23:38 -07:00
064d65a009
rework the druid docs and fix many mistakes
2015-03-10 14:19:19 -07:00
ef842b2eae
add test coverage
2015-03-09 14:32:13 -07:00
e161021902
Merge pull request #1171 from druid-io/fix-rt-task
...
Add more metrics and fail when realtime tasks fail to persist and merge
2015-03-06 16:42:23 -08:00
86a0a3cab8
add more metrics and fail when realtime tasks fail to persist and merge
2015-03-06 11:00:34 -08:00
956361dbbd
Refactoring: Moved BaseJettyServerInitializer.java to JettyServerInitUtils.java and changed it from abstract class to a utils class containing static methods
...
And, moved jetty stuff to its own package io.druid.server.initialization.jetty
2015-03-05 19:33:27 -06:00
a8648d8f3d
feature to add servlet filters in a druid node via extension modules
2015-03-03 22:35:02 -06:00
d8e199a3f5
Merge pull request #1161 from himanshug/zk_ood_updates
...
Fix to maintain correctness when out-of-order ZK updates are received
2015-03-03 14:42:31 -08:00
dda2a62ff5
on receiving ZK ADD/UPDATE events, get latest data from zookeeper instead of taking it from the event which might be stale due to event coming out of order etc
2015-03-03 16:21:56 -06:00
9578304d6f
Merge pull request #1130 from housejester/jde-building
...
Various improvements to the tutorial and building experience.
2015-03-02 14:48:32 -08:00
e8605c63a9
Merge pull request #1150 from himanshug/broker-parallel-chunk-process
...
interval chunk query runner now processes individual chunk in a threadpool
2015-03-02 13:50:23 -08:00
29039fd541
interval chunk query runner now processes individual chunk in a thread pool and prints metrics query/time per chunk
2015-03-02 15:45:09 -06:00
a76a24742b
Merge pull request #1149 from himanshug/broker-get-datasource-update
...
updating ClientInfoResource.getDatasource() with a full flag
2015-03-02 13:28:27 -08:00
a39f4f8bf4
updating ClientInfoResource.getDatasource() with a full flag to return all the served intervals for given data source along with the dimension and metric information
2015-03-02 14:18:52 -06:00
17743b94f3
Cleanup router servlet + tests for gzip proxying
...
- Uses method overrides instead of modified Jetty code, now that
ProxyServlet provides enough method hooks for proper overrides.
This means we may also benefit from any Jetty ProxyServlet fixes
- Adds test for async proxy servlet to make sure gzip encoding is
properly proxied.
- Router now proxies POST requests for requests that are not Druid
queries, by only treating /druid/v2/* endpoints as queries.
2015-03-02 11:36:15 -08:00
c21086b6a8
Merge pull request #1113 from metamx/fix1111
...
Make Realtime queries not submit to thread pools for the master merger
2015-02-27 11:13:25 -08:00
9cb5d9bd00
Various improvements to the tutorial and building experience.
...
- Improve geo ip download stability
- Retry default geo ip download if corrupt
- Clarify docs on building & where to get artifact
2015-02-25 18:49:16 -07:00
b633111c76
Merge pull request #1156 from druid-io/fix-log
...
Print better messages when there are problems with ingestion
2015-02-25 17:02:46 -08:00
9a3ed353dc
Have a more meaningful message when events are dropped or unparseable
2015-02-25 16:59:46 -08:00
2192d0cb57
UTs for server
2015-02-25 15:45:58 -08:00
69ba802abb
Merge pull request #1109 from metamx/fixAnnouncerTest
...
Make the Announcer closing and unannouncing atomic.
2015-02-25 15:24:18 -08:00
aa9dbb773d
Merge pull request #1153 from metamx/announcerTestForWaitingFix
...
Use forWaiting() in AnnouncerTest
2015-02-24 21:48:54 -08:00
29a3522708
Use forWaiting() in AnnouncerTest
...
* Helps reduce random failures in unit test on slow systems
2015-02-24 08:35:07 -08:00
b167dcf82c
[maven-release-plugin] prepare for next development iteration
2015-02-23 14:28:06 -08:00
e81ac2ba43
[maven-release-plugin] prepare release druid-0.7.0
2015-02-23 14:27:58 -08:00
25db9abb7f
Merge pull request #1138 from metamx/better-default-hostname
...
Better default hostname
2015-02-18 17:37:34 -08:00
53d2b961c5
default to canonical hostname instead of localhost
2015-02-18 16:44:48 -08:00
3fab045b2d
Go through and fix mistakes in tutorials and docs
2015-02-17 15:21:16 -08:00
2a45501393
Merge pull request #1110 from metamx/move-to-sonatype
...
Move Druid release artifacts to Sonatype
2015-02-17 11:54:16 -08:00
b42b41cca7
Bind value of column "partitioned" as boolean, as it is defined in the "create table" ddl.
2015-02-17 10:22:10 -08:00
2593864c52
Merge pull request #1126 from gianm/md-insert-or-update-tx
...
Use jdbi inTransaction for SQLMetadataConnector's insertOrUpdate.
2015-02-16 08:23:05 -08:00
e00de8c19f
Use jdbi inTransaction for SQLMetadataConnector's insertOrUpdate.
...
This fixes an issue where failed transactions were not properly rolled back.
2015-02-16 07:42:43 -08:00
78df7f6165
Move Druid release artifacts to Sonatype
...
- Switch to using Druid parent POM
- Add required fields for Sonatype
- Common plugin versions and settings have been moved to the parent pom
- Cleanup artifacts and POMs for consistent formatting
- Remove org.hyperic.sigar dependency and update docs to reflect necessary jars to add at runtime when sigar is needed
2015-02-13 14:26:31 -08:00
d29740ed9f
[maven-release-plugin] prepare for next development iteration
2015-02-12 16:16:00 -08:00
211fd15b7e
[maven-release-plugin] prepare release druid-0.7.0-rc3
2015-02-12 16:15:56 -08:00
90bc62eb5c
Merge pull request #1108 from metamx/improve-groupby-perf
...
Improve groupby by removing conversion to case insensitive row
2015-02-12 11:45:20 -08:00
bd500fcf4b
Make the Announcer closing and unannouncing atomic.
2015-02-11 19:47:04 -08:00
e116600c7f
Make Realtime queries not submit to thread pools for the master merger
...
Fix https://github.com/druid-io/druid/issues/1111
2015-02-11 15:47:40 -08:00
15cf432b74
remove conversion to case insensitive row
...
this is not required after death to casing in 0.7
2015-02-11 19:40:36 +05:30
d51b37cfd5
Remove QueryResource's ObjectMapper::copy() in favor of using the injected ObjectMapper directly
...
* Workaround for https://github.com/FasterXML/jackson-databind/issues/696
2015-02-10 16:50:23 -08:00
31e248736d
Merge pull request #1106 from druid-io/update-http-client
...
Update http client to 1.0.0
2015-02-10 13:37:27 -08:00
708759e1e0
Update http-client to 1.0.0
2015-02-10 13:36:47 -08:00
1418cbb0e9
Merge pull request #1103 from metamx/fix-request-time
...
fix change in request/time meaning introduced by d2c729afixes #1102
2015-02-09 17:23:05 -08:00
2db3144f04
fix change in request/time meaning introduced by d2c729a, fixed #1102
2015-02-09 16:41:30 -08:00
040e771da2
Merge pull request #1099 from metamx/console_cleanup
...
console cleanup
2015-02-08 09:06:05 -08:00
f47f0d51c7
removed unecessary logging
...
fixed simplifyInterval logic
2015-02-06 21:13:25 -08:00
a0fc20fc09
better regex for durations
2015-02-06 17:02:31 -08:00
dae91e5894
fixed links back to old console
2015-02-04 14:21:46 -08:00
1f12c5b2f1
[maven-release-plugin] prepare for next development iteration
2015-02-03 12:06:49 -08:00
e82d431be7
[maven-release-plugin] prepare release druid-0.7.0-rc2
2015-02-03 12:06:41 -08:00
4eff269536
Merge pull request #1079 from druid-io/cleanup-deps
...
Remove non friendly dependencies from Druid
2015-02-03 11:56:41 -08:00
ffb180e29d
Merge pull request #1080 from metamx/fix-cost-balancer-strategy
...
fix integer value overflow and add test
2015-02-03 11:45:03 -08:00
6da6b9eeda
fix integer value overflow and add test
2015-02-03 11:39:30 -08:00
3e5d338c8e
Remove non friendly dependencies from Druid
2015-02-03 11:36:08 -08:00
42eba986ce
Towards consistent null handling
...
This commit also includes
1) the addition of a context parameter on timeseries queries that allows it to ignore empty buckets instead of generating results for them
2) A cleanup of an unused method on an interface
2015-02-02 12:53:07 -08:00
92e616de11
Merge pull request #1077 from metamx/remove-unused-imports
...
remove unused imports
2015-02-02 10:45:27 -08:00
ba932bb1f2
remove unused imports
2015-02-02 21:53:39 +05:30
d05032b98a
towards a community led druid
2015-01-31 20:57:36 -08:00
b94b24c46e
Merge pull request #1072 from metamx/common-aws-config
...
Common AWS module + consistent credential chain for both S3 and auto-scaling
2015-01-30 14:41:24 -08:00
14837d7808
Merge pull request #1075 from metamx/fix-test-timeout
...
increase test timeouts on DruidClusterBridgeTest
2015-01-30 14:31:32 -08:00
cd7e768640
increase test timeouts on DruidClusterBridgeTest
2015-01-30 13:18:35 -08:00
bc1405bee0
fix worker status path announcement with indexer zk config
2015-01-30 12:26:08 -08:00
f00872c41b
move common AWS related classes into a separate module
2015-01-29 13:55:49 -08:00
2d96b62e6c
Merge pull request #1066 from metamx/fix-background-caching-ordering
...
fix background caching not preserving result order
2015-01-28 06:09:17 +08:00
59c1907322
add test for out-of-order background-caching execution
2015-01-27 13:38:08 -08:00
cd9ba1f852
fix serde bug
2015-01-27 18:47:36 +08:00
b334d7973d
fix background caching not preserving result order
2015-01-26 09:59:18 -08:00
26c5c2193f
fix BatchServerInventoryViewTest test timeout
2015-01-20 17:14:57 -08:00
1f94de22c6
[maven-release-plugin] prepare for next development iteration
2015-01-20 14:23:55 -08:00
17476edc31
[maven-release-plugin] prepare release druid-0.7.0-rc1
2015-01-20 14:23:51 -08:00
5a2030c49b
Merge pull request #1059 from druid-io/remove-test
...
Remove test rejection policy - it has no place anywhere
2015-01-20 13:39:46 -08:00
1ff5829d6d
remove test rejection policy
2015-01-20 13:22:30 -08:00
3d27747f7e
Upgrade to log4j2
...
Default behavior is as before.
Added documentation for how to enable synchronous logging for select chatty classes:
* io.druid.client.ServerInventoryView
* io.druid.client.BatchServerInventoryView
* io.druid.curator.inventory.CuratorInventoryManager
* com.metamx.http.client.pool.ChannelResourceFactory
2015-01-20 12:35:18 -08:00
91a79dbf95
Merge pull request #1031 from metamx/ingestmetadata-query
...
DataSourceMetadata query
2015-01-19 21:55:35 -08:00
56bad2114a
Merge pull request #1054 from metamx/increase-test-wait-time
...
use waiting multiple when waiting in AnnouncerTest
2015-01-19 11:03:35 -08:00
b5f0d119f1
use waiting multiple when waiting in AnnouncerTest
2015-01-19 10:19:13 -08:00
52f65b1273
Fix a small bug where an NPE can occur if a closeable is not present
2015-01-17 15:23:01 -08:00
38c155e143
Merge branch 'master' of https://github.com/druid-io/druid
2015-01-16 13:14:49 -08:00
b4041c13e5
Merge pull request #1029 from metamx/fixChainedExecutionQueryRunnerTest
...
Address spurious test failures
2015-01-16 13:08:32 -08:00
6bc6e71b0d
Add messages to catch spurious test failures
2015-01-15 16:49:43 -08:00
f7ea308b25
fix minor error output problem
2015-01-15 15:57:45 -08:00
5bfcc43377
Merge pull request #1008 from metamx/stringConversionJavaUtilUpdate
...
Update all String conversions to and from byte[] to use the java-util StringUtils functions
2015-01-15 13:50:27 -08:00
c7452b75f6
Merge branch 'master' into ingestmetadata-query
2015-01-15 18:00:31 +05:30
021a7de759
Add retry capabilities for realtime logic
2015-01-13 14:30:49 -08:00
84cc32ba32
Add more logging for when ingestion may be throttled
2015-01-08 14:51:46 -08:00
c1295202ff
Allow background caching run after the futures are finished
2015-01-07 13:23:34 -08:00
f6fbb733b8
Added a few places where tests were using Object instead of Module
2015-01-05 13:47:25 -08:00
852e863425
Merge pull request #981 from druid-io/strictModuleTyping
...
Use Module instead of generic Object in Guice related items
2015-01-05 12:43:20 -08:00
ab0ddc0d7c
Merge pull request #998 from drcrallen/zkConfigJackson
...
Change zk configs to use Jackson injection instead of Skife
2015-01-05 12:42:40 -08:00
b1b5c9099e
Update all String conversions to and from byte[] to use the java-util StringUtils functions
...
* Speedup of GroupBy with javaScript filters by ~10%
* Requires https://github.com/metamx/java-util/pull/15
2015-01-05 11:22:32 -08:00
81fdc791e0
Add memory hinting and concurrency helpers in CachingClusteredClient
2015-01-02 15:54:26 -08:00
65286a24e0
Change zk configs to use Jackson injection instead of Skife
...
* Also added generic config testing class JsonConfigTesterBase
2014-12-29 10:36:12 -08:00
574ab3017c
fix test timeout
2014-12-23 16:25:38 -08:00
bf4b52384c
Merge pull request #988 from drcrallen/queryErrorFixer
...
Fix NPE in QueryResource on bad query
2014-12-22 23:16:59 -08:00
458236c8fd
Add simple unit tests for QueryResource
2014-12-22 21:53:58 -08:00
4338af0e3f
Fix NPE in QueryResource on bad query
2014-12-22 12:59:47 -08:00
badc09e85c
Disable checkGzExists in the default GzipFilter.
...
It is unhelpful because we don't actually have any precomputed .gz resources,
and checking for them inside jars is expensive.
2014-12-22 12:45:48 -05:00
7c8d4a7433
Use Module instead of generic Object in Guice related items
2014-12-19 10:54:06 -08:00
a0b2a63d7f
Fix deadlock in DirectDruidClient
2014-12-16 18:34:30 -08:00
e9238161cd
fix broken test
2014-12-15 18:12:44 -08:00
242edd14e5
Add deprecated smile media type for backwards compatability during upgrades
2014-12-15 18:00:10 -08:00
a520835972
Merge pull request #957 from metamx/fix-gzip-encoding
...
fix gzip compression
2014-12-15 13:20:36 -08:00
e166260f80
more inventory logs
2014-12-15 13:19:47 -08:00
9c6e32fc3f
address cr
2014-12-15 13:06:36 -08:00
61101bee6c
more logs for CIM
2014-12-15 12:57:40 -08:00
b3fe91bb50
Merge pull request #830 from metamx/union-merge-on-historical
...
Union merge on historical
2014-12-15 13:36:47 -07:00
31cda0b830
Merge pull request #958 from metamx/fix-router-service-discovery
...
fix router discovering brokers with default names
2014-12-15 13:35:39 -07:00
77bf5da317
fix typo
2014-12-15 10:07:51 -08:00
3c33118f3a
Merge pull request #962 from metamx/add-broker-loadstatus
...
add loadstatus endpoint for serverView status
2014-12-12 17:13:48 -08:00
ca8300a461
Replace AppendableByteArrayInputStream in DirectDruidClient
...
* Replace with SequenceInputStream fueled by an enumeration of ChannelBufferInputStream which directly wrap the response context ChannelBuffer
* Added zero-length byte array when DirectDruidClient is done. This prevents an odd race condition on `done`.
2014-12-12 14:01:13 -08:00
4b5282d224
add loadstatus endpoint for serverView status
2014-12-12 13:18:31 -08:00
352480ac77
make method protected to prevent accidental usage
...
- add documentation as to why
2014-12-11 15:20:36 -08:00
3abf348d2c
fix router discovering brokers with default names
2014-12-11 12:16:44 -08:00
7ad03087c0
Merge branch 'master' into ingestmetadata-query
2014-12-11 16:54:38 +05:30
32b4f55b8a
review comments refactoring
2014-12-11 16:33:14 +05:30
eb661d3130
/status replaces /health
2014-12-10 17:35:05 -08:00
0de56efe5c
fix gzip compression
...
- fixes compression not working for POST requests
- router now forwards raw bytes instead of decompressing
- cleanded up router servlet initialization
- add test for gzip on get and post methods
- use Jersey annotation when possible in QueryResource
2014-12-10 17:30:17 -08:00
7878d391bd
use first available port for JettyTest
2014-12-10 12:53:11 -08:00
6f6cc463b0
Merge pull request #950 from metamx/fix-bysegment-serde
...
Fix bySegment metrics de-serialization
2014-12-10 12:42:40 -08:00
5f3f4e0a60
prevent /status from redirect, fixes #953
2014-12-10 11:32:22 -08:00
7853e801d5
fixes for bySegment queries and caching
...
- prevent bySegment queries from getting cached
- avoid computing cache keys if not populating or using cache
- avoid computing cache data when not populating cache
2014-12-10 10:57:48 -08:00
3763357f6e
Ingest metadata query implementation
2014-12-10 19:44:00 +05:30
c44242f0af
add comment about de-serialing bySegment
2014-12-09 17:02:24 -08:00
a7da9aed9f
fix metrics de-serialization for bySegment results
2014-12-09 17:02:23 -08:00
7620dfbd1a
fix broken ut
2014-12-09 15:43:07 -08:00
d6d3ec6846
Merge pull request #948 from metamx/ingestion-docs
...
Redocumenting ingestion
2014-12-09 15:30:03 -07:00
1a1b0e6f23
merge from master and review comments
2014-12-09 13:16:45 +05:30
28b72a69ad
redocumenting ingestion
2014-12-08 16:15:46 -08:00
623443a145
Merge pull request #946 from metamx/MoreMediaTypes
...
Changed all "application/json" to MediaType.APPLICATION_JSON except for in druid.js
2014-12-08 15:24:21 -07:00
a0f9f9877e
Changed all "application/json" to MediaType.APPLICATION_JSON except for in druid.js
2014-12-08 14:21:49 -08:00
1392e2731f
Merge pull request #936 from metamx/cachingRunnerImprovements
...
General Caching Query Runners cleanup (40% query time reduction for HLL)
2014-12-08 14:07:52 -08:00
7b65f0635d
General Caching Query Runners cleanup
...
* Add type strictness to CachingClusteredClient.
* Add background caching to CachingClusteredClient. Gives between 0% and 5% query speed increase.
* Add @BackgroundCaching annotation for injected ExecutorService items
* Add `numBackgroundThreads' configuration options to CacheConfig (default 0 aka same thread legacy behavior)
* Add unit tests for CacheConfig
* Add an abstract caching query runner class, currently it doesn't do anything exceppt simply make the two caching queries distinct.
* Add caching to CachingQueryRunner. Gives up to a WHOPPING 40% reduction in query time on HLL queries
* Updated docs with more info on cache settings.
2014-12-08 13:29:32 -08:00
c48839a0a0
Merge pull request #945 from metamx/MediaTypeFixes
...
Added checks for backwards compat on MediaTypes
2014-12-08 14:11:58 -07:00
43041ed4ba
Added checks for backwards compat on MediaTypes
2014-12-08 13:11:00 -08:00
eb233fe50e
Merge pull request #921 from metamx/onheap-incremental-index
...
separate implementations of incremental index
2014-12-08 13:59:08 -07:00
03f7fae3e8
Changed references to MediaType to pull from javax.ws.rs.core.MediaType or com.fasterxml.jackson.jaxrs.smile.MediaType
2014-12-08 12:39:36 -08:00
c88de97370
Merge branch 'master' into onheap-incremental-index
...
Conflicts:
server/src/main/java/io/druid/segment/realtime/RealtimeManager.java
2014-12-08 11:48:21 -08:00
3a7f571aeb
Merge pull request #942 from metamx/fix-tiered-selector-naming
...
fix tiered selector service name usage
2014-12-08 12:09:45 -07:00
ef86dec660
remove unused but required parameter
2014-12-08 19:25:46 +05:30
4883f5526b
fix tiered selector service name usage
2014-12-08 19:22:27 +05:30
c255c16c1a
Merge pull request #894 from zhaown/fix-query-realtime-with-multi-partition
...
bugfix for #887
2014-12-07 09:49:31 -07:00
629f6bc759
moved methods to java-util
2014-12-05 15:53:23 -08:00
3119770ce9
Reformatting using intellij_formatting.jar
2014-12-05 14:21:24 +08:00
6e03a6245f
Merge branch 'master' into onheap-incremental-index
2014-12-05 10:40:28 +05:30
d4a1752c5d
add code to release ByteBuffers
2014-12-04 16:18:55 -08:00
7cd45a6e1f
IncrementalIndex throws exception if limit exceeded
...
- For now uses a hardcoded ratio of aggregator to timeanddim buffer sizes
- canAppendRow is a workaround for realtime index since the
Firehose currently does not have a way of rolling back the last event in
case of error
- canAppendRow needs a fudge factor; there is a race between checking
if we can add a row and actually adding a row, because of the way MapDB
reports its size.
2014-12-04 14:38:16 -08:00
20a7239ffd
Replace google-http-client imports with real guava imports.
2014-12-04 10:57:57 -08:00
b48fc51ed1
SSL-capable ServiceEmitter.
2014-12-04 10:57:56 -08:00
0c521e0a77
update joda-time and fix min/max instant
2014-12-04 10:57:56 -08:00
187126dd4b
fix broken unit tests
2014-12-04 10:49:36 -08:00
332e4a4d1c
Merge pull request #902 from metamx/cache-config
...
Commonalize the cache config and change default to disable cache
2014-12-04 11:30:24 -07:00
2681da4420
Merge pull request #929 from metamx/google-cleanup
...
Replace google-http-client imports with real guava imports.
2014-12-03 11:50:19 -08:00
d388a8fe89
Replace google-http-client imports with real guava imports.
2014-12-03 10:52:57 -08:00
03ae70ae54
SSL-capable ServiceEmitter.
2014-12-03 10:43:24 -08:00
4dc0fdba8a
consider mapped size in limit calculation & review comments
2014-12-03 23:47:30 +05:30
da8bd7836b
Introduce buffer size
2014-12-03 16:28:22 +05:30
a79389a9e5
update joda-time and fix min/max instant
2014-12-02 17:27:22 -08:00
eac776f1a7
tests passing with on heap incremental index
2014-12-02 22:29:28 +05:30
Charles Allen
Jonathan Wei
David Lim
Nishant
binlijin
Himanshu
Parag Jain
Slim
Gian Merlino
Xavier Léauté
Jisoo Kim
Fangjin Yang
Xavier Léauté
jon-wei
jisookim
Bingkun Guo
kilida
Parag Jain
navis.ryu
rasahner
Erik Dubbelboer
Slim Bouguerra
Björn Zettergren
Eric Tschetter
Harsh Jain
Himank Chaudhary
Charles Allen
Jaehong Choi
Kurt Young
Anubhav Gupta
Robin
Nikita Geer
刘麒贇
Bartosz Ługowski
lvjq
Parag Jain
samjhecht
msprunck
cheddar
Himanshu Gupta
James Estes
extpvaan
Tarek Rached
Weinan Zhao