Commit Graph

3067 Commits

Author SHA1 Message Date
Jonathan Wei c2a0e753b6 Extension points for authentication/authorization (#4271)
* Extension points for authentication/authorization

* Address some PR comments

* Authorization result caching

* Add unit tests for SecuritySanityCheckFilter and PreResponseAuthorizationCheckFilter

* Use Set for auth caching, close outputstreams in filters

* Don't close output stream on success in sanity check filter

* Add ConfigResourceFilter to coordinator lookups

* Fix filtering authorization check for empty resource list

* HttpClient users must explicitly escalate the client

* Remove response modification from PreResponseAuthorizationCheckFilter

* Remove extraneous pom.xml

* Fix unit test

* Better lifecycle management

* Rename AuthorizationManager to Authorizer

* Fix authorization denials for empty supervisor list

* Address some PR comments

* Address more PR comments

* Small cleanup

* Add Jetty HttpClient wrapper to Authenticator

* Remove Authorizer start/stop

* Restore immutable context map in DruidConnection, UT fix

* Fix/update docs

* Add authorization checks to EventReceiverFirehose

* Fix router authorization check failure, restore PreResponseAuthorizationFilter changes

* Compile fixes

* Test fixes

* Update Authenticator/Authorizer doc comments

* Merge fixes

* PR comments

* Fix test

* Fix IT

* More PR comments

* PR comments

* SSL fix
2017-09-15 23:45:48 -07:00
Himanshu d37be5e6e9 don't hold thread while waiting after failure from server (#4795) 2017-09-14 17:19:25 -05:00
Akash Dwivedi a17e48fe69 search package name correction. (#4785)
* search package name correction.

* Refactor search.search pkg to search.

* remove unused import.
2017-09-14 13:50:23 -07:00
Roman Leventov 267f415dc3 Update emitter library and add support for ParametrizedUriEmitter (#4722)
* Move emitters from io.druid.server.initialization to the dedicated io.druid.server.emitter package; Update emitter library to 0.6.0; Add support for ParametrizedUriEmitter; Support hierarical properties in JsonConfigurator (was needed for ParametrizedUriEmitter)

* Log created RequestLoggers

* Fix forbidden API

* Test fix

* More Http and Parametrized Http Emitter docs

* Switch to debug level
2017-09-13 17:17:19 -05:00
Himanshu 7919469de6 fixes HttpServerInventoryView to call server/segment callbacks correctly and Unit Tests for the class (#4767)
* fixes HttpServerInventoryView to call server/segment callbacks correctly and Unit Tests for the class

* fix checkstyle and forbidden-api errors

* HttpServerInventoryView to finish start() only after server inventory is initialized

* fix compilation errors

* address review comments

* add exponential backoff instead of fixed 5 secs on successive failures

* update test to exercise server fail scenarios

* use AtomicInteger for requestNum and increment only once
2017-09-13 14:24:19 -05:00
Gian Merlino 2ce8123bdb Move scan-query from a contrib extension into core. (#4751)
* Move scan-query from a contrib extension into core.

Based on a proposal at: https://groups.google.com/d/topic/druid-development/ME_OatUDnbk/discussion

This patch also adds support for virtual columns to the Scan query,
and updates Druid SQL to use Scan instead of Select.

This patch also makes some behavioral changes to handling of the __time
column. In particular, it is now is returned as "__time" rather than
"timestamp"; it is no longer included if you do not specifically ask for
it in your "columns"; and it is returned as a long rather than a string.

Users can revert time handling to the legacy extension behavior by
setting "legacy" : true in their queries, or setting the property
druid.query.scan.legacy = true. This is meant to provide a migration
path for users that were formerly using the contrib extension.

* Adjustments from review.

* Add back Select query.

* Adjust SQL docs.

* Restore SelectQuery link.
2017-09-13 09:51:24 -07:00
Roman Leventov 832cc293ef Refactoring of ReferenceCountingSegment and FireHydrant (#4154)
* Refactoring of ReferenceCountingSegment and FireHydrant

* Address comment

* Fix FireHydrant.closeSegment()

* Address comment

* Added comments to ReferenceCountingSegment
2017-09-12 14:28:35 -05:00
Gian Merlino 4909c48b0c SQL: Full TRIM support. (#4750)
* SQL: Full TRIM support.

- Support trimming arbitrary characters
- Support BOTH, LEADING, and TRAILING

* Remove unused import.

* Fix tests, add RTRIM / LTRIM.

* Remove unused imports.

* BTRIM and docs.

* Replace for with foreach.
2017-09-12 11:49:08 -07:00
Parag Jain b5e839b3db injectable sslcontextfactory for jetty server and key manager factory algorithm (#4769)
* injectable sslcontextfactory for jetty server

key manager factory algorithm

* explicitly set trustAll certificates to false in sslcontextfactory
2017-09-12 11:45:03 -07:00
Himanshu 834e050bc4 Use internal-discovery and http for talking to overlord/coordinator leaders (#4735)
* Use internal-discovery and http for talking to overlord/coordinator leaders

* CuratorDruidNodeDiscovery.getAllNodes() best effort 30 sec wait for cache initialization

* DruidLeaderClientProvider to eagerly instantiate DruidNodeDiscovery when needed so that DruidNodeDiscovery impl cache gets initialized well in time

* Revert "DruidLeaderClientProvider to eagerly instantiate DruidNodeDiscovery when needed so that DruidNodeDiscovery impl cache gets initialized well in time"

This reverts commit f1a2432614ba56ddc2d55fe47e990d17fcfd6129.

* add lifecycle to DruidLeaderClient to early initialize DruidNodeDiscovery so that it has its cache update well in time
2017-09-11 11:18:01 -07:00
Niketh Sabbineni 975a50fc81 Generate UUID if query id is empty (#4772) 2017-09-11 10:40:26 -07:00
dgolitsyn 752151f6cb Add CachingCostBalancerStrategy (#4731)
* Add CachingCostBalancerStrategy; Rename ServerView.ServerCallback to ServerRemovedCallback

* Fix benchmark units

* Style, forbidden-api, review, bug fixes

* Add docs

* Address comments
2017-09-08 12:23:04 -05:00
Himanshu d43687d578 increase test timeout to 15sec as leader election jitter is random sleep between 1 to 5 secs (#4756) 2017-09-06 09:56:25 -07:00
Kevin Conaway 83e94e8b31 Expose version property for `CustomVersioningPolicy` so that it may serialize/deserialize correctly (#4747)
Expose version property for  so that it may serialize/deserialize correctly

Expose version property for `CustomVersioningPolicy` so that it may serialize/deserialize correctly

Expose version property for CustomVersioningPolicy so that it may serialize/deserialize correctly

Expose version property for `CustomVersioningPolicy` so that it may serialize/deserialize correctly
2017-09-03 13:36:18 -07:00
Himanshu 06ac6678e6 DruidLeaderSelector interface for leader election and Curator based impl. (#4699)
* DruidLeaderSelector interface for leader election and Curator based impl. DruidCoordinator/TaskMaster are updated to use the new interface.

* add fake DruidNode binding in integration-tests module

* add docs on DruidLeaderSelector interface

* remove start/stop and keep register/unregister Listener in DruidLeaderSelector interface

* updated comments on DruidLeaderSelector

* cache the listener executor in CuratorDruidLeaderSelector

* use same latch owner name that was used before

* remove stuff related to druid.zk.paths.indexer.leaderLatchPath config

* randomize the delay when giving up leadership and restarting leader latch
2017-09-01 09:49:04 -07:00
Charles Allen bdfc6fe25e Move common TypeReference into JacksonUtils (#4738) 2017-08-31 13:40:16 -07:00
Roman Leventov 4d109a358a Refactoring of Storage Adapters (#4710)
* Factor QueryableIndexColumnSelectorFactory and IncrementalIndexColumnSelectorFactory out of QueryableIndexStorageAdapter and IncrementalIndexStorageAdapter; Add Offset.getBaseReadableOffset(); Remove OffsetHolder interface; Replace Cursor extends ColumnSelectorFactory with composition; Reduce indirection in ColumnValueSelectors created by QueryableIndexColumnSelectorFactory

* Don't override clone() in FilteredOffset (the prev. implementation was broken); Some warnings fixed

* Simplify Cursors in QueryableIndexStorageAdapter

* Address comments

* Remove unused and unimplemented methods from GenericColumn interface

* Comments
2017-08-28 18:07:31 -07:00
Parag Jain 594a66f3c0 add scheme to AsyncQueryForwardingServlet (#4688)
* add scheme to AsyncQueryForwardingServlet

* add sslContext binding for Router
2017-08-28 15:03:43 -07:00
Gian Merlino 9fbfc1be32 Add @ExtensionPoint and @PublicApi annotations. (#4433)
* Add @ExtensionPoint and @PublicApi annotations.

* Clean up wording.

* Remove unused import.

* Remove unused imports.

* Only types can be extension points.

* Adjust annotations some more.

* Remove unused import.

* Make ServletFilterHolder an extension point.

* Add a couple extension points, and update docs.
2017-08-28 14:50:58 -07:00
dgolitsyn fd0f349c68 Mark segments that are going to be dropped from the server and use it in DruidBalancer (#4486)
* Do not remove segment that should not be moved from currentlyMovingSegments (segments are removed by callbacks or not inserted)

* Mark segments that are going to be dropped from server and use this information in CostBalancerStrategy

* Fix tests
2017-08-25 18:25:56 -05:00
Himanshu 74538c3288 update internal-discovery Listener for node list and use same at router and coordinator (#4697)
* update LookupCoordinatorManager to use internal discovery to discover lookup nodes

* router:use internal-discovery to discover brokers

* minor [Curator]DruidDiscoveryProvider refactoring

* add initialized() method to DruidNodeDiscovery.Listener

* update HttpServerInventoryView to use initialized() and call segment callback initialization asynchronously

* Revert "update HttpServerInventoryView to use initialized() and call segment callback initialization asynchronously"

This reverts commit f796e441221fe8b0e9df87fdec6c9f47bcedd890.

* Revert "add initialized() method to DruidNodeDiscovery.Listener"

This reverts commit f0661541d073683f28fce2dd4f30ec37db90deb0.

* minor refactoring removing synchronized from DruidNodeDiscoveryProvider

* updated DruidNodeDiscovery.Listener contract to take List of nodes and first call marks initialization

* update HttpServerInventoryView to handle new contract and handle initialization

* router update to handle updated listener contract

* document DruidNodeDiscovery.Listener contract

* fix forbidden-api error

* change log level to info for unknown path children cache events in CuratorDruidNodeDiscoveryProvider

* announce broker only after segment inventory is initialized
2017-08-25 09:28:15 -07:00
Parag Jain 421e02203f fix null sslPort issue (#4696)
* fix null sslPort issue

* add tests
2017-08-24 15:19:07 -05:00
Roman Leventov cbd1902db8 Add forbidden-apis plugin; prohibit using system time zone (#4611)
* Forbidden APIs WIP

* Remove some tests

* Restore io.druid.math.expr.Function

* Integration tests fix

* Add comments

* Fix in SimpleWorkerProvisioningStrategy

* Formatting

* Replace String.format() with StringUtils.format() in RemoteTaskRunnerTest

* Address comments

* Fix GroupByMultiSegmentTest
2017-08-21 13:02:42 -07:00
Himanshu 74a64c88ab internal-discovery: interfaces for announcement/discovery, curator based impls (#4634)
* internal-discovery: interfaces for announcement/discovery, curator impls

* more tests

* address some review comments

* more fixes

* address more review comments

* simplify ObjectMapper setup in CuratorDruidNodeAnnouncerAndDiscoveryTest

* fix KafkaIndexTaskTest

* make lookupTier overridable via RealtimeIndexTask and KafkaIndexTask context

* make teamcity build happy
2017-08-16 13:07:16 -07:00
Niketh Sabbineni dba7c7d3cd Reduce excessive logging (#4680)
* Reduce excessive logging

* Refactoring code as per comments
2017-08-15 10:14:20 -07:00
Niketh Sabbineni eb0deba54a Fix NPE when locations are empty (#4667)
* Fix NPE when locations are empty

* Addressing comments
2017-08-10 23:31:28 -07:00
Roman Leventov bf28d0775b Remove QueryRunner.run(Query, responseContext) and related legacy methods (#4482)
* Remove QueryRunner.run(Query, responseContext) and related legacy methods

* Remove local var
2017-08-11 09:12:38 +09:00
Roman Leventov 7454fd86a0 Polymorphic numeric getters for ColumnValueSelector (#4623)
* Add methods getFloat(), getDouble() and getLong() to ColumnValueSelector

* Fix copy-paste mistake in docs

* Spelling
2017-08-08 18:38:06 -07:00
Jihoon Son d5606bc558 Passing lockTimeout as a parameter for TaskLockbox.lock() (#4549)
* Passing lockTimeout as a parameter for TaskLockbox.lock()

* Remove TIME_UNIT

* Fix tc fail

* Add taskLockTimeout to TaskContext

* Add caution
2017-08-08 18:21:07 -07:00
Roman Leventov f5d4171459 Prohibit for loops which could be foreach with IntelliJ (#4653)
* Replace for with foreach

* Replace for with for-each in GroupByQueryEngineV2

* Remove io.druid.collections.IntList
2017-08-08 18:05:33 -07:00
dgolitsyn 4dd1e2b59e Do not remove segments from currentlyMovingSegments in DruidBalancer if move is impossible or not needed (#4472)
* Do not remove segment that should not be moved from currentlyMovingSegments (segments are removed by callbacks or not inserted)

* Replace putIfAbsent with computeIfAbsent in DruidBalancer

* Refactoring
2017-08-08 17:22:59 +03:00
Roman Leventov aa7e4ae5e4 Enforce correct spacing with Checkstyle (#4651) 2017-08-05 10:18:25 -07:00
Himanshu 163b0edd79 Router misc fixes (#4517)
* make BrokerQueryResource instantiation singleton

* fix druid.router.http.* handling so that they are actually used and introduce numRequestsQueued for jetty http client at router

* address comments

* address review comment
2017-08-03 16:37:57 +09:00
Himanshu 6d60ef67ce misc http segment discovery fixes (#4618)
* Use ConcurrentHashMap to store segment servers or else getInventory() would need to clone the values list

* introduce unstableTimeout for segment servers

* address review comment

* add HttpServerInventoryViewConfigTest
2017-08-02 14:11:26 -07:00
Niketh Sabbineni cbb9f7d214 Log used flag (#4614) 2017-07-31 16:58:36 -05:00
Egor Riashin 2005c5532f ArchiveTask list-unused query optimization (#4600)
* ArchiveTask list unused query optimization

* ArchiveTask list unused query optimization
2017-07-26 20:51:20 +03:00
Gian Merlino 5048ab3e96 Add metrics to the native queries underpinning SQL. (#4561)
* Add metrics to the native queries underpinning SQL.

This is done by factoring out the metrics and request log emitting
code from QueryResource into a new QueryLifecycle class. That class
is used by both QueryResource and the SQL DruidSchema and QueryMaker.

Also fixes a couple of bugs in QueryResource:

- RequestLogLine start time was set to `TimeUnit.NANOSECONDS.toMillis(startNs)`,
  which is incorrect since absolute nanos cannot be converted to millis.
- DruidMetrics.makeRequestMetrics was called with null `query` on
  unparseable queries, which led to spurious "Unable to log query"
  errors.

Partial fix for #4047.

* Code style

* Remove unused imports.

* Fix tests.

* Remove unused import.
2017-07-24 21:26:27 -07:00
Atul Mohan 4bd0f174ba Changes for deduplication (#4581) 2017-07-24 11:12:23 -05:00
Roman Leventov 7408a7c4ed Refactor CachingClusteredClient.run() (#4489)
* Refactor CachingClusteredClient

* Comments

* Refactoring

* Readability fixes
2017-07-23 23:10:36 +09:00
Roman Leventov c0beb78ffd Enforce brace formatting with Checkstyle (#4564) 2017-07-21 10:26:59 -05:00
Gian Merlino 2be7068f6e Fixes and improvements to SQL metadata caching. (#4551)
* Fixes and improvements to SQL metadata caching.

Also adds support for MultipleSpecificSegmentSpec to CachingClusteredClient.

SQL changes:
- Cache metadata on a per-segment level, in addition to per-dataSource, so
  we don't need to re-query all segments whenever a single new one appears.
  This should lower the load placed on the cluster by metadata queries.
- Fix race condition in DruidSchema that can cause us to miss metadata. It was
  possible to notice new segments, then issue a query, and have that query
  not actually hit those segments, and not notice that it didn't hit those segments.
  Then, the metadata from those segments would be ignored.
- Fix assumption in DruidSchema that all segments are immutable. Now, mutable
  segments are periodically re-queried.
- Fix inappropriate re-use of SchemaPlus. Now we create one for each planning
  cycle, rather than sharing one. It caches table objects, which we want to
  avoid, since it can cause stale metadata. We do the caching in DruidSchema
  so we don't need the SchemaPlus caching.

Server changes:
- Add a TimelineCallback to TimelineServerView, for callers that want to get updates
  when the timeline has been modified.
- Change CachingClusteredClient from a QueryRunner to a QuerySegmentWalker. This
  allows it to accept queries that are segment-descriptor-based rather than
  intervals-based. In particular it will now support MultipleSpecificSegmentSpec.

* Fix DruidSchema, and unused imports.

* Remove unused import.

* Fix SqlBenchmark.
2017-07-20 10:14:15 -07:00
Slim 71e7a4c054 Adding double colums supports (#4491)
* add double columns support

* Fix numbers and expected results in UTs

* adding float aggregators

* fix IT expected test results

* fix comments

* more fixes

* fix comp

* fix test

* refactor double and float aggregator factories

* fix

* fix UTs

* fix comments

* clean unused code

* fix more comments

* undo unnecessary changes

* fix null issue

* refactor TopNColumnSelectorStrategyFactory

* fix docs

* refactor NumericTopNColumnSelectorStrategy

* fix return

* fix comments

* handle the null case in DimesionIndexer

* more null fixing

* cosmetic changes
2017-07-20 10:14:14 +03:00
Himanshu a862cc716f re-implement updation of CoordinatorDynamicConfig to not call manager.watch(..) in its constructor (#4557)
* re-implement updation of CoordinatorDynamicConfig to not call manager.watch(..) in its constructor

* address review comments
2017-07-17 14:33:59 -07:00
Roman Leventov 60cdf94677 Add PMD and prohibit unnecessary fully qualified class names in code (#4350)
* Add PMD and prohibit unnecessary fully qualified class names in code

* Extra fixes

* Remove extra unnecessary fully-qualified names

* Remove qualifiers

* Remove qualifier
2017-07-17 22:22:29 +09:00
Roman Leventov b7203510b8 Fix RemoteTaskRunner's auto-scaling (#3768)
* Rename ResourceManagementStrategy to ProvisioningStrategy, similarly for related classes. Make ProvisioningService non-global, created per RemoteTaskRunner instead. Add OverlordBlinkLeadershipTest.

* Fix RemoteTaskRunnerFactoryTest.testExecNotSharedBetweenRunners()

* Small fix

* Make SimpleProvisioner and PendingProvisioner more similar in details

* Fix executor name

* Style fixes

* Use LifecycleLock in RemoteTaskRunner
2017-07-14 09:11:39 +09:00
Chris Gavin 960cb07ea6 Fix some unnecessary use of boxed types and incorrect format strings spotted by lgtm. (#4474)
* Remove some unnecessary use of boxed types.

* Fix some incorrect format strings.

* Enable IDEA's MalformedFormatString inspection.

* Add a Checkstyle check for finding uses of incorrect logging packages.

* Fix some incorrect usages of the metamx logger.

* Bypass incorrect logger Checkstyle check where using the correct logger is not simple.

* Fix some more places where the wrong number of arguments are provided to format strings.

* Suppress `MalformedFormatString` inspection on legacy logging test.

* Use @SuppressWarnings rather than a noinspection suppression comment.

* Fix some more incorrect format strings.

* Suppress some more incorrect format string warnings where the incorrect string is intentional.

* Log the aggregator when closing it fails.

* Remove some unneeded log lines.
2017-07-13 12:15:32 -07:00
Roman Leventov b2865b7c7b Make possible to start Peon without DI loading of any querying-related stuff (#4516)
* Make QueryRunnerFactoryConglomerate injection lazy in TaskToolbox/TaskToolboxFactory

* Extract QueryablePeonModule and add druid.modules.excludeList config

* Typo
2017-07-12 13:18:25 -05:00
Jihoon Son 98b1385bcd Remove published segments immediately (#4530) 2017-07-11 10:09:32 -07:00
Jihoon Son c5d5263c10 Disable CacheDistributionTest by default (#4527) 2017-07-11 09:16:12 -07:00
Himanshu 298310ad78 router endpoint to expose brokers map (#4523) 2017-07-11 09:11:07 -07:00
Jihoon Son cc20260078 Early publishing segments in the middle of data ingestion (#4238)
* Early publishing segments in the middle of data ingestion

* Remove unnecessary logs

* Address comments

* Refactoring the patch according to #4292 and address comments

* Set the total shard number of NumberedShardSpec to 0

* refactoring

* Address comments

* Fix tests

* Address comments

* Fix sync problem of committer and retry push only

* Fix doc

* Fix build failure

* Address comments

* Fix compilation failure

* Fix transient test failure
2017-07-10 22:35:36 -07:00
Slim 8ffac8f5e7 fix issue-4513 (#4525) 2017-07-10 15:58:04 -07:00
Parag Jain 6e2f78f552 TLS support (#4270) 2017-07-06 17:40:12 -07:00
Jihoon Son a6d648a728 Fix a bug in DruidCluster.getAllServers() (#4500)
* Fix a bug in getAllServers

* Change to old style
2017-07-05 08:57:38 -07:00
Himanshu d757ef3e9b record remote address in jetty request log (#4492) 2017-06-30 10:38:39 -07:00
Roman Leventov 9ae457f7ad Avoid using the default system Locale and printing to System.out in production code (#4409)
* Avoid usages of Default system Locale and printing to System.out or System.err in production code

* Fix Charset in DruidKerberosUtil

* Remove redundant string format in GenericIndexed

* Rename StringUtils.safeFormat() to unimportantSafeFormat(); add StringUtils.format() which fails as well as String.format()

* Fix testSafeFormat()

* More fixes of redundant StringUtils.format() inside ISE

* Rename unimportantSafeFormat() to nonStrictFormat()
2017-06-29 14:06:19 -07:00
dgolitsyn a5651ea474 Remove primitive value instead of CoordinatorStats from LoadRule.assign method (#4469) 2017-06-29 12:48:49 -05:00
Roman Leventov ae900a4934 Update versions to 0.11.0-SNAPSHOT (#4483) 2017-06-28 17:05:58 -07:00
Roman Leventov 6173570425 Add ExtensionsConfig.excludeModules (#4438)
* Add ExtensionsConfig.excludeModules

* Add branch

* Refactor Initialization.getFromExtensions()

* excludeModules -> moduleExcludeList

* Initialization.getFromExtensions() and getLoadedModules() should return Collection, not Set

* Fix doc
2017-06-28 14:01:31 -07:00
Gian Merlino 4c33d0a00f Add some new expression functions and macros. (#4442)
* Add some new expression functions and macros.

See misc/math-expr.md for the list of added functions, except for
"like", which previously existed but was not documented.

* Add easymock to datasketches tests.

* Add easymock to distinctcount tests.

* Add easymock to virtual-columns tests.

* Code review comments.

* Clean up code a bit.

* Add easymock to scan-query tests.

* Rework ExprMacros that have multiple impls.

* Improve test coverage.
2017-06-28 10:15:58 -07:00
Roman Leventov 2fa4b10145 More fine-grained DI for management node types. Don't allocate processing resources on Router (#4429)
* Remove DruidProcessingModule, QueryableModule and QueryRunnerFactoryModule from DI for coordinator, overlord, middle-manager. Add RouterDruidProcessing not to allocate processing resources on router

* Fix examples

* Fixes

* Revert Peon configs and add comments

* Remove qualifier
2017-06-27 22:58:01 -07:00
dgolitsyn e04b8be52e maxSegmentsInQueue in CoordinatorDinamicConfig (#4445)
* Add maxSegmentsInQueue parameter to CoordinatorDinamicConfig and use it in LoadRule to improve segments loading and replication time

* Rename maxSegmentsInQueue to maxSegmentsInNodeLoadingQueue

* Make CoordinatorDynamicConfig constructor private; add/fix tests; set default maxSegmentsInNodeLoadingQueue to 0 (unbounded)

* Docs added for maxSegmentsInNodeLoadingQueue parameter in CoordinatorDynamicConfig

* More docs for maxSegmentsInNodeLoadingQueue and style fixes
2017-06-27 22:58:36 -05:00
Roman Leventov 05d58689ad Remove the ability to create segments in v8 format (#4420)
* Remove ability to create segments in v8 format

* Fix IndexGeneratorJobTest

* Fix parameterized test name in IndexMergerTest

* Remove extra legacy merging stuff

* Remove legacy serializer builders

* Remove ConciseBitmapIndexMergerTest and RoaringBitmapIndexMergerTest
2017-06-26 13:21:39 -07:00
Jihoon Son 3e60c9125d Increase timeout of GroupByQueryMergeBufferTest and AppenderatorDriverTest (#4441) 2017-06-22 06:09:52 -07:00
Goh Wei Xiang f68a0693f3 Allow use of non-threadsafe ObjectCachingColumnSelectorFactory (#4397)
* Adding a flag to indicate when ObjectCachingColumnSelectorFactory need not be threadsafe.

* - Use of computeIfAbsent over putIfAbsent
- Replace Maps.newXXXMap() with normal instantiation
- Documentations on when is thread-safe required.
- Use Builders for On/OffheapIncrementalIndex

* - Optimization on computeIfAbsent
- Constant EMPTY DimensionsSpec
- Improvement on IncrementalIndexSchema.Builder
  - Remove setting of default values
  - Use var args for metrics
- Correction on On/OffheapIncrementalIndex Builders
- Combine On/OffheapIncrementalIndex Builders

* - Removing unused imports.

* - Helper method for testing with IncrementalIndex.Builder

* - Correction on javadoc.

* Style fix
2017-06-16 16:04:19 -05:00
Gian Merlino e78d8584a1 JettyQosTest, DruidAvaticaHandlerTest: Extend timeout. (#4416)
Fixes #4408, probably.
2017-06-15 18:28:50 -07:00
Jihoon Son bf537f9226 Increase publish timeout (#4407) 2017-06-15 14:55:30 -05:00
Roman Leventov 976492c186 Make PolyBind to fail if property value is not found (fixes #4369) (#4374)
* Make PolyBind to fail if property value is not found

* Fix test

* Add onHeap option in NamespaceExtractionModule

* Add PolyBind.createChoiceWithDefaultNoScope()

* Fix NPE

* Fix

* Configure MetadataStorageProvider option for MySQL, PostgreSQL and SQLServer

* Deprecate PolyBind.createChoiceWithDefault form with unused defaultKey

* Fix NPE
2017-06-13 09:45:43 -07:00
Jihoon Son 073695d311 Simple improvement for realtime manager (#4377)
* Simple improvement for realtime manager

* Address comments

* tmp

* Address comments and add more tests

* Add catch for InterruptedException

* Address comments
2017-06-12 15:25:34 -07:00
Roman Leventov c121845102 Avoid using Guava in DataSegmentPushers because of incompatibilities (#4391)
* Avoid using Guava in DataSegmentPushers because of Hadoop incompatibilities

* Clarify comments
2017-06-12 09:58:34 -07:00
Roman Leventov 5285eb961b Update dependencies (#4313)
* Update dependencies

* Downgrade curator

* Rollback aws-java-sdk dependency to 1.10.77

* Revert exclusions in integration-tests

* Depend only on aws-java-sdk-ec2 instead of umbrella aws-java-sdk (fixes #4382)
2017-06-09 14:32:07 -07:00
Akash Dwivedi f22e646168 Use MetadataSegmentManager info during segment move. (#4260)
* Use MetadataSegmentManager info to move segments.

* Use original segment to drop.

* Formatting.

* Addressed comments.
2017-06-08 12:12:21 -07:00
Gian Merlino 1f2afccdf8 Expressions: Add ExprMacros. (#4365)
* Expressions: Add ExprMacros, which have the same syntax as functions, but
can convert themselves to any kind of Expr at parse-time.

ExprMacroTable is an extension point for adding new ExprMacros. Anything
that might need to parse expressions needs an ExprMacroTable, which can
be injected through Guice.

* Address code review comments.
2017-06-08 09:32:10 -04:00
Roman Leventov 63a897c278 Enable most IntelliJ 'Probable bugs' inspections (#4353)
* Enable most IntelliJ 'Probable bugs' inspections

* Fix in RemoteTestNG

* Fix IndexSpec's equals() and hashCode() to include longEncoding

* Fix inspection errors

* Extract global isntance of natural().nullsFirst(); address comments

* Fix

* Use noinspection comments instead of SuppressWarnings on method for IntelliJ-specific inspections

* Prohibit Ordering.natural().nullsFirst() using Checkstyle
2017-06-07 09:54:25 -07:00
Roman Leventov b487fa355b More methods in QueryMetrics and TopNQueryMetrics (the last part of #3798) (#4284)
* Add more methods to QueryMetrics and TopNQueryMetrics, add BitmapResultFactory

* Add implementor expectations section to BitmapResultFactory javadoc
2017-06-07 09:49:08 -07:00
kaijianding 551a89bd67 serialize DateTime As Long to improve json serde performance (#4038) 2017-06-06 10:08:51 -07:00
Roman Leventov 31d33b333e Make using implicit system Charset an error (#4326)
* Make using implicit system charset an error

* Use StringUtils.toUtf8() and fromUtf8() instead of String.getBytes() and new String()

* Use English locale in StringUtils.safeFormat()

* Restore comment
2017-06-05 23:57:25 -07:00
Charles Allen ada498cbf3 Make the Coordinator Cost Balancer thread naming human friendly (#4354) 2017-06-06 00:17:31 -05:00
Slim a2584d214a Delagate creation of segmentPath/LoadSpec to DataSegmentPushers and add S3a support (#4116)
* Adding s3a schema and s3a implem to hdfs storage module.

* use 2.7.3

* use segment pusher to make loadspec

* move getStorageDir and makeLoad spec under DataSegmentPusher

* fix uts

* fix comment part1

* move to hadoop 2.8

* inject deep storage properties

* set version to 2.7.3

* fix build issue about static class

* fix comments

* fix default hadoop default coordinate

* fix create filesytem

* downgrade aws sdk

* bump the version
2017-06-04 00:55:09 -06:00
Jihoon Son f876246af7 Rename FiniteAppenderatorDriver to AppenderatorDriver (#4356) 2017-06-03 00:48:44 +09:00
kaijianding 0efd18247b explicitly unmap hydrant files when abandonSegment to recycle mmap memory (#4341)
* fix TestKafkaExtractionCluster fail due to port already used

* explicitly unmap hydrant files when abandonSegment to recyle mmap memory

* address the comments

* apply to AppenderatorImpl
2017-06-01 18:15:30 -05:00
Jihoon Son 1150bf7a2c Refactoring Appenderator Driver (#4292)
* Refactoring Appenderator

1) Added publishExecutor and handoffExecutor for background publishing and handing segments off
2) Change add() to not move segments out in it

* Address comments

1) Remove publishTimeout for KafkaIndexTask
2) Simplifying registerHandoff()
3) Add increamental handoff test

* Remove unused variable

* Add persist() to Appenderator and more tests for AppenderatorDriver

* Remove unused imports

* Fix strict build

* Address comments
2017-06-02 07:09:11 +09:00
Roman Leventov 50e72c6aea Fix bugs (core) (#4339)
* Fix bugs

* Add test for GoogleDataSegmentPusher.buildPath()

* Exclude extension changes

* Address comments

* Brace
2017-06-02 06:47:59 +09:00
Roman Leventov 78179ef74d Inject QueryMetrics factories via PolyBind (#4336) 2017-05-31 09:07:03 -07:00
Kenji Noguchi 3400f601db Protobuf extension (#4039)
* move ProtoBufInputRowParser from processing module to protobuf extensions

* Ported PR #3509

* add DynamicMessage

* fix local test stuff that slipped in

* add license header

* removed redundant type name

* removed commented code

* fix code style

* rename ProtoBuf -> Protobuf

* pom.xml: shade protobuf classes, handle .desc resource file as binary file

* clean up error messages

* pick first message type from descriptor if not specified

* fix protoMessageType null check. add test case

* move protobuf-extension from contrib to core

* document: add new configuration keys, and descriptions

* update document. add examples

* move protobuf-extension from contrib to core (2nd try)

* touch

* include protobuf extensions in the distribution

* fix whitespace

* include protobuf example in the distribution

* example: create new pb obj everytime

* document: use properly quoted json

* fix whitespace

* bump parent version to 0.10.1-SNAPSHOT

* ignore Override check

* touch
2017-05-30 13:11:58 -07:00
dgolitsyn 515fabce96 Server selector improvement (#4315)
* Do not re-create prioritized servers on each call in server selector and extend TierSelectorStrategy interface with a method to pick multiple elements at once

* Fix compilation
2017-05-26 11:02:09 -05:00
Jihoon Son 11b7b1bea6 Add support for HttpFirehose (#4297)
* Add support for HttpFirehose

* Fix document

* Add documents
2017-05-25 16:13:04 -05:00
Goh Wei Xiang b77fab8a30 Replace usages of CountingMap with Object2LongMap (#4320)
* Replaces use of CountingMap with Object2LongMap from fastutil.

* Remove CountingMap classes and minor fixes

* Added additional test cases for DatasourceInputFormat.

* Added additional test cases for CoordinatorStats.

* Not materializing segment list.

* Put in this fix because it is failing the test on its expected behavior.

* Added missing header.
2017-05-24 17:40:32 -07:00
Jihoon Son b578adacae Improve concurrency of SegmentManager (#4298)
* Improve concurrency of SegmentManager

* Fix SegmentManager and add more tests

* Add more tests

* Add null check to TimelineEntry

* Remove empty data source and check null in getTimeline()

* Add a comment for returning null in compute()

* Make SegmentManager LazySingleton
2017-05-24 04:41:24 +09:00
Roman Leventov f97c49ba0e Don't use QueryMetrics from multiple threads in DirectDruidClient (fixes #4308) (#4309)
* Don't use QueryMetrics from multiple threads in DirectDruidClient

* reponseMetrics
2017-05-23 10:07:27 -07:00
Jonathan Wei d49e53e6c2 Timeout and maxScatterGatherBytes handling for queries run by Druid SQL (#4305)
* Timeout and maxScatterGatherBytes handling for queries run by Druid SQL

* Address PR comments

* Fix contexts in CalciteQueryTest

* Fix contexts in QuantileSqlAggregatorTest
2017-05-23 16:57:51 +09:00
Jonathan Wei e043bf88ec Add a ServerType for peons (#4295)
* Add a ServerType for peons

* Add toString() method, toString() test, unsupported type check

* Use ServerType enum in DruidServer and DruidServerMetadata
2017-05-22 17:24:59 -05:00
Roman Leventov 8ec3a29af0 Don't pass QueryMetrics down in concurrent and async QueryRunners (fixes #4279) (#4288)
* Don't pass QueryMetrics down in concurrent and async QueryRunners

* Rename QueryPlus.threadSafe() to withoutThreadUnsafeState(); Update QueryPlus.withQueryMetrics() Javadocs; Fix generics in MetricsEmittingQueryRunner and CpuTimeMetricQueryRunner; Make DefaultQueryMetrics to fail fast on modifications from concurrent threads
2017-05-22 13:42:09 -05:00
Gian Merlino adeecc0e72 Add /isLeader call to overlord and coordinator. (#4282)
This is useful for putting them behind load balancers or proxies, as it lets
the load balancer know which server is currently active through an http health
check.

Also makes the method naming a little more consistent between coordinator and
overlord code.
2017-05-18 20:46:13 -05:00
Jihoon Son 5c0a7ad2f8 Make realtimes available for loading segments (#4148)
* Add ServerType

* Add realtimes to DruidCluster

* fix test fails

* Add SegmentManager

* Fix equals and hashCode of ServerHolder

* Address comments and add more tests

* Address comments
2017-05-18 10:03:39 -05:00
Jihoon Son 733dfc9b30 Add PrefetchableTextFilesFirehoseFactory for cloud storage types (#4193)
* Add PrefetcheableTextFilesFirehoseFactory

* fix comment

* exception handling

* Fix wrong json property

* Remove ReplayableFirehoseFactory and fix misspelling

* Defer object initialization

* Add a temporaryDirectory parameter to FirehoseFactory.connect()

* fix when cache and fetch are disabled

* Address comments

* Add more test

* Increase timeout for test

* Add wrapObjectStream

* Move methods to Firehose from PrefetchableFirehoseFactory

* Cleanup comment

* add directory listing to s3 firehose

* Rename a variable

* Addressing comments

* Update document

* Support disabling prefetch

* Fix race condition

* Add fetchLock

* Remove ReplayableFirehoseFactoryTest

* Fix compilation error

* Fix test failure

* Address comments

* Add default implementation for new method
2017-05-18 15:37:18 +09:00
Himanshu daa8ef8658 Optional long-polling based segment announcement via HTTP instead of Zookeeper (#3902)
* Optional long-polling based segment announcement via HTTP instead of Zookeeper

* address review comments

* make endpoint /druid-internal/v1 instead of /druid/internal so that jetty qos filters can be configured easily when needed

* update segment callback initialization to be called only after first segment list fetch has been succeeded from all servers

* address review comments

* remove size check not required anymore as only segment servers announce themselves and not all peon processes

* annouce segment server on historical only after cached segments are loaded

* fix checkstyle errors
2017-05-17 16:31:58 -05:00
Himanshu 0e056863e4 fix timeout check bug in DirectDruidClient (#4287) 2017-05-17 13:47:32 -07:00
Roman Leventov d9f423f55d Make QueryMetrics factories configurable (#4268)
* Ensure QueryMetrics factories accept Json ObjectMapper; Make QueryMetrics factories configurable

* Update QueryMetrics Javadocs

* Add javadocs to QueryMetrics factories

* Move queryMetricsFactory defaults to getter methods of config classes
2017-05-17 08:41:59 -07:00
Gian Merlino 22f20f2207 IngestSegmentFirehoseTest: Add more tests for reindexing. (#4285)
* IngestSegmentFirehoseTest: Add more tests for reindexing.

* Nix unused imports.
2017-05-16 22:12:26 -07:00
Roman Leventov b7a52286e8 Make @Override annotation obligatory (#4274)
* Make MissingOverride an error

* Make travis stript to fail fast

* Add missing Override annotations

* Comment
2017-05-16 13:30:30 -05:00
Himanshu 136b2fae72 improve query timeout handling and limit max scatter-gather bytes (#4229)
* improve query timeout handling and limit max scatter-gather bytes

* address review comments
2017-05-16 12:47:32 -05:00
Benedict Jin e823085866 Improve `collection` related things that reusing a immutable object instead of creating a new object (#4135) 2017-05-17 01:38:51 +09:00
Jihoon Son 50a4ec2b0b Add support for headers and skipping thereof for CSV and TSV (#4254)
* initial commit

* small fixes

* fix bug

* fix bug

* address code review

* more cr

* more cr

* more cr

* fix

* Skip head rows for CSV and TSV

* Move checking skipHeadRows to FileIteratingFirehose

* Remove checking null iterators

* Remove unused imports

* Address comments

* Fix compilation error

* Address comments

* Add more tests

* Add a comment to ReplayableFirehose

* Addressing comments

* Add docs and fix typos
2017-05-15 22:57:31 -07:00
satishbhor 5e6539fec6 Average Server Percent Used: NaN% Error when server startup is in progress (fixes #4214) (#4240)
* Fix lz4 library incompatibility in kafka-indexing-service extension #3266

* Bumped Kafka version to 0.10.2.0 for : Fix lz4 library incompatibility in kafka-indexing-service extension #3266

* Replaced Lists.newArrayList() with Collections.singletonList() For Fix lz4 library incompatibility in kafka-indexing-service extension #4115

* Fixed: Average Server Percent Used: NaN% Error when server startup is in progress #4214
2017-05-12 15:56:17 +09:00
Roman Leventov 1ebfa22955 Update Error prone configuration; Fix bugs (#4252)
* Make Errorprone the default compiler

* Address comments

* Make Error Prone's ClassCanBeStatic rule a error

* Preconditions allow only %s pattern

* Fix DruidCoordinatorBalancerTester

* Try to give the compiler more memory

* Remove distribution module activation on jdk 1.8 because only jdk 1.8 is used now

* Don't show compiler warnings

* Try different travis script

* Fix travis.yml

* Make Error Prone optional again

* For error-prone compiler

* Increase compiler's maxmem

* Don't run Error Prone for benchmarks because of OOM

* Skip install step in Travis

* Remove MetricHolder.writeToChannel()

* In travis.yml, check compilation before tests, because it may fail faster
2017-05-12 15:55:17 +09:00
Roman Leventov e09e892477 Refactor QueryRunner to accept QueryPlus: Query + QueryMetrics (part of #3798) (#4184)
* Add QueryPlus. Add QueryRunner.run(QueryPlus, Map) method with default implementation, to replace QueryRunner.run(Query, Map).

* Fix GroupByMergingQueryRunnerV2

* Fix QueryResourceTest

* Expand the comment to Query.run(walker, context)

* Remove legacy version of BySegmentSkippingQueryRunner.doRun()

* Add LegacyApiQueryRunnerTest and be more specific about legacy API removal plans in Druid 0.11 in Javadocs
2017-05-10 12:25:00 -07:00
David Lim 11538e2ece kafka generated shards were wrongly being marked as overshadowed if they extended a NumberedShardSpec with a non-zero number of total shards (#4257) 2017-05-09 12:19:24 +09:00
Himanshu 417714d228 additional lookup status discovery http endpoints at coordinator (#4228)
* additional lookup status discovery http endpoints at coordinator

* more changes

* jsonize the error msgs as well

* fix tests
2017-05-04 11:15:30 -07:00
Gian Merlino d0f89e969a Ignore misnamed segment cache info files. (#4245)
* Ignore misnamed segment cache info files.

Fixes a bug where historical nodes could announce the same segment twice,
ultimately leading to historicals and their watchers (like coordinators
and brokers) being out of sync about which segments are served.

This could be caused if Druid is switched from local time to UTC, because
that causes the same segment descriptors to lead to different identifiers
(an identifier with local time interval before the switch, and UTC interval
after the switch). In turn this causes that segment descriptor to be written
to multiple segment cache info files and potentially get announced twice.

Later, if the historical receives a drop request, it drops the segment and
unannounces it once, but the other announcement would stick around in an
ephemeral znode forever, confusing coordinators and brokers.

* Only alert once.
2017-05-03 22:02:37 -06:00
Jihoon Son 7411b18df9 Add BroadcastDistributionRule (#4077)
* Add BroadcastDistributionRule

* Add missing null check

* Rename variable 'colocateDataSource' to 'colocatedDatasource'

* Address comments

* Document for broadcast rules

* Drop segments which are not co-located anymore

* Remove duplicated segment loading and dropping

* Add caveat

* address comments
2017-05-01 09:55:17 -07:00
Himanshu 5a5a2749cd improvements to coordinator lookups management (#3855)
* coordinator lookups mgmt improvements

* revert replaces removal, deprecate it instead

* convert and use older specs stored in db

* more tests and updates

* review comments

* add behavior for 0.10.0 to 0.9.2 downgrade

* incorporating more review comments

* remove explicit lock and use LifecycleLock in LookupReferencesManager. use LifecycleLock in LookupCoordinatorManager as well

* wip on LookupCoordinatorManager

* lifecycle lock

* refactor thread creation into utility method

* more review comments addressed

* support smooth roll back of lookup snapshots from 0.10.0 to 0.9.2

* correctly use LifecycleLock in LookupCoordinatorManager and remove synchronization from start/stop

* run lookup mgmt on leader coordinator only

* wip: changes to do multiple start() and stop() on LookupCoordinatorManager

* lifecycleLock fix usage in LookupReferencesManagerTest

* add LifecycleLock back

* fix license hdr

* some fixes

* make LookupReferencesManager.getAllLookupsState() consistent while still being lockless

* address review comments

* addressing leventov's comments

* address charle's comments

* add IOE.java

* for safety in LookupReferencesManager mainThread check for lifecycle started state on each loop in addition to interrupt

* move thread creation utility method to Execs

* fix names

* add tests for LookupCoordinatorManager.lookupManagementLoop()

* add further tests for figuring out toBeLoaded and toBeDropped on LookupCoordinatorManager

* address leventov comments

* remove LookupsStateWithMap and parameterize LookupsState

* address review comments

* address more review comments

* misc fixes
2017-04-28 08:41:38 -05:00
Himanshu 9b9e1cfecb coordinator dynamic config POST to update only explicitly specified fields (#4141)
* coordinator dynamic config POST to update only explicitly specified fields instead of resetting everything else to zeros

* address review comments
2017-04-26 14:59:20 -07:00
David Lim 52f7bb091d suppress warn message if metricsSpec is absent when using no-rollup ingestion (#4211) 2017-04-25 22:52:49 -06:00
Himanshu 4d3745d6c9 log the exception on failure to send query response (#4179) 2017-04-25 10:27:20 -07:00
Gian Merlino 809112cd5f DirectDruidClient: Fix division by zero. (#4206)
* DirectDruidClient: Fix division by zero.

Introduced in #3954 when some floating math was changed to
integer math. This patch restores the old math.

* Added comment.
2017-04-25 13:03:00 +09:00
Gian Merlino 2ca7b00346 Update versions to 0.10.1-SNAPSHOT. (#4191) 2017-04-20 18:12:28 -07:00
Gian Merlino 60caa641f3 Restore backwards compatibility of Query. (#4185) 2017-04-19 19:47:50 +03:00
Jihoon Son 5b69f2eff2 Make timeout behavior consistent to document (#4134)
* Make timeout behavior consistent to document

* Refactoring BlockingPool and add more methods to QueryContexts

* remove unused imports

* Addressed comments

* Address comments

* remove unused method

* Make default query timeout configurable

* Fix test failure

* Change timeout from period to millis
2017-04-19 09:47:53 +09:00
Himanshu 8432138a07 log query in requestLog when client has disconnected and response write fails (#4169) 2017-04-15 04:04:52 +05:30
Himanshu 5c9198347b fix over counting of bytes in ByteCountingLRUMap (#4168) 2017-04-14 07:06:34 +09:00
Roman Leventov 15f3a94474 Copy closer into Druid codebase (fixes #3652) (#4153) 2017-04-10 09:38:45 +09:00
Parag Jain 7e0d4c9555 secure supervisor endpoints (#3985) 2017-04-05 16:42:32 -07:00
Jihoon Son b166e13d2b Make DruidCoordinatorTest fail fast (#4142) 2017-04-04 13:58:32 -07:00
Roman Leventov 81ba0224c5 Report unsuccessful queries in AsyncQueryForwardingServlet (#4143) 2017-04-04 13:53:07 -07:00
Himanshu de081c711b RealtimeIndexTask to support alertTimeout in context (#4089)
* RealtimeIndexTask to support alertTimeout in context and raise alert if task process exists after the timeout

* move alertTimeout config to tuningConfig and document
2017-03-24 12:48:12 -07:00
Gian Merlino b4289c0004 Remove "granularity" from IngestSegmentFirehose. (#4110)
It wasn't doing anything useful (the sequences were being concatted, and
cursor.getTime() wasn't being called) and it defaulted to Granularities.NONE.
Changing it to Granularities.ALL gave me a 700x+ performance boost on a
small dataset I was reindexing (2m27s to 365ms). Most of that was from avoiding
making a lot of unnecessary column selectors.
2017-03-24 10:28:54 -07:00
Roman Leventov 4b5ae31207 QueryMetrics: abstraction layer of query metrics emitting (part of #3798) (#3954)
* QueryMetrics: abstraction layer of query metrics emitting

* Minor fixes

* QueryMetrics.emit() for bulk emit and improve Javadoc

* Fixes

* Fix

* Javadoc fixes

* Typo

* Use DefaultObjectMapper

* Add tests

* Address PR comments

* Remove QueryMetrics.userDimensions(); Rename QueryMetric.register() to report()

* Dedicated TopNQueryMetricsFactory, GroupByQueryMetricsFactory and TimeseriesQueryMetricsFactory

* Typo

* More elaborate Javadoc of QueryMetrics

* Formatting

* Replace QueryMetric enum with lambdas

* Add comments and VisibleForTesting annotations
2017-03-23 17:23:59 -07:00
Charles Allen 805d85afda Allow compilation as Java8 source and target (#3328)
* Allow compilation as Java8 source and target for everything except API

* Remove conditions in tests which assume that we may run with Java 7

* Update easymock to 3.4

* Make Animal Sniffer to check Java 1.8 usage; remove redundant druid-caffeine-cache configuration

* Use try-with-resources in LargeColumnSupportedComplexColumnSerializerTest.testSanity()

* Remove java7 special for druid-api
2017-03-14 22:23:47 -06:00
Gian Merlino df623ebfe3 Fix a couple bugs due to calling Period.getMillis(). (#4006) 2017-03-05 18:44:20 +05:30
Roman Leventov 81a5f9851f TmpFileIOPeons to create files under the merging output directory, instead of java.io.tmpdir (#3990)
* In IndexMerger and IndexMergerV9, create temporary files under the output directory/tmpPeonFiles, instead of java.io.tmpdir

* Use FileUtils.forceMkdir() across the codebase and remove some unused code

* Fix test

* Fix PullDependencies.run()

* Unused import
2017-03-02 14:05:12 -08:00
Jonathan Wei 5fb1638534 Add default configuration for select query 'fromNext' parameter (#3986)
* Add default configuration for select query 'fromNext' parameter

* PR comments

* Fix PagingSpec config injection

* Injection fix for test
2017-03-01 17:05:35 -08:00
praveev 5ccfdcc48b Fix testDeadlock timeout delay (#3979)
* No more singleton. Reduce iterations

* Granularities

* Fix the delay in the test

* Add license header

* Remove unused imports

* Lot more unused imports from all the rearranging

* CR feedback

* Move javadoc to constructor
2017-02-28 12:51:41 -06:00
praveev c3bf40108d One granularity (#3850)
* Refactor Segment Granularity

* Beginning of one granularity

* Copy the fix for custom periods in segment-grunalrity over here.

* Remove the custom serialization for now.

* Compilation cleanup

* Reformat code

* Fixing unit tests

* Unify to use a single iterable

* Backward compatibility for rolling upgrade

* Minor check style. Cosmetic changes.

* Rename length and millis to duration

* CR feedback

* Minor changes.
2017-02-25 01:02:29 -06:00
Niketh Sabbineni 02fc625b5f Fix bug with SegmentLoaderLocalCacheManager (#3929)
* Fix bug with SegmentLoaderLocalCacheManager

* Use collections.sort
2017-02-23 12:31:38 -08:00
Himanshu 2ead572639 make QueryResource return ETag header if If-None-Match header is provided (#3955)
also do not process query and return HTTP 304 NOT MODIFIED if given If-None-Match value matches current ETag
2017-02-23 13:39:59 -06:00
Jihoon Son ebd100cbb0 Set default query granularity for null value (#3965) 2017-02-22 17:38:43 -08:00
Gian Merlino f910c050fd CachingClusteredClientTest: Use groupBy v1. (#3958)
GroupBy v2 doesn't cache on the broker, so it isn't actually testing
what the test was supposed to be testing. Also, the test failed due
to mismatched expectations.
2017-02-21 16:53:34 -06:00
Jihoon Son 128274c6f0 Disable caching on brokers for groupBy v2 (#3950)
* Disable caching on brokers for groupBy v2

* Rename parameter

* address comments
2017-02-21 09:49:49 -08:00
kaijianding a029b33499 fix cache populate incorrect content when numBackgroundThreads>1 (#3943)
* fix cache populate incorrect content when numBackgroundThreads>1

* simplify code by using Futures.allAsList and use CountDownLatch in UT

* fix test code style and assert countDownLatch.await()
2017-02-17 20:47:19 +05:30
Gian Merlino 78b0d134ae Require Java 8 and include some Java 8 dependencies. (#3914)
* Require Java 8 and include some Java 8 dependencies.

- Upgrade Jetty to 9.3.16.v20170120.
- Upgrade DataSketches to 0.8.4.
- Bundle caffeine-cache by default.
- Still target Java 7 when compiling base Druid classes.

* Update cluster, quickstart docs.

* Remove oraclejdk7 from travis.yml.
2017-02-14 12:51:51 -08:00
Jakub Kukul 28d85702ad Fix rolling of request log files. (#3916)
* Use common date format for request log files.

* Remove code duplication in creating logging FileWriter.
2017-02-14 09:33:43 -08:00
Akash Dwivedi 8854ce018e File.deleteOnExit() (#3923)
* Less use of File.deleteOnExit()
 * removed deleteOnExit from most of the tests/benchmarks/iopeon
 * Made IOpeon closable

* Formatting.

* Revert DeterminePartitionsJobTest, remove cleanup method from IOPeon
2017-02-13 15:12:14 -08:00
Himanshu 9dfcf0763a disable javascript execution by default (#3818) 2017-02-13 15:11:18 -08:00
Himanshu 8cf7ad1e3a druid.coordinator.asOverlord.enabled flag at coordinator to make it an overlord too (#3711) 2017-02-13 15:03:59 -08:00
michaelschiff c1eee9bbf3 modified "end" column to `end` (#3903)
* modified "end" column to `end`.  "end" is interpretted as a string rather than dereferencing the column value

* SQLMetadataConnector.getQuoteString defines the string that should be used to quote string fields

* positional arguments for String.format

* for Connectors that use " need to include the \ escape as well
2017-02-13 12:36:27 -08:00
Jihoon Son 991e2852da Add PostAggregators to generator cache keys for top-n queries (#3899)
* Add PostAggregators to generator cache keys for top-n queries

* Add tests for strings

* Remove debug comments

* Add type keys and list sizes to cache key

* Make post aggregators used for sort are considered for cache key generation

* Use assertArrayEquals()

* Improve findPostAggregatorsForSort()

* Address comments

* fix test failure

* address comments
2017-02-13 12:23:44 -08:00
Parag Jain 8e31a465ad report hand off count finite appenderator driver (#3925) 2017-02-13 10:41:24 -08:00
Jonathan Wei ca2b04f0fd Add long/float ColumnSelectorStrategy implementations (#3838)
* Add long/float ColumnSelectorStrategy implementations

* Address PR comments

* Add String strategy with internal dictionary to V2 groupby, remove dict from numeric wrapping selectors, more tests

* PR comments

* Use BaseSingleValueDimensionSelector for long/float wrapping

* remove unused import

* Address PR comments

* PR comments

* PR comments

* More PR comments

* Fix failing calcite histogram subquery tests

* ScanQuery test and comment about isInputRaw

* Add outputType to extractionDimensionSpec, tweak SQL tests

* Fix limit spec optimization for numerics

* Add cardinality sanity checks to TopN

* Fix import from merge

* Add tests for filtered dimension spec outputType

* Address PR comments

* Allow filtered dimspecs on numerics

* More comments
2017-02-08 20:39:29 -08:00
Himanshu e08cd0066b verify no duplicate aggregator names in DataSchema (#3917) 2017-02-08 16:12:07 -08:00