Commit Graph

2996 Commits

Author SHA1 Message Date
Jihoon Son d95915f8d2 Implement get methods for PrefetchableFirehose (#4948) 2017-10-12 16:14:45 +09:00
Jihoon Son dfa9cdc982 Prioritized locking (#4550)
* Implementation of prioritized locking

* Fix build failure

* Fix tc fail

* Fix typos

* Fix IndexTaskTest

* Addressed comments

* Fix test

* Fix spacing

* Fix build error

* Fix build error

* Add lock status

* Cleanup suspicious method

* Add nullables

*  add doInCriticalSection to TaskLockBox and revert return type of task actions

* fix build

* refactor CriticalAction

* make replaceLock transactional

* fix formatting

* fix javadoc

* fix build
2017-10-11 23:16:31 -07:00
Roman Leventov 7a9940d624 Add /readiness to HistoricalResource (#4916)
* Add /loadStatusCode to HistoricalResource

* Address comments

* Fixes
2017-10-11 20:35:52 -07:00
Jihoon Son 56fb11ce0b Lazy initialization for JavaScript functions (#4871)
* Lazy initialization of JavaScript functions

* Fix test failure

* Fix thread-safety and postpone js conf check

* Fix test fail

* Fix test

* Fix KafkaIndexTaskTest

* Move config check
2017-10-10 21:52:42 -07:00
Roman Leventov e725ff4146 1-based counts in ZkCoordinator (#4917) 2017-10-10 13:00:51 -07:00
Kevin Conaway 1bc4b71a34 Reduce Chance of Duplicates in EventReceiverFireHose (#4903)
* Add ability to optionally specify a sequence identifier to reduce the possibility of duplicate events entering the event receiver firehose

* Add ability to optionally specify a sequence identifier to reduce the possibility of duplicate events entering the event receiver firehose

* Add a hard coded limit to the maximum number of possible producer IDs to prevent a malicious (or uninformed) client from overflowing the heap
2017-10-10 11:17:17 -07:00
Parag Jain 7cc18226cd add more tls configs to enable/disable specific cipher suites and protocols (#4902)
* add more tls configs to enable/disable specific cipher suites and protocols

* fix doc, allow empty list
2017-10-09 13:53:12 -07:00
Gian Merlino 797b54d283 DruidLeaderClient: Throw IOException on retryable errors. (#4913)
* DruidLeaderClient: Throw IOException on retryable errors.

Fixes #4911.

* Adjustments.
2017-10-06 15:12:09 -05:00
Himanshu 0e856ee806 add configs to enable fast request failure on broker and historical (#4540)
* add configs to enable fast request failure on broker

* address review comments

* fix styling error

* fix style error

* have enableRequestLimit config instead of having user specify max limit

* add comment

* fix style error

* add UT fo LimitRequestsFilter

* address review comments

* fix test

* make LimitRequestsFilterTest more robust

* fix JettyQosTest
2017-10-06 14:45:13 -05:00
praveev 4ff12e4394 Hadoop indexing: Fix NPE when intervals not provided (#4686)
* Fix #4647

* NPE protect bucketInterval as well

* Add test to verify timezone as well

* Also handle case when intervals are already present

* Fix checkstyle error

* Use factory method instead for Datetime

* Use Intervals factory method
2017-10-05 22:46:07 -07:00
Akash Dwivedi 2ee32399ff granularity method in QueryMetrics. (#4570)
* granularity method in QueryMetrics.

PR to emit granularity dimension for timeseries, search, groupBy,
select and topN queries.

* QueryMetricsFactory classes for search and select queries.

* Empty implementation  for  Granularity() method.

* Review comment changes.

* Remove unused import.

* empty query() method.

* checkstyle fix.

* Import fix.
2017-10-04 09:42:52 -07:00
Jonathan Wei 07aa405a6f Fix PreResponseAuthorizationCheckFilter HTTP error masking (#4900)
* Fix PreResponseAuthorizationCheckFilter HTTP error masking

* Add remote addr and host to missing auth check log message
2017-10-03 16:58:57 -05:00
Jonathan Wei 5e60ccade1 Add context map to AuthenticationResult (#4870) 2017-10-02 17:08:14 -05:00
Jonathan Wei 9deab26d8b Fix auth check in InventoryViewUtils (#4869) 2017-10-02 11:38:45 -07:00
Niketh Sabbineni 3e9391433d Coord resource throws NPE when segments are requested (#4759) 2017-10-02 10:13:27 -07:00
Gian Merlino 1f2074c247 Bump versions in master to 0.11.1-SNAPSHOT. (#4878)
* Bump versions in master to 0.11.1-SNAPSHOT.

* Missed a few.
2017-09-28 17:09:51 -05:00
Goh Wei Xiang 26fd2b3a8e Priority on loading for primary replica (#4757)
* Priority on loading for primary replica

* Simplicity fixes

* Fix on skipping drop for quick return.

* change to debug logging for no replicants.

* Fix on filter logic

* swapping if-else

* Fix on wrong "hasTier" logic

* Refactoring of LoadRule

* Rename createPredicate to createLoadQueueSizeLimitingPredicate

* Rename getHolderList to getFilteredHolders

* remove varargs

* extract out currentReplicantsInTier

* rename holders to holdersInTier

* don't do temporary removal of tier.

* rename primaryTier to tierToSkip

* change LinkedList to ArrayList

* Change MinMaxPriorityQueue in DruidCluster to TreeSet.

* Adding some comments.

* Modify log messages in light of predicates.

* Add in-method comments

* Don't create new Object2IntOpenHashMap for each run() call.

* Cache result from strategy call in the primary assignment to be reused during the same run.

* Spelling mistake

* Cleaning up javadoc.

* refactor out loading in progress check.

* Removed redundant comment.

* Removed forbidden API

* Correct non-forbidden API.

* Precision in variable type for NavigableSet.

* Obsolete comment.

* Clarity in method call and moving retrieval of ServerHolder into method call.

* Comment on mutability of CoordinatoorStats.

* Added auxiliary fixture for dropping.
2017-09-28 13:02:05 -07:00
Gian Merlino a19f22b5bb Add identity to query metrics, logs. (#4862)
* Add identity to query metrics, logs.

Also fix a bug where unauthorized requests would not emit any logs or metrics,
and instead would log a "Tried to emit logs and metrics twice" warning.

Also rename QueryResource's "getServer" to "cancelQuery", because that's what
it does.

* Do not emit identity by default.
2017-09-28 11:45:23 -07:00
Himanshu f69c9280c4 remove ServerConfig from DruidNode as all information needs to be present in DruidNode serialized form (#4858)
* remove ServerConfig from DruidNode as all information needs to be present in DruidNode serialized form

* sanitize output of /druid/coordinator/v1/cluster endpoint
2017-09-28 10:40:59 -05:00
Goh Wei Xiang 2c30d5ba55 Add org.joda.time.DateTime.parse() to forbidden APIs (#4857)
* Added org.joda.time.DateTime#(java.lang.String) to forbidden API.

* Added org.joda.time.DateTime#(java.lang.String, org.joda.time.format.DateTimeFormatter) to forbidden API.

* Add additional APIs that may create DateTime with default time zone

* Add helper function that accepts formatter to parse String.

* Add additional forbidden APIs

* Replace existing usage of forbidden APIs

* Use wrapper class to enforce Chronology on DateTimeFormatter.

* Creates constant UtcFormatter for constant ISODateTimeFormat.
2017-09-27 17:46:44 -05:00
Gian Merlino 999c6d800e Fix Router handling of SQL queries. (#4851) 2017-09-27 10:58:24 -07:00
Roman Leventov 9c126e2aa9 Forbid MapMaker (#4845)
* Forbid MapMaker

* Shorter syntax

* Forbid Maps.newConcurrentMap()
2017-09-27 06:49:47 -07:00
Charles Allen a6470c1d03 Move caffeine out of extension and make it the default cache implementation. (#4810)
* Move caffeine out of extension.

* Remove `JsonTypeName` from the class itself

* Fix bad docs

* Fix distribution pom

* Fix unused import

* Make caffeine default

* Address code comments

* Add more description around the jre version in the readme

* Add suggested comments
2017-09-22 10:46:55 -07:00
Jonathan Wei 09fcb75583 Add RequestLogEvent emitters config to graphite-emitter (#4678)
* Add RequestLogEvent emitters config to graphite-emitter

* eagerly compute emitter list

* use lambdas

* checkstyle
2017-09-22 06:14:32 -07:00
Roman Leventov e267f3901b Enforce Indentation with Checkstyle (#4799) 2017-09-21 13:06:48 -07:00
Roman Leventov d8b3bfa63c Remove SegmentLoaderConfig.numLoadingThreads config (#4829) 2017-09-20 21:27:43 -07:00
Charles Allen 47ebc48059 Use java 8 features in TierSelectorStrategy implementations (#4827)
* Use java 8 features in TierSelectorStrategy implementations
* Minor code cleanup

* More java8 coolness

* Code comments
2017-09-20 22:09:29 -05:00
Roman Leventov 88e9a80636 Rename ObjectValueSelector.get() to getObject(); Add getObject() and classOfObject() to ColumnValueSelector (#4801) 2017-09-19 14:47:20 -05:00
Jonathan Wei 3a4a483bb0 Single auth check for authorized resource filtering (#4818)
* Single auth check for authorized resource filtering

* PR comment

* PR comments
2017-09-19 21:46:08 +05:30
Jonathan Wei c2a0e753b6 Extension points for authentication/authorization (#4271)
* Extension points for authentication/authorization

* Address some PR comments

* Authorization result caching

* Add unit tests for SecuritySanityCheckFilter and PreResponseAuthorizationCheckFilter

* Use Set for auth caching, close outputstreams in filters

* Don't close output stream on success in sanity check filter

* Add ConfigResourceFilter to coordinator lookups

* Fix filtering authorization check for empty resource list

* HttpClient users must explicitly escalate the client

* Remove response modification from PreResponseAuthorizationCheckFilter

* Remove extraneous pom.xml

* Fix unit test

* Better lifecycle management

* Rename AuthorizationManager to Authorizer

* Fix authorization denials for empty supervisor list

* Address some PR comments

* Address more PR comments

* Small cleanup

* Add Jetty HttpClient wrapper to Authenticator

* Remove Authorizer start/stop

* Restore immutable context map in DruidConnection, UT fix

* Fix/update docs

* Add authorization checks to EventReceiverFirehose

* Fix router authorization check failure, restore PreResponseAuthorizationFilter changes

* Compile fixes

* Test fixes

* Update Authenticator/Authorizer doc comments

* Merge fixes

* PR comments

* Fix test

* Fix IT

* More PR comments

* PR comments

* SSL fix
2017-09-15 23:45:48 -07:00
Himanshu d37be5e6e9 don't hold thread while waiting after failure from server (#4795) 2017-09-14 17:19:25 -05:00
Akash Dwivedi a17e48fe69 search package name correction. (#4785)
* search package name correction.

* Refactor search.search pkg to search.

* remove unused import.
2017-09-14 13:50:23 -07:00
Roman Leventov 267f415dc3 Update emitter library and add support for ParametrizedUriEmitter (#4722)
* Move emitters from io.druid.server.initialization to the dedicated io.druid.server.emitter package; Update emitter library to 0.6.0; Add support for ParametrizedUriEmitter; Support hierarical properties in JsonConfigurator (was needed for ParametrizedUriEmitter)

* Log created RequestLoggers

* Fix forbidden API

* Test fix

* More Http and Parametrized Http Emitter docs

* Switch to debug level
2017-09-13 17:17:19 -05:00
Himanshu 7919469de6 fixes HttpServerInventoryView to call server/segment callbacks correctly and Unit Tests for the class (#4767)
* fixes HttpServerInventoryView to call server/segment callbacks correctly and Unit Tests for the class

* fix checkstyle and forbidden-api errors

* HttpServerInventoryView to finish start() only after server inventory is initialized

* fix compilation errors

* address review comments

* add exponential backoff instead of fixed 5 secs on successive failures

* update test to exercise server fail scenarios

* use AtomicInteger for requestNum and increment only once
2017-09-13 14:24:19 -05:00
Gian Merlino 2ce8123bdb Move scan-query from a contrib extension into core. (#4751)
* Move scan-query from a contrib extension into core.

Based on a proposal at: https://groups.google.com/d/topic/druid-development/ME_OatUDnbk/discussion

This patch also adds support for virtual columns to the Scan query,
and updates Druid SQL to use Scan instead of Select.

This patch also makes some behavioral changes to handling of the __time
column. In particular, it is now is returned as "__time" rather than
"timestamp"; it is no longer included if you do not specifically ask for
it in your "columns"; and it is returned as a long rather than a string.

Users can revert time handling to the legacy extension behavior by
setting "legacy" : true in their queries, or setting the property
druid.query.scan.legacy = true. This is meant to provide a migration
path for users that were formerly using the contrib extension.

* Adjustments from review.

* Add back Select query.

* Adjust SQL docs.

* Restore SelectQuery link.
2017-09-13 09:51:24 -07:00
Roman Leventov 832cc293ef Refactoring of ReferenceCountingSegment and FireHydrant (#4154)
* Refactoring of ReferenceCountingSegment and FireHydrant

* Address comment

* Fix FireHydrant.closeSegment()

* Address comment

* Added comments to ReferenceCountingSegment
2017-09-12 14:28:35 -05:00
Gian Merlino 4909c48b0c SQL: Full TRIM support. (#4750)
* SQL: Full TRIM support.

- Support trimming arbitrary characters
- Support BOTH, LEADING, and TRAILING

* Remove unused import.

* Fix tests, add RTRIM / LTRIM.

* Remove unused imports.

* BTRIM and docs.

* Replace for with foreach.
2017-09-12 11:49:08 -07:00
Parag Jain b5e839b3db injectable sslcontextfactory for jetty server and key manager factory algorithm (#4769)
* injectable sslcontextfactory for jetty server

key manager factory algorithm

* explicitly set trustAll certificates to false in sslcontextfactory
2017-09-12 11:45:03 -07:00
Himanshu 834e050bc4 Use internal-discovery and http for talking to overlord/coordinator leaders (#4735)
* Use internal-discovery and http for talking to overlord/coordinator leaders

* CuratorDruidNodeDiscovery.getAllNodes() best effort 30 sec wait for cache initialization

* DruidLeaderClientProvider to eagerly instantiate DruidNodeDiscovery when needed so that DruidNodeDiscovery impl cache gets initialized well in time

* Revert "DruidLeaderClientProvider to eagerly instantiate DruidNodeDiscovery when needed so that DruidNodeDiscovery impl cache gets initialized well in time"

This reverts commit f1a2432614ba56ddc2d55fe47e990d17fcfd6129.

* add lifecycle to DruidLeaderClient to early initialize DruidNodeDiscovery so that it has its cache update well in time
2017-09-11 11:18:01 -07:00
Niketh Sabbineni 975a50fc81 Generate UUID if query id is empty (#4772) 2017-09-11 10:40:26 -07:00
dgolitsyn 752151f6cb Add CachingCostBalancerStrategy (#4731)
* Add CachingCostBalancerStrategy; Rename ServerView.ServerCallback to ServerRemovedCallback

* Fix benchmark units

* Style, forbidden-api, review, bug fixes

* Add docs

* Address comments
2017-09-08 12:23:04 -05:00
Himanshu d43687d578 increase test timeout to 15sec as leader election jitter is random sleep between 1 to 5 secs (#4756) 2017-09-06 09:56:25 -07:00
Kevin Conaway 83e94e8b31 Expose version property for `CustomVersioningPolicy` so that it may serialize/deserialize correctly (#4747)
Expose version property for  so that it may serialize/deserialize correctly

Expose version property for `CustomVersioningPolicy` so that it may serialize/deserialize correctly

Expose version property for CustomVersioningPolicy so that it may serialize/deserialize correctly

Expose version property for `CustomVersioningPolicy` so that it may serialize/deserialize correctly
2017-09-03 13:36:18 -07:00
Himanshu 06ac6678e6 DruidLeaderSelector interface for leader election and Curator based impl. (#4699)
* DruidLeaderSelector interface for leader election and Curator based impl. DruidCoordinator/TaskMaster are updated to use the new interface.

* add fake DruidNode binding in integration-tests module

* add docs on DruidLeaderSelector interface

* remove start/stop and keep register/unregister Listener in DruidLeaderSelector interface

* updated comments on DruidLeaderSelector

* cache the listener executor in CuratorDruidLeaderSelector

* use same latch owner name that was used before

* remove stuff related to druid.zk.paths.indexer.leaderLatchPath config

* randomize the delay when giving up leadership and restarting leader latch
2017-09-01 09:49:04 -07:00
Charles Allen bdfc6fe25e Move common TypeReference into JacksonUtils (#4738) 2017-08-31 13:40:16 -07:00
Roman Leventov 4d109a358a Refactoring of Storage Adapters (#4710)
* Factor QueryableIndexColumnSelectorFactory and IncrementalIndexColumnSelectorFactory out of QueryableIndexStorageAdapter and IncrementalIndexStorageAdapter; Add Offset.getBaseReadableOffset(); Remove OffsetHolder interface; Replace Cursor extends ColumnSelectorFactory with composition; Reduce indirection in ColumnValueSelectors created by QueryableIndexColumnSelectorFactory

* Don't override clone() in FilteredOffset (the prev. implementation was broken); Some warnings fixed

* Simplify Cursors in QueryableIndexStorageAdapter

* Address comments

* Remove unused and unimplemented methods from GenericColumn interface

* Comments
2017-08-28 18:07:31 -07:00
Parag Jain 594a66f3c0 add scheme to AsyncQueryForwardingServlet (#4688)
* add scheme to AsyncQueryForwardingServlet

* add sslContext binding for Router
2017-08-28 15:03:43 -07:00
Gian Merlino 9fbfc1be32 Add @ExtensionPoint and @PublicApi annotations. (#4433)
* Add @ExtensionPoint and @PublicApi annotations.

* Clean up wording.

* Remove unused import.

* Remove unused imports.

* Only types can be extension points.

* Adjust annotations some more.

* Remove unused import.

* Make ServletFilterHolder an extension point.

* Add a couple extension points, and update docs.
2017-08-28 14:50:58 -07:00
dgolitsyn fd0f349c68 Mark segments that are going to be dropped from the server and use it in DruidBalancer (#4486)
* Do not remove segment that should not be moved from currentlyMovingSegments (segments are removed by callbacks or not inserted)

* Mark segments that are going to be dropped from server and use this information in CostBalancerStrategy

* Fix tests
2017-08-25 18:25:56 -05:00
Himanshu 74538c3288 update internal-discovery Listener for node list and use same at router and coordinator (#4697)
* update LookupCoordinatorManager to use internal discovery to discover lookup nodes

* router:use internal-discovery to discover brokers

* minor [Curator]DruidDiscoveryProvider refactoring

* add initialized() method to DruidNodeDiscovery.Listener

* update HttpServerInventoryView to use initialized() and call segment callback initialization asynchronously

* Revert "update HttpServerInventoryView to use initialized() and call segment callback initialization asynchronously"

This reverts commit f796e441221fe8b0e9df87fdec6c9f47bcedd890.

* Revert "add initialized() method to DruidNodeDiscovery.Listener"

This reverts commit f0661541d073683f28fce2dd4f30ec37db90deb0.

* minor refactoring removing synchronized from DruidNodeDiscoveryProvider

* updated DruidNodeDiscovery.Listener contract to take List of nodes and first call marks initialization

* update HttpServerInventoryView to handle new contract and handle initialization

* router update to handle updated listener contract

* document DruidNodeDiscovery.Listener contract

* fix forbidden-api error

* change log level to info for unknown path children cache events in CuratorDruidNodeDiscoveryProvider

* announce broker only after segment inventory is initialized
2017-08-25 09:28:15 -07:00