Commit Graph

8159 Commits

Author SHA1 Message Date
Himanshu a36adc63e4 [documentation] add more jvm and os guidelines (#4793)
* add more jvm and os guidelines

* address review comments

* add not so general guidelines too

* duplicate statement removal
2017-09-20 13:12:57 -07:00
Roman Leventov 88e9a80636 Rename ObjectValueSelector.get() to getObject(); Add getObject() and classOfObject() to ColumnValueSelector (#4801) 2017-09-19 14:47:20 -05:00
Jonathan Wei 164c73f2b2 Fix kerberos authenticator docs (#4822) 2017-09-19 14:32:22 -05:00
Roman Leventov 24646ac76a LZ4 decompression forward compatibility (#4824) 2017-09-19 10:18:37 -07:00
Jonathan Wei 3a4a483bb0 Single auth check for authorized resource filtering (#4818)
* Single auth check for authorized resource filtering

* PR comment

* PR comments
2017-09-19 21:46:08 +05:30
Charles Allen 00d39ce7a5 Move checks for bitmap size == 0 to isEmpty (#4820) 2017-09-19 21:45:16 +05:30
Charles Allen edd9c76fa5 Add profile for building for use with Spark 2.x (#4808)
* Add profile for building for use with Spark 2.x

* Update aws sdk version
2017-09-18 23:39:40 -05:00
Charles Allen e38705e348 Add timing to log for URI based Lookup fetching (#4805)
* Add timing to log for URI based metrics

* Reformat
2017-09-18 11:18:32 -05:00
Gian Merlino 96612cc665 Fix incorrect log formatting in DruidKerberosAuthenticationHandler. (#4817) 2017-09-17 22:41:36 -07:00
Jonathan Wei c2a0e753b6 Extension points for authentication/authorization (#4271)
* Extension points for authentication/authorization

* Address some PR comments

* Authorization result caching

* Add unit tests for SecuritySanityCheckFilter and PreResponseAuthorizationCheckFilter

* Use Set for auth caching, close outputstreams in filters

* Don't close output stream on success in sanity check filter

* Add ConfigResourceFilter to coordinator lookups

* Fix filtering authorization check for empty resource list

* HttpClient users must explicitly escalate the client

* Remove response modification from PreResponseAuthorizationCheckFilter

* Remove extraneous pom.xml

* Fix unit test

* Better lifecycle management

* Rename AuthorizationManager to Authorizer

* Fix authorization denials for empty supervisor list

* Address some PR comments

* Address more PR comments

* Small cleanup

* Add Jetty HttpClient wrapper to Authenticator

* Remove Authorizer start/stop

* Restore immutable context map in DruidConnection, UT fix

* Fix/update docs

* Add authorization checks to EventReceiverFirehose

* Fix router authorization check failure, restore PreResponseAuthorizationFilter changes

* Compile fixes

* Test fixes

* Update Authenticator/Authorizer doc comments

* Merge fixes

* PR comments

* Fix test

* Fix IT

* More PR comments

* PR comments

* SSL fix
2017-09-15 23:45:48 -07:00
Roman Leventov 3f92184dd8 Inspection fixes (#4809) 2017-09-15 17:48:29 -07:00
Yuya Fujiwara 0fe734805b formatted table. (#4797) 2017-09-15 17:39:06 -07:00
Jihoon Son d606bd72de Upgrade curator (#4786) 2017-09-15 10:48:32 -07:00
Himanshu d37be5e6e9 don't hold thread while waiting after failure from server (#4795) 2017-09-14 17:19:25 -05:00
Roman Leventov b61248fdb1 Replace HistoricalFloatColumnSelector with more generic HistoricalColumnSelector (#4796) 2017-09-14 13:52:06 -07:00
Akash Dwivedi a17e48fe69 search package name correction. (#4785)
* search package name correction.

* Refactor search.search pkg to search.

* remove unused import.
2017-09-14 13:50:23 -07:00
Roman Leventov cd5de123bd Self-checking S3DataSegmentMover.safeMove() (#4725)
* Self-checking S3DataSegmentMover.safeMove()

* Remove unused in S3DataSegmentMoverTest

* Address comments

* More specific excpetions

* Remove delete check
2017-09-14 13:49:21 -07:00
Roman Leventov 267f415dc3 Update emitter library and add support for ParametrizedUriEmitter (#4722)
* Move emitters from io.druid.server.initialization to the dedicated io.druid.server.emitter package; Update emitter library to 0.6.0; Add support for ParametrizedUriEmitter; Support hierarical properties in JsonConfigurator (was needed for ParametrizedUriEmitter)

* Log created RequestLoggers

* Fix forbidden API

* Test fix

* More Http and Parametrized Http Emitter docs

* Switch to debug level
2017-09-13 17:17:19 -05:00
Niketh Sabbineni 4f6eb47e40 Allow timezone info in timestamp column (#4727)
* Allow timezone info in timestamp column

* Address code review comments

* Incorporating code comments

* Add support for more valid timezone strings

* Incorporate review comments

* Incorporate code review comments
2017-09-13 15:19:39 -05:00
Himanshu 7919469de6 fixes HttpServerInventoryView to call server/segment callbacks correctly and Unit Tests for the class (#4767)
* fixes HttpServerInventoryView to call server/segment callbacks correctly and Unit Tests for the class

* fix checkstyle and forbidden-api errors

* HttpServerInventoryView to finish start() only after server inventory is initialized

* fix compilation errors

* address review comments

* add exponential backoff instead of fixed 5 secs on successive failures

* update test to exercise server fail scenarios

* use AtomicInteger for requestNum and increment only once
2017-09-13 14:24:19 -05:00
Dayue Gao 39a0b171e8 fix a race condition of ParseCtx (#4791) 2017-09-13 10:03:48 -07:00
Gian Merlino eb6791a60c TimestampAggregator: Avoid cross-classloader access of package-private field. (#4788)
* TimestampAggregator: Avoid cross-classloader access of package-private field.

* Simplify.

* Remove unused import.
2017-09-13 09:52:01 -07:00
Gian Merlino 2ce8123bdb Move scan-query from a contrib extension into core. (#4751)
* Move scan-query from a contrib extension into core.

Based on a proposal at: https://groups.google.com/d/topic/druid-development/ME_OatUDnbk/discussion

This patch also adds support for virtual columns to the Scan query,
and updates Druid SQL to use Scan instead of Select.

This patch also makes some behavioral changes to handling of the __time
column. In particular, it is now is returned as "__time" rather than
"timestamp"; it is no longer included if you do not specifically ask for
it in your "columns"; and it is returned as a long rather than a string.

Users can revert time handling to the legacy extension behavior by
setting "legacy" : true in their queries, or setting the property
druid.query.scan.legacy = true. This is meant to provide a migration
path for users that were formerly using the contrib extension.

* Adjustments from review.

* Add back Select query.

* Adjust SQL docs.

* Restore SelectQuery link.
2017-09-13 09:51:24 -07:00
Gian Merlino 587f1807c2 Try to improve CI reliability. (#4787)
* Try to improve CI reliability.

Unset _JAVA_OPTIONS just in case it is causing us to use more memory than expected.

* Replace for loop with foreach.
2017-09-12 22:24:45 -05:00
Gian Merlino c3a1ce6933 SQL: Fix toTimeseriesQuery and toTopNQuery. (#4780)
The former would sometimes eat limits, and the latter would sometimes
use the wrong dimension comparator.
2017-09-12 14:37:27 -07:00
Jonathan Wei 3a29521273 Fix GroupBy limit push down error when buffer is too small (#4745)
* Fix GroupBy limit push down error when buffer is too small

* Address PR comments
2017-09-12 12:34:50 -07:00
Roman Leventov 832cc293ef Refactoring of ReferenceCountingSegment and FireHydrant (#4154)
* Refactoring of ReferenceCountingSegment and FireHydrant

* Address comment

* Fix FireHydrant.closeSegment()

* Address comment

* Added comments to ReferenceCountingSegment
2017-09-12 14:28:35 -05:00
Kenji Noguchi c0be050242 Add jq expression support in flattenSpec (#4171)
* add jq expression in the flattenSpec

* more tests

* add benchmark

* fix style

* use JsonNode for both JSONPath and JQ

* clean up

* more clean up

* add documentation

* fix style

* move jackson-jq version to dependencyManagement section. remove commented code

* oops. revert wrong fix

* throw IllegalArgumentException for JQ syntax error

* remove e.printStackTrace() that is forbidden

* touch
2017-09-12 14:18:34 -05:00
Gian Merlino 4909c48b0c SQL: Full TRIM support. (#4750)
* SQL: Full TRIM support.

- Support trimming arbitrary characters
- Support BOTH, LEADING, and TRAILING

* Remove unused import.

* Fix tests, add RTRIM / LTRIM.

* Remove unused imports.

* BTRIM and docs.

* Replace for with foreach.
2017-09-12 11:49:08 -07:00
Parag Jain b5e839b3db injectable sslcontextfactory for jetty server and key manager factory algorithm (#4769)
* injectable sslcontextfactory for jetty server

key manager factory algorithm

* explicitly set trustAll certificates to false in sslcontextfactory
2017-09-12 11:45:03 -07:00
Egor Riashin 6f3e52b3db Make optional Peon "stdin" check (#4760) 2017-09-11 16:37:01 -05:00
Gian Merlino 23c0357816 BufferHashGrouperTest: Better behavior with regard to large buffers. (#4779)
* BufferHashGrouperTest: Better behavior with regard to large buffers.

1) Free buffers after each test
2) Avoid mmaping past the end of a file

* Use CloserRule.
2017-09-11 12:10:31 -07:00
Himanshu 834e050bc4 Use internal-discovery and http for talking to overlord/coordinator leaders (#4735)
* Use internal-discovery and http for talking to overlord/coordinator leaders

* CuratorDruidNodeDiscovery.getAllNodes() best effort 30 sec wait for cache initialization

* DruidLeaderClientProvider to eagerly instantiate DruidNodeDiscovery when needed so that DruidNodeDiscovery impl cache gets initialized well in time

* Revert "DruidLeaderClientProvider to eagerly instantiate DruidNodeDiscovery when needed so that DruidNodeDiscovery impl cache gets initialized well in time"

This reverts commit f1a2432614ba56ddc2d55fe47e990d17fcfd6129.

* add lifecycle to DruidLeaderClient to early initialize DruidNodeDiscovery so that it has its cache update well in time
2017-09-11 11:18:01 -07:00
Niketh Sabbineni 975a50fc81 Generate UUID if query id is empty (#4772) 2017-09-11 10:40:26 -07:00
Andy Sloane 706747cc8c Fix for sort order in select/topN query cache (#4766)
When historical caching is enabled, and a select or topN query is
issued, and then a following query with "descending": true is set, the
cached query returns the ascending result (or vice versa), often
resulting in invalid paging identifiers.

The CacheKey for these queries doesn't include the "descending" flag;
this change adds it, and fixes the problem.
2017-09-10 11:33:00 +09:00
dgolitsyn 752151f6cb Add CachingCostBalancerStrategy (#4731)
* Add CachingCostBalancerStrategy; Rename ServerView.ServerCallback to ServerRemovedCallback

* Fix benchmark units

* Style, forbidden-api, review, bug fixes

* Add docs

* Address comments
2017-09-08 12:23:04 -05:00
Himanshu d43687d578 increase test timeout to 15sec as leader election jitter is random sleep between 1 to 5 secs (#4756) 2017-09-06 09:56:25 -07:00
Gian Merlino dc5c6f13b1 Use trusty for travis jobs. (#4755)
* Use trusty for travis jobs.

The distro was set to "precise" in #4572 due to memory issues on trusty,
but we've been seeing performance issues on "precise" recently so let's
see how trusty is working these days.

* Less quiet.

* Adjust memory settings.

* Add back -q option.

* Tweak memory again.

* Adjustments.

* Try squeezing memory a bit more.
2017-09-06 18:51:06 +09:00
Gian Merlino 33c0928bed Collapse worker select strategies, change default, add strong affinity. (#4534)
* Collapse worker select strategies, change default, add strong affinity.

- Change default worker select strategy to equalDistribution. It is
  more generally useful than fillCapacity.
- Collapse the *WithAffinity strategies into the regular ones. The
  *WithAffinity strategies are retained for backwards compatibility.
- Change WorkerSelectStrategy to return nullable instead of Optional.
- Fix a couple of errors in the docs.

* Fix test.

* Review adjustments.

* Remove unused imports.

* Switch to DateTimes.nowUtc.

* Simplify code.

* Fix tests (worker assignment started off on a different foot)
2017-09-04 14:40:55 -07:00
Roman Leventov c7b8116b3a Remove HadoopIOPeon (#4742) 2017-09-03 13:36:38 -07:00
Kevin Conaway 83e94e8b31 Expose version property for `CustomVersioningPolicy` so that it may serialize/deserialize correctly (#4747)
Expose version property for  so that it may serialize/deserialize correctly

Expose version property for `CustomVersioningPolicy` so that it may serialize/deserialize correctly

Expose version property for CustomVersioningPolicy so that it may serialize/deserialize correctly

Expose version property for `CustomVersioningPolicy` so that it may serialize/deserialize correctly
2017-09-03 13:36:18 -07:00
Himanshu 06ac6678e6 DruidLeaderSelector interface for leader election and Curator based impl. (#4699)
* DruidLeaderSelector interface for leader election and Curator based impl. DruidCoordinator/TaskMaster are updated to use the new interface.

* add fake DruidNode binding in integration-tests module

* add docs on DruidLeaderSelector interface

* remove start/stop and keep register/unregister Listener in DruidLeaderSelector interface

* updated comments on DruidLeaderSelector

* cache the listener executor in CuratorDruidLeaderSelector

* use same latch owner name that was used before

* remove stuff related to druid.zk.paths.indexer.leaderLatchPath config

* randomize the delay when giving up leadership and restarting leader latch
2017-09-01 09:49:04 -07:00
Gian Merlino 34a03b8e6c SQL: EXPLAIN improvements. (#4733)
* SQL: EXPLAIN improvements.

- Include query JSON in explain output.
- Fix a bug where semi-joins and nested groupBys were not fully explained.
- Fix a bug where limits were not included in "select" query explanations.

* Fix compile error.

* Fix compile error.

* Fix tests.
2017-09-01 09:35:13 -07:00
Himanshu 4c04083926 kafkaIndexTask unannounce service in final block (#4736) 2017-09-01 09:31:15 -07:00
Charles Allen bdfc6fe25e Move common TypeReference into JacksonUtils (#4738) 2017-08-31 13:40:16 -07:00
Gian Merlino 9078925cab Docs for finalizingFieldAccess post-aggregator. (#4737) 2017-08-31 11:45:49 -07:00
T R Kyaw d6179126ed Allow index job to utilize hadoop cluster information from job config. (#4626)
* Allow ndex job to utilize hadoop cluster information from job config.

* Add new method that inject system configuration and then job configuration.

* Make changes to use HadoopDruidIndexerConfig.addJobProperties method.

* refactor code for overloaded addJobProperties.
2017-08-30 16:44:33 -05:00
Bartosz Ługowski 8dddccc687 Graphite emitter - add plaintext protocol (#4265)
* Graphite emitter - add plaintext protocol. Configurable option of replacing slash to dot in metric name.

* Graphite emitter - fix misspelling in docs.

* Graphite emitter - extend docs.

* Graphite emitter - fix code style.
2017-08-29 06:23:06 -07:00
Niketh Sabbineni beecb9e210 Fix failing build, remove unused import (#4726)
LGTM
2017-08-29 14:46:38 +09:00
Roman Leventov 4d109a358a Refactoring of Storage Adapters (#4710)
* Factor QueryableIndexColumnSelectorFactory and IncrementalIndexColumnSelectorFactory out of QueryableIndexStorageAdapter and IncrementalIndexStorageAdapter; Add Offset.getBaseReadableOffset(); Remove OffsetHolder interface; Replace Cursor extends ColumnSelectorFactory with composition; Reduce indirection in ColumnValueSelectors created by QueryableIndexColumnSelectorFactory

* Don't override clone() in FilteredOffset (the prev. implementation was broken); Some warnings fixed

* Simplify Cursors in QueryableIndexStorageAdapter

* Address comments

* Remove unused and unimplemented methods from GenericColumn interface

* Comments
2017-08-28 18:07:31 -07:00