Commit Graph

9247 Commits

Author SHA1 Message Date
Gian Merlino d0400a0688 SegmentWithState: Add toString method. (#5635)
The class appears in log messages, and the default toString method
isn't very informative.
2018-04-12 14:01:09 -05:00
palanieppan-m dbea5cb9b7 Load rules should honor partial overlap (#5595)
Load rules should load segments that partially overlap with rule window,
instead of loading only segments that fully overlap.
2018-04-12 09:46:00 -07:00
Atul Mohan 19f359957f Add getters for AlertEvent (#5522)
* Add getters for AlertEvent

* Move PublicApi and ExtensionPoint to java-util

* Fix publicapi annotation usage

* Add publicapi annotations to ServiceMetricEvent and RequestLogEvent
2018-04-12 23:38:20 +07:00
Caroline1000 48c1a1ef57 change header from Data Schema to Ingestion Spec (#5631) 2018-04-11 21:42:54 -07:00
Caroline1000 d709d1a59f correct overlord console header. Because it's not the coordinator console, it's the overlord console. (#5627) 2018-04-11 20:44:31 -04:00
Gian Merlino 72d6dcda4f
ParallelCombiner: Fix buffer leak on exception in "combine". (#5630)
Once a buffer is acquired, we need to make sure to release it if an
exception is thrown before the closeable iterator is created.
2018-04-11 20:39:39 -04:00
Nishant Bangarwa e6efd75a3d Add config to allow setting up custom unsecured paths for druid nodes. (#5614)
* Add config to allow setting up custom unsecured paths for druid nodes.

* return all resources for Unsecured paths

* review comment - Add test

* fix tests

* fix test
2018-04-11 17:10:07 -07:00
Caroline1000 afa75e04b7 change header in overlord console; minor querydoc change (#5625)
* change header in overlord console; minor querydoc change

* remove change to overlord console

* address Gian comments
2018-04-11 12:57:22 -07:00
Nishant Bangarwa b32aad9ab4 Fix some broken links in druid docs (#5622)
* Fix some broken links in druid docs

* review comment
2018-04-11 10:27:33 -07:00
Gian Merlino ff27c54774 SQL: Remove useless boolean CASTs in filters. (#5619) 2018-04-10 23:05:09 +05:30
Nishant Bangarwa 80fa5094e8 Fix Kerberos Authentication failing requests without cookies and excludedPaths config. (#5596)
* Fix Kerberos Authentication failing requests without cookies.

KerberosAuthenticator was failing `First` request from the clients.
After authentication we were setting the cookie properly but not
setting the the authenticated flag in the request. This PR fixed that.

Additional Fixes -
* Removing of Unused SpnegoFilterConfig - replaced by
KerberosAuthenticator
* Unused internalClientKeytab and principal from KerberosAuthenticator
* Fix docs accordingly and add docs for configuring an escalated
client.

* Fix excluded path config behavior

* spelling correction

* Revert "spelling correction"

This reverts commit fb754b43d8.

* Revert "Fix excluded path config behavior"

This reverts commit 3901047769.
2018-04-09 20:45:35 -07:00
Gian Merlino ee37ff2502 CompressionUtils: Make gzipInputStream public once again. (#5590)
But add a reference to "decompress" and mention that it's preferred
when reading from streams that come from files.
2018-04-09 17:39:19 -07:00
Jihoon Son 3a5d516308 Add missing type for MapVirtualColumn (#5598) 2018-04-09 17:37:24 -07:00
Gian Merlino 685f4063d4 DoublesSketchModule: Fix serde for DoublesSketchMergeAggregatorFactory. (#5587)
Fixes #5580.
2018-04-07 06:45:55 -07:00
Alexander T ad6f234e1e Update lookups-cached-global.md (#5525)
Update lookup creation example to work with version 0.12.0
2018-04-06 16:13:17 -07:00
Clint Wylie ea4f8544fb revert lambda conversion to fix occasional jvm error (#5591) 2018-04-06 14:18:55 -07:00
Gian Merlino 5ab17668c0 CompressionUtils: Add support for decompressing xz, bz2, zip. (#5586)
Also switch various firehoses to the new method.

Fixes #5585.
2018-04-06 08:06:45 -07:00
Charles Allen b86ed99d9a
Deprecate spark2 profile in pom.xml (#5581)
Deprecated due to https://github.com/druid-io/druid/pull/5382
2018-04-06 05:37:16 -07:00
Alexander T c228eed500 Update sql.md (#5519)
Example code is wrong. 'Statement' has to be created from the Connection Object
2018-04-06 00:21:18 -07:00
Jihoon Son 298ed1755d Fix indexTask to respect forceExtendableShardSpecs (#5509)
* Fix indexTask to respect forceExtendableShardSpecs

* add comments
2018-04-05 23:54:59 -07:00
Jihoon Son 723857699c Add doc for automatic pendingSegments (#5565)
* Add missing doc for automatic pendingSegments

* address comments
2018-04-05 23:53:43 -07:00
Dylan Wylie ddd23a11e6 Fix taskDuration docs for KafkaIndexingService (#5572)
* With incremental handoff the changed line is no longer true.
2018-04-05 23:52:58 -07:00
Senthil Kumar L S 371c672828 Adding feature thetaSketchConstant to do some set operation in PostAgg (#5551)
* Adding feature thetaSketchConstant to do some set operation in PostAggregator

* Updated review comments for PR #5551 - Adding thetaSketchConstant

* Fixed CI build issue

* Updated review comments 2 for PR #5551 - Adding thetaSketchConstant
2018-04-05 22:56:59 -07:00
Niketh Sabbineni 270fd1ea15 Allow getDomain to return disjointed intervals (#5570)
* Allow getDomain to return disjointed intervals

* Indentation issues
2018-04-05 22:12:30 -07:00
Jonathan Wei 969342cd28
More error reporting and stats for ingestion tasks (#5418)
* Add more indexing task status and error reporting

* PR comments, add support in AppenderatorDriverRealtimeIndexTask

* Use TaskReport instead of metrics/context

* Fix tests

* Use TaskReport uploads

* Refactor fire department metrics retrieval

* Refactor input row serde in hadoop task

* Refactor hadoop task loader names

* Truncate error message in TaskStatus, add errorMsg to task report

* PR comments
2018-04-05 21:38:57 -07:00
Jonathan Wei 818091ec60
Add overlord unsecured paths to coordinator when using combined service (#5579)
* Add overlord unsecured paths to coordinator when using combined service

* PR comment
2018-04-05 14:16:06 -07:00
Jihoon Son 7239f56131 Fix NPE in RemoteTaskRunner when some tasks in ZooKeeper but not in Overlord (#5511)
* Fix NPE in RemoteTaskRunner when some tasks in ZooKeeper but not in Overlord

* revert unnecessary change
2018-04-03 21:15:58 -07:00
Niketh Sabbineni f0a94f5035 Remove unused config (#5564)
* Remove unused config

* Fix failing tests
2018-04-03 13:23:46 -07:00
Clint Wylie f31dba6c5b Coordinator drop segment selection through cost balancer (#5529)
* drop selection through cost balancer

* use collections.emptyIterator

* add test to ensure does not drop from server with larger loading queue with cost balancer

* javadocs and comments to clear things up

* random drop for completeness
2018-04-03 11:22:51 -07:00
Clint Wylie a81ae99021 add 'stopped' check and handling to HttpLoadQueuePeon load and drop segment methods (#5555)
* add stopped check and handling to HttpLoadQueuePeon load and drop segment methods

* fix unrelated timeout :(

* revert unintended change

* PR feedback: change logging

* fix dumb
2018-04-03 11:21:52 -07:00
Jonathan Wei 723f7ac550
Add support for task reports, upload reports to deep storage (#5524)
* Add support for task reports, upload reports to deep storage

* PR comments

* Better name for method

* Fix report file upload

* Use TaskReportFileWriter

* Checkstyle

* More PR comments
2018-04-02 12:10:56 -07:00
Clint Wylie 6feac204e3 Coordinator primary segment assignment fix (#5532)
* fix issue where assign primary assigns segments to all historical servers in cluster

* fix test

* add test to ensure primary assignment will not assign to another server while loading is in progress
2018-04-02 09:40:20 -07:00
Jihoon Son 05547e29b2
Fix SQLMetadataSegmentManager to allow succesive start and stop (#5554)
* Fix SQLMetadataSegmentManager to allow succesive start and stop

* address comment

* add synchronization
2018-03-30 12:43:19 -07:00
Arup Malakar 0c4598c1fe Fix typo in avatica java client code documenation (#5553) 2018-03-29 16:36:40 -05:00
Clint Wylie 30fc4d3ba0 Coordinator balancer move then drop fix (#5528)
* #5521 part 1

* formatting

* oops

* less magic tests
2018-03-29 10:30:12 -07:00
Kirill Kozlov 8878a7ff94 Replace guava Charsets with native java StandardCharsets (#5545) 2018-03-28 21:00:08 -07:00
Clint Wylie 81be1b3966 this will fix it (#5549) 2018-03-28 18:58:39 -07:00
Niketh Sabbineni 912adcc284 ArrayAggregation: Use long to avoid overflow (#5544)
* ArrayAggregation: Use long to avoid overflow

* Add Tests
2018-03-28 16:37:53 -07:00
Jihoon Son 024e0a9cca Respect forceHashAggregation in queryContext (#5533)
* Respect forceHashAggregation in queryContext

* address comment
2018-03-28 14:15:38 -07:00
Dyana Rose db508cf3ca [docs] fix invalid example json (#5547)
https://github.com/druid-io/druid/issues/5546
2018-03-28 13:53:38 -07:00
Clint Wylie 50e0e7f97d Correct lookup documentation (#5537)
fixes #5536
2018-03-26 17:01:02 -07:00
Nathan Hartwell ea30c05355 Adding ParserSpec for Influx Line Protocol (#5440)
* Adding ParserSpec for Influx Line Protocol

* Addressing PR feedback

- Remove extraneous TODO
- Better handling of parse errors (e.g. invalid timestamp)
- Handle sub-millisecond timestamps

* Adding documentation for Influx parser

* Fixing docs
2018-03-26 14:28:46 -07:00
Atul Mohan ec17a44e09 Add result level caching to Brokers (#5028)
* Add result level caching to Brokers

* Minor doc changes

* Simplify sequences

*  Move etag execution

* Modify cacheLimit criteria

* Fix incorrect etag computation

* Fix docs

* Add separate query runner for result level caching

* Update docs

* Add post aggregated results to result level cache

* Fix indents

* Check byte size for exceeding cache limit

* Fix indents

* Fix indents

* Add flag for result caching

* Remove logs

* Make cache object generation synchronous

* Avoid saving intermediate cache results to list

* Fix changes that handle etag based response

* Release bytestream after use

*  Address PR comments

*  Discard resultcache stream after use

* Fix docs

* Address comments

* Add comment about fluent workflow issue
2018-03-23 19:11:52 -07:00
Charles Allen ef21ce5a64
Add graceful shutdown timeout for Jetty (#5429)
* Add graceful shutdown timeout

* Handle interruptedException

* Incorporate code review comments

* Address code review comments

* Poll for activeConnections to be zero

* Use statistics handler to get active requests

* Use native jetty shutdown gracefully

* Move log line back to where it was

* Add unannounce wait time

* Make the default retain prior behavior

* Update docs with new config defaults

* Make duration handling on jetty shutdown more consistent

* StatisticsHandler is a wrapper

* Move jetty lifecycle error logging to error
2018-03-23 09:38:17 -07:00
Gian Merlino 0851f2206c
Expanded documentation for DataSketches aggregators. (#5513)
Originally written by @AlexanderSaydakov in druid-io/druid-io.github.io#448.
I also added redirects and updated links to point to the new
datasketches-extension.html landing page for the extension, rather than to
the old page about theta sketches.
2018-03-21 18:19:27 -07:00
Jihoon Son 1ad898bde2
Use the official aws-sdk instead of jet3t (#5382)
* Use the official aws-sdk instead of jet3t

* fix compile and serde tests

* address comments and fix test

* add http version string

* remove redundant dependencies, fix potential NPE, and fix test

* resolve TODOs

* fix build

* downgrade jackson version to 2.6.7

* fix test

* resolve the last TODO

* support proxy and endpoint configurations

* fix build

* remove debugging log

* downgrade hadoop version to 2.8.3

* fix tests

* remove unused log

* fix it test

* revert KerberosAuthenticator change

* change hadoop-aws scope to provided in hdfs-storage

* address comments

* address comments
2018-03-21 15:36:54 -07:00
Clint Wylie 885b975c95 fix LongsColumnWithNulls and FloatsColumnWithNulls to override isNull in order to actually use nullValueBitmap (#5510) 2018-03-20 16:04:08 -07:00
Charles Allen 58f110f7f8 Future-proof some Guava usage (#5414)
* Future-proof some Guava usage

* Use a java-util EmptyIterator instead of Guava's
* Change some of the guava future handling to do manual async
transforms. Guava changes transform into transformAsync by deprecating
transform in ONLY Guava 19. Then its gone in 20

* Use `Collections.emptyIterator()`

* Pretty formatting

* Make listenable future transforms a thing in default druid

* Format fix

* Add forbidden guava apis

* Make the ListenableFutrues.transformAsync have comments

* Undo intellij bad pattern matching in comments

* Futrues --> Futures

* Add empty iterators forbidding

* Fix extra `A`

* Correct method signature

* Address review comments

* Finish Gian review comments

* Proper syntax from https://github.com/policeman-tools/forbidden-apis/wiki/SignaturesSyntax
2018-03-20 08:59:33 -07:00
Slim 17c71a2a60
Make Doubles aggregators use 64bits by default (#5478)
* use 64-bit float representation for double based aggregator

Change-Id: Ia4f442037052add178f6ac68138c9d52f96c6e09

* review comments

Change-Id: I5a588f7364f236bf22f2b138e9d743bfb27c67fe
2018-03-19 19:13:04 -07:00
Jonathan Wei b22455b924
Fix supervisor tombstone auth handling (#5504) 2018-03-19 12:55:47 -07:00