Commit Graph

8859 Commits

Author SHA1 Message Date
Jonathan Wei c7ac8785a1 Prevent failed KafkaConsumer creation from blocking overlord startup (#6383)
* Prevent failed KafkaConsumer creation from blocking overlord startup

* PR comments

* Fix random task ID length

* Adjust test timer

* Use Integer.SIZE
2018-10-03 19:08:20 -07:00
QiuMM 0b8085aff7 Prohibit jackson ObjectMapper#reader methods which are deprecated (#6386)
* Prohibit jackson ObjectMapper#reader methods which are deprecated

* address comments
2018-10-03 17:55:20 -03:00
Roman Leventov 3ae563263a
Renamed 'Generic Column' -> 'Numeric Column'; Fixed a few resource leaks in processing; misc refinements (#5957)
This PR accumulates many refactorings and small improvements that I did while preparing the next change set of https://github.com/druid-io/druid/projects/2. I finally decided to make them a separate PR to minimize the volume of the main PR.

Some of the changes:
 - Renamed confusing "Generic Column" term to "Numeric Column" (what it actually implies) in many class names.
 - Generified `ComplexMetricExtractor`
2018-10-02 14:50:22 -03:00
Gian Merlino 244046fda5 SQL: Fix too-long headers in http responses. (#6411)
Fixes #6409 by moving column name info from HTTP headers into the
result body.
2018-10-01 18:13:08 -07:00
Surekha 42e5385e56 make 0.13 tasks API backwards compatible with 0.12 (#6333) (#6334)
* Replace statusCode with status (#6333)

Also changed runnerStatusCode to runnerStatus to keep things consistent

* Add unit test

* Add status param to TaskStatusPlus

Revert to statusCode and runnerStatusCode

* Add additional status member to TaskStatusPlus

* Change TaskResponseObject to match overlord's response object

* Address PR comments

* address comments

* Add runtime exception after logging error

* Remove (deprecated)status member variable from TaskStatusPlus

* Minor change
2018-10-01 15:33:24 -07:00
Jihoon Son cb14a43038 Remove ConvertSegmentTask, HadoopConverterTask, and ConvertSegmentBackwardsCompatibleTask (#6393)
* Remove ConvertSegmentTask, HadoopConverterTask, and ConvertSegmentBackwardsCompatibleTask

* update doc and remove auto conversion

* remove remaining doc

* fix teamcity
2018-10-01 12:03:35 -07:00
Gian Merlino 6e878249a0 ITTLSTest: Retry on "Connection reset by peer". (#6406)
* ITTLSTest: Retry on "Connection reset by peer".

This sometimes happens and causes tests to fail spuriously.

* Better error messages when retries are exhausted.
2018-09-30 12:30:10 -07:00
Gian Merlino 9fa4afdb8e URL encode datasources, task ids, authenticator names. (#5938)
* URL encode datasources, task ids, authenticator names.

* Fix URL encoding for router forwarding servlets.

* Fix log-with-offset API.

* Fix test.

* Test adjustments.

* Task client fixes.

* Remove unused import.
2018-09-30 12:29:51 -07:00
Gian Merlino 3548396a45 SQL: Update to Calcite 1.17.0. (#6404)
* SQL: Update to Calcite 1.17.0.

Other than keeping things fresh, another motivation is that
this fixes CALCITE-1436 (AggregateNode NPE for aggregators other
than SUM/COUNT), which affects aggregate functions on our system
tables.

Also sets shouldConvertRaggedUnionTypesToVarying = true, a new
type system parameter that prefers VARCHAR over CHAR. This is
better for Druid, because we don't really have support for a
true CHAR type.

* Remove unused import.
2018-09-29 18:33:29 -07:00
Shiv Toolsidass a56ffe6ab2 Added backpressure metric to docs and defaultMetricDimensions (#6405)
* Added backpressure metric to docs and defaultMetricDimensions.json

* Reworded description for backpressure metric in docs
2018-09-29 17:57:29 -07:00
Shiv Toolsidass 5a894f830b Added backpressure metric (#6335)
* Added backpressure metric

* Updated channelReadable to AtomicBoolean and fixed broken test

* Moved backpressure metric logic to NettyHttpClient

* Fix placement of calculating backPressureDuration
2018-09-29 14:24:04 -07:00
Jihoon Son f09e718c68 Implement MapVirtualColumn.makeDimensionSelector properly (#6396)
* Implement MapVirtualColumn.makeDimensionSelector properly

* address comments
2018-09-29 14:13:05 -07:00
Gian Merlino 3922582d8c
SQL: Fix too-strict check in SortProject. (#6403)
The "Duplicate field name" check on inputRowSignature is too strict:
it is actually fine for a row signature to have the same field name
twice. It happens when the same expression is selected twice, and
both selections map to the same Druid object (dimension, aggregator,
etc).

I did not succeed in writing a test that triggers this, but I did see
it occur in production for a complex query with hundreds of aggregators.
2018-09-29 13:54:34 -07:00
dyf6372 63ba7f7bec overlord check task whether is present before get lock (#6308) 2018-09-28 16:57:40 -07:00
Jihoon Son 122caec7b1
Add support targetCompactionSizeBytes for compactionTask (#6203)
* Add support targetCompactionSizeBytes for compactionTask

* fix test

* fix a bug in keepSegmentGranularity

* fix wrong noinspection comment

* address comments
2018-09-28 11:16:35 -07:00
adursun 6f44e568db Add missing comma (#6399) 2018-09-28 09:02:36 -07:00
QiuMM 13bbbbf608 Fix issue that forbidden-api check prevents building individual modules (#6394) 2018-09-27 11:25:10 -07:00
QiuMM 47a6cca013 Add TimestampSpec format for microsecond (#6395) 2018-09-27 09:38:44 -07:00
Jihoon Son aef022de98 Fix race in taskMaster (#6388) 2018-09-26 21:48:02 -07:00
Clint Wylie fc1d5795c1 remove wikipedia irc firehose and dependencies from core server module to examples (#6391) 2018-09-26 21:46:37 -07:00
QiuMM 993bc5e9d3 Fix Kafka Indexing Service notice handle thread may never terminate (#6337)
* Fix Kafka Indexing Service notice handle thread may never terminate

* address comment

* handle null value
2018-09-26 20:09:53 -07:00
Jihoon Son faf3f1e426 Fix cache keys of DefaultDimensionSpec and ExtractionDimensionSpec (#6390) 2018-09-26 20:08:53 -07:00
QiuMM 00ea8c00ac using Entry directly instead of Map.Entry in KafkaSupervisor (#6291) 2018-09-26 19:01:36 -07:00
Jihoon Son 6fb503c073 Deprecate task audit logging (#6368)
* Deprecate task audit logging

* fix test

* fix it test
2018-09-26 16:28:02 -07:00
Nishant Bangarwa c9d281a2e9 Add ability to pass in Bloom filter from Hive Queries (#6222)
* Bloom filter initial implementation

fix checkstyle

review comments

Fix wierd failure

review comments

Revert "Fix wierd failure"

This reverts commit a13a83ad7887e679f6d539191b52aeaaea85b613.

* fix test

* review comment
2018-09-26 16:04:26 -07:00
Clint Wylie 0f2dfe6fe8 fix issue with parsing 'date' columns with druid-orc-extensions (#6380)
* fix issue with parsing date columns with druid-orc-extensions

* stuff

* fix forbidden api
2018-09-26 13:18:15 -07:00
Jonathan Wei 6a909c85d0
Fix TLS IT for docker-machine (#6379) 2018-09-26 12:59:05 -07:00
QiuMM 6843cbba1d Fix issue that the forbidden-apis check do not always work (#6371)
* Fix the forbidden apis check do not work issue

* use SuppressForbidden annotation
2018-09-26 19:39:39 +03:00
Roman Leventov 8978d3751b Don't convert DruidServer to ImmutableDruidServers multiple times in CoordinatorHistoricalManagerRunnable (#6385) 2018-09-26 09:14:14 -07:00
Gian Merlino a92a20e037 Fix indexes introduced in #6348. (#6356)
The indexes introduced in #6348 were on the wrong table. The tests
did not catch them due to retries on the create table steps (the
first try created the table but not the bogus indexes; the second
try noticed that the table already existed and did nothing). This
patch doesn't fix the issue with the tests, since the best way to
do that would be to do the table and index creation in a
transaction; but, this is not supported by all of our supported
database engines.
2018-09-25 20:49:13 -07:00
Caroline1000 034d006d24 add to docs on including Drop rule with Load rule (#6378) 2018-09-25 20:13:52 -07:00
Jonathan Wei 207dd3a0ee Reduce travis license check log output (#6382) 2018-09-25 20:02:00 -07:00
Roman Leventov 4afa85e4e8 Move the Rat plugin to a separate Maven profile (#6376)
* Move the Rat plugin to a separate Maven profile

* Revert self-check
2018-09-25 14:39:24 -07:00
Gian Merlino 0da042cdd9
SQL: Unwrap IS_TRUE, IS_FALSE and friends when building a filter. (#6374)
* SQL: Unwrap IS_TRUE, IS_FALSE and friends when building a filter.

* Fix test.
2018-09-25 10:37:02 -07:00
Benedict Jin e5d9fcfe8f Add maven.exec.xxx.skip option for exec-maven-plugin (#6162)
* Fix conflicts

* Modify io.druid into org.apache.druid
2018-09-25 10:05:26 -07:00
Jihoon Son d08c2c5eba
Make JvmThreadsMonitor injectable (#6369) 2018-09-24 20:41:17 -07:00
Jihoon Son 99428e20d2 Deprecate dimensions / metrics APIs on brokers (#6361)
* Deprecate dimensions / metrics APIs on brokers

* add segmentMetadataQuery link

* add more doc
2018-09-24 17:56:38 -07:00
Jonathan Wei ee7b565469 Docs for ingestion stat reports and new parse exception handling (#6373) 2018-09-24 17:45:05 -07:00
Jonathan Wei 00b0a156e9
Tweak isInvalidRows behavior in HadoopTuningConfig (#6339)
* Tweak isInvalidRows behavior in HadoopTuningConfig

* Fix tests
2018-09-24 16:13:13 -07:00
Alexander Saydakov 93345064b5 HllSketch module (#5712)
* HllSketch module

* updated license and imports

* updated package name

* implemented makeAggregateCombiner()

* removed json marks

* style fix

* added module

* removed unnecessary import, side effect of package renaming

* use TreadLocalRandom

* addressing code review points, mostly formatting and comments

* javadoc

* natural order with nulls

* typo

* factored out raw input value extraction

* singleton

* style fix

* style fix

* use Collections.singletonList instead of Arrays.asList

* suppress warning
2018-09-24 08:41:56 -07:00
Roman Leventov 9a3195e98c Improve interning in SQLMetadataSegmentManager (#6357)
* Improve interning in SQLMetadataSegmentManager

* typo
2018-09-22 13:23:30 -07:00
Jonathan Wei 364bf9d1f9 Fix non org.apache.druid files and add package name checkstyle rule (#6367)
* Fix non org.apache.druid files and add package name checkstyle rule

* PR comment
2018-09-21 17:58:19 -07:00
Clint Wylie 399a5659b2 fix incorrect precondition check in `SupervisorManager.suspendOrResumeSupervisor` (#6364)
This check is reverse from the intention
2018-09-21 17:40:14 -07:00
Jonathan Wei f12ffd19a8
Add Kafka reset instructions for tutorial (#6362) 2018-09-21 14:18:31 -07:00
QiuMM 255214cbe6 correct variable name in KafkaSupervisor (#6354) 2018-09-20 16:22:03 -07:00
Gian Merlino e1c649e906 Add metadata indexes to help with segment allocation. (#6348)
Segment allocation queries can take a long time (10s of seconds) when
you have a lot of segments. Adding these indexes helps greatly.
2018-09-19 15:54:13 -07:00
Jonathan Wei 8972244c68 Mutual TLS support (#6076)
* Mutual TLS support

* Kafka test fixes

* TeamCity fix

* Split integration tests

* Use localhost DOCKER_IP

* Increase server thread count

* Increase SSL handshake timeouts

* Add broken pipe retries, use injected client config params

* PR comments, Rat license check exclusion
2018-09-19 09:56:15 -07:00
Joshua Sun 4fafc2ccc9 fixes race condition in kafkasupervisor (#6304)
* fixes race condition in kafkasupervisor

* async verify checkpoints

* fixes race condition in kafkasupervisor

* replace commonly used methods with variables

* remove countdownlatch import

* reformat

* fixes
2018-09-18 22:37:22 -07:00
Jonathan Wei 2e82edc5e0 More exclusions for Rat license check (#6346) 2018-09-18 20:47:56 -07:00
Slim Bouguerra 028354eea8 Adding licenses and enable apache-rat-plugin. (#6215)
* Adding licenses and enable apache-rat-plugi.

Change-Id: I4685a2d9f1e147855dba69329b286f2d5bee3c18

* restore the copywrite of demo_table and add it to the list of allowed ones

Change-Id: I2a9efde6f4b984bc1ac90483e90d98e71f818a14

* revirew comments

Change-Id: I0256c930b7f9a5bb09b44b5e7a149e6ec48cb0ca

* more fixup

Change-Id: I1355e8a2549e76cd44487abec142be79bec59de2

* align

Change-Id: I70bc47ecb577bdf6b91639dd91b6f5642aa6b02f
2018-09-18 08:39:26 -07:00