Commit Graph

9695 Commits

Author SHA1 Message Date
Jihoon Son 22d6384d36
Fix unrealistic test variables in KafkaSupervisorTest and tidy up unused variable in checkpointing process (#7319)
* Fix unrealistic test arguments in KafkaSupervisorTest

* remove currentCheckpoint from checkpoint action

* rename variable
2019-08-21 10:58:22 -07:00
Gian Merlino d007477742
Docusaurus build framework + ingestion doc refresh. (#8311)
* Docusaurus build framework + ingestion doc refresh.

* stick to npm instead of yarn

* fix typos

* restore some _bin

* Adjustments.

* detect and fix redirect anchors

* update anchor lint

* Web-console: remove specific column filters (#8343)

* add clear filter

* update tool kit

* remove usless check

* auto run

* add %

* Fix resource leak (#8337)

* Fix resource leak

* Patch comments

* Enable Spotbugs NP_NONNULL_RETURN_VIOLATION (#8234)

* Fixes from PR review.

* Fix more anchors.

* Preamble nix.

* Fix more anchors, headers

* clean up placeholder page

* add to website lint to travis config

* better broken link checking

* travis fix

* Fixed more broken links

* better redirects

* unfancy catch

* fix LGTM error

* link fixes

* fix md issues

* Addl fixes
2019-08-20 21:48:59 -07:00
Benedict Jin c4db83608f
Add grade shield (#8344) 2019-08-21 12:20:19 +08:00
mcbrewster e4aa7fb268 Web-console: fix alerts from lgtm (#8346)
* fix alerts from lgtm

* remove unordered imports

* fix introduced alert

* move getExpanded to renderSchemaSelector

* use getDerivedStateFromProps

* use prevState

* add semi colons, remove unused imports

* fixes
2019-08-20 20:40:47 -07:00
Clint Wylie c87b68d2a4 use Number instead of long for response context (#8342)
* use Number instead of long for response context to be forgiving of json serde to int or long

* test that encounters issue without fix

* now with more test

* is ints
2019-08-20 19:05:49 -07:00
Fokko Driesprong d5a19675dd Remove fromPigAvroStorage from the docs (#8340)
This one has been deprecated a while ago
2019-08-20 16:34:55 -07:00
Benedict Jin 14a4238381 Bump JUnitParams from 1.0.4 to 1.1.1 (#8017) 2019-08-20 16:15:12 -07:00
Fokko Driesprong 8821ac330d Bump opencsv from 4.2 to 4.6 (#8294)
* Bump opencsv from 4.2 to 4.6

* Fix transitive dependencies
2019-08-20 16:12:03 -07:00
Fokko Driesprong 3a58431bff Bump jackson-jq from 0.0.7 to 0.0.10 (#8293)
* Bump jackson-jq from 0.0.7 to 0.0.10

For the changelog:
https://github.com/eiiches/jackson-jq/releases

* Update dependent licenses
2019-08-20 16:09:04 -07:00
Chi Cao Minh 6fa22f6939 Enable code coverage (#8303)
* Enable code coverage

Code coverage was disabled via
https://github.com/apache/incubator-druid/pull/3122 due to an issue with
cobertura in Travis CI. Switch code coverage tool from cobertura to
jacoco to avoid issue and re-enable coveralls for Travis CI.

* Exclude non-production code

* Exclude benchmark generated code

* Exclude DruidTestRunnerFactory
2019-08-20 15:36:19 -07:00
Fokko Driesprong cb1339e19a Bump derby from 10.11.1.1 to 10.14.2.0 (#8292)
* Bump derby from 10.11.1.1 to 10.15.1.3

* Update server/pom.xml as well

* Move to derby 10.14.2.0

10.15.* is Java9+
https://db.apache.org/derby/derby_downloads.html
2019-08-20 14:03:32 -07:00
Jonathan Wei e2a25fb51e
Add logging for LZ4Factory instance type (#8341) 2019-08-20 15:24:53 -05:00
Benedict Jin 5e35500e0e Fix unused format argument (#8345) 2019-08-20 12:45:46 -07:00
pphust ffcbd1ecb8 Ensure ReferenceCountingSegment.decrement() is invoked correctly (#8323)
* fix issue8291. Make sure ReferenceCountingSegment.decrement() is invoked correctly

* add some comments in SinkQuerySegmentWalker.java

* extracting perSegmentRunners and perHydrantRunners to reduce the level of nesting
2019-08-20 22:01:16 +03:00
Fokko Driesprong 818bf4990c Enable Spotbugs NP_NONNULL_RETURN_VIOLATION (#8234) 2019-08-20 17:23:46 +03:00
Benedict Jin 781873ba53 Fix resource leak (#8337)
* Fix resource leak

* Patch comments
2019-08-20 12:55:41 +03:00
mcbrewster e64070b8bf Web-console: remove specific column filters (#8343)
* add clear filter

* update tool kit

* remove usless check

* auto run

* add %
2019-08-19 19:54:18 -07:00
Jonathan Wei dd2e53baf4
Clarify Avro decoder docs (#8302) 2019-08-19 15:37:18 -05:00
Vadim Ogievetsky 56e440383f Web console: data loader native batch shuffle UI (#8229)
* data loader shuffle UI

* better support for inline firehose

* feedback changes

* make numShards mandatory sometimes

* edit context dialog fixes

* make auto form less agro

* download full log

* improve auto column detection

* add sample for example
2019-08-19 13:27:34 -07:00
Benedict Jin 566dc8c719
Fix missing format argument (#8331) 2019-08-19 16:19:44 +08:00
mcbrewster cae1361c32 Web-console: gate auto complete on current table and schema (#8322)
* gate auto complete on current table and schema

* reset defaults

* add static functions

* move completions to state
2019-08-17 09:34:36 -07:00
Jihoon Son 31af4eb9ad
Rename maxNumSubTasks to maxNumConcurrentSubTasks for native parallel index task (#8324) 2019-08-16 15:57:13 -07:00
Vadim Ogievetsky 0d69438395 Web console: make console's Explain more honest (#8327)
* fix structure

* update snapshots
2019-08-16 15:51:49 -07:00
Clint Wylie 7362b1d8fc exclude avro extension dependencies that are already included in druid libs (#8309) 2019-08-16 17:42:06 -05:00
Clint Wylie a34cfd3e68 exclude kerberos extension dependencies that are already included in druid core libraries (#8310)
* exclude kerberos extension dependencies that are already included in druid libs

* missing net

* exclude json-smart

* eh might as well go aggro and remove all the ones it looks like we do not need

* guess we actually need this one
2019-08-16 17:41:47 -05:00
Jihoon Son b654096194
Fix equals for ArrayOfDoublesSketchAggregatorFactory (#8326) 2019-08-16 14:47:37 -07:00
Benedict Jin 399b3ad7de
Hide descriptive comments (#8313) 2019-08-16 10:08:34 +08:00
Jihoon Son 5dac6375f3
Add support for parallel native indexing with shuffle for perfect rollup (#8257)
* Add TaskResourceCleaner; fix a couple of concurrency bugs in batch tasks

* kill runner when it's ready

* add comment

* kill run thread

* fix test

* Take closeable out of Appenderator

* add javadoc

* fix test

* fix test

* update javadoc

* add javadoc about killed task

* address comment

* Add support for parallel native indexing with shuffle for perfect rollup.

* Add comment about volatiles

* fix test

* fix test

* handling missing exceptions

* more clear javadoc for stopGracefully

* unused import

* update javadoc

* Add missing statement in javadoc

* address comments; fix doc

* add javadoc for isGuaranteedRollup

* Rename confusing variable name and fix typos

* fix typos; move fetch() to a better home; fix the expiration time

* add support https
2019-08-15 17:43:35 -07:00
Jihoon Son eeae5d9365 Add a warning about experimental segment locking (#8301)
* Add a warning about experimental segment locking

* fix typo
2019-08-15 16:07:59 -07:00
Jonathan Wei ef7b9606f2
Keep track of task location for completed tasks (#8286)
* Keep track of task location for completed tasks

* Add TaskLifecycleTest location checks
2019-08-15 16:57:02 -05:00
mcbrewster 8924d285dc Web-Console: add more side column functions to query view (#8283)
* small fixes

* run jest -u

* add trim

* add to groupby

* save

* add functionality to column tree menu

* remove history dialog

* rename file

* fixes

* add new values to time-menu

* add query dialog

* add more functions to collumn-tree menu

* add menu functions

* remove test

* update tests, fix extra 0s, clear in between time filters

* fixes

* add deffered

* save

* add alias's

* change menu style

* fix substring

* small fixes

* add auto run to run button

* move divider to a reasonable position

* fix auto expland

* run jest -u

* fix auto run

* prevent aggregate cooumn changes

* fix group by bug

* add quertAst to state

* ui fixes

* prevent filter on non existant ast

* add auto run to shows

* use handle run

* handle *

* add alias and tslint fix
2019-08-15 01:22:25 -07:00
Clint Wylie ff26fcb159 add copyright info back to NOTICE and NOTICE.BINARY (#8298)
* add Apache copyrights back to NOTICE per https://lists.apache.org/thread.html/4e296f3ede34aa5aeb0b2a8ed0f3540b320bce07936aa3c8bfa135b3@%3Cgeneral.incubator.apache.org%3E

* fixup notice.binary

* now with 3x the hive notices

* less redundant
2019-08-14 19:42:47 -05:00
Sayat 1f3a99616d Upgrade Kafka library for kafka-lookup module (#8078)
* Upgrade Kafka library for kafka-lookup module

* Update licenes.yaml

* Adopt class workaround from KafkaRecordSupplier#getKafkaConsumer

* Update lisences for kafka clients
2019-08-14 13:46:25 -07:00
Clint Wylie 7fa0ff5e11 adjust azure storage retry conditions (#8296) 2019-08-13 21:48:45 -07:00
Chi Cao Minh c8d7e0a868 Add slack badge (#8295)
* Add slack badge

* Disable broken badges
2019-08-13 21:48:17 -07:00
Fokko Driesprong 1a3aa1cfc0 Bump commons-io from 2.5 to 2.6 (#8006)
* Bump commons-io from 2.5 to 2.6

* Update licenses.yaml

* Address comments
2019-08-13 17:10:37 -07:00
Himanshu 176da53996
make double sum/min/max agg work on string columns (#8243)
* make double sum/min/max agg work on string columns

* style and compilation fixes

* fix tests

* address review comments

* add comment on SimpleDoubleAggregatorFactory

* make checkstyle happy
2019-08-13 15:55:14 -07:00
Jihoon Son a5c9c2950f Add missing maxBytesInMemory in tuningConfig for auto compaction (#8274)
* Add missing tuningConfigs for auto compaciton

* Add doc

* add test
2019-08-13 14:10:26 -05:00
Xavier Léauté eaa4651fa4 docker build cleanup and speed improvements (#8288)
* switch builder image to smaller "slim" debian base
* use built-in pyyaml
* skip various build checks to speed up compilation
* create user/group in target image directly
2019-08-13 08:33:06 -07:00
Jihoon Son 312cdc2452 Add TaskResourceCleaner; fix a couple of concurrency bugs in batch tasks (#8236)
* Add TaskResourceCleaner; fix a couple of concurrency bugs in batch tasks

* kill runner when it's ready

* add comment

* kill run thread

* fix test

* Take closeable out of Appenderator

* add javadoc

* fix test

* fix test

* update javadoc

* add javadoc about killed task

* address comment

* handling missing exceptions

* more clear javadoc for stopGracefully

* update javadoc

* Add missing statement in javadoc

* typo
2019-08-12 19:42:06 -05:00
Chi Cao Minh ccd87d2667 Speedup package check job (#8282)
The package check job sometimes hits the 50 minute Travis CI job time
limit. Move license checking tasks in "package check" job to "license
check" job to rebalance the job runtime (the "license check" job
currently takes about 1 minute). Moving the logic from build.sh to
.travis.yml also gives more visibility into how long each step takes
(i.e., generate-license-dependency-reports.py vs generate-license.py).
2019-08-12 13:25:08 -07:00
Alexandre Yang 6b4d028b96 [statsd-emitter] Add config to send Druid process/service as tag (#8238)
* [statsd-emitter] Add serviceAsTag option

* [statsd-emitter] Refactor serviceAsTag option

* [statsd-emitter] Update statsd.md

* [statsd-emitter] add default prefix

* [statsd-emitter] update statsd.md

* [statsd-emitter] Remove extra spaces

* [statsd-emitter] Improve docs for config `dogstatsdServiceAsTag`

* [statsd-emitter] Simplify equals() for StatsDEmitterConfig.java

* [statsd-emitter] Add @Nullable for StatsDEmitterConfig.java
2019-08-12 13:18:44 -07:00
Clint Wylie 1054d85171
add mechanism to control filter optimization in historical query processing (#8209)
* add support for mechanism to control filter optimization in historical query processing

* oops

* adjust

* woo

* javadoc

* review comments

* fix

* default

* oops

* oof

* this will fix it

* more nullable, refactor DimFilter.getRequiredColumns to use Set, formatting

* extract class DimFilterToStringBuilder with common code from custom DimFilter toString implementations

* adjust variable naming

* missing nullable

* more nullable

* fix javadocs

* nullable

* address review comments

* javadocs, precondition

* nullable

* rename method to be consistent

* review comments

* remove tuning from ColumnComparisonFilter/ColumnComparisonDimFilter
2019-08-09 16:36:18 -07:00
Nathan b28e252d9a Minor Spelling Error (#8277)
* Minor Spelling Error

* Update mySQL password in docs

/extensions-core/mysql update druid.metadata.storage.connector.password
2019-08-09 16:06:02 -05:00
Jonathan Wei e8727dc98b
Fix DeterminePartitionsJob reducer when total rows < targetPartitionSize * SHARD_COMBINE_THRESHOLD (#8273)
* Fix DeterminePartitionsJob reducer when rows < targetPartitionSize

* use isEmpty()
2019-08-09 16:03:30 -05:00
Benedict Jin 170368999d Bump rhino from 1.7R5 to 1.7.11 (#8008)
* Bump rhino from 1.7R5 to 1.7.11

* Update the version of rhino in licenses.yaml
2019-08-09 13:10:54 -07:00
Benedict Jin f7cf2f7cad Bump httpcore from 4.4.4 to 4.4.11 (#7870)
* Bump httpcore from 4.4.4 to 4.4.11

* Update the version of httpcore in licenses.yaml
2019-08-09 19:53:20 +03:00
mcbrewster d1f286b7f4 fix no from clause error (#8275) 2019-08-08 21:42:52 -07:00
Chi Cao Minh b359c5b3d9 Fix SIGAR dependency connection timeout (#8258)
After enabling parallel builds for "mvn install", the sigar dependency
would sometimes resolve to the incorrect artifact repo for some of the
maven modules. This issue seems to be fixed by moving the definition of
the sigar dependency's artifact repo to the root POM.

Also, depending on network speeds, "mvn -q install" may take longer than
the default 10 minute timeout to print any output. Use travis_wait to
extend the timeout to 15 minutes.
2019-08-08 20:13:18 -05:00
Fangjin Yang 76675d6163
Update README.md (#8261)
remove IRC
2019-08-08 10:23:41 -07:00