Commit Graph

9637 Commits

Author SHA1 Message Date
Sashidhar Thallam 6cc8802b8e #7875: Setting ACL on S3 task logs on similar lines as that of data segments pushed to S3 (#7907)
* #7875: Setting ACL on S3 task logs on similar lines as that of data segment pushed to S3

* #7875 1. Extracting a method (which uploads a file to S3 setting appropriate access control list to the file being uploaded) and moving it to utils class. 2. Adding S3TaskLogsTest.java file to test acl (permissions) on the task log files pushed to S3.

* fixing checkstyle errors

* #7875 Incorporating review comments
2019-06-24 17:25:59 -06:00
Jihoon Son 0d5fbfa0eb
Add missing license pointer for Porter Stemmer (#7941)
* Add missing license pointer for Porter Stemmer

* add missing file
2019-06-24 12:21:40 -07:00
Fokko Driesprong 82b248cc17 Spotbugs: Enable MS_SHOULD_BE_FINAL (#7946) 2019-06-23 15:42:18 -07:00
Jonathan Wei 35601bb7a0 Add finalizeAsBase64Binary option to FixedBucketsHistogramAggregatorFactory (#7784)
* Add finalizeAsBase64Binary option to FixedBucketsHistogramAggregatorFactory

* Add finalizeAsBase64Binary option to ApproximateHistogramFactory

* Update approx histogram doc
2019-06-21 18:00:19 -07:00
Vadim Ogievetsky 6c8f9482c7 Web console: prettier code (#7939)
* prettier code

* update tests

* scss also

* more ,

* update pachage json prettier config
2019-06-21 16:52:33 -07:00
Clint Wylie 183d5be7ec
fix spotbugs failure in Expr.java (#7944) 2019-06-21 14:49:22 -07:00
Vadim Ogievetsky 2bc8e7c0e8 more acurate keyword auto complete (#7934) 2019-06-20 21:14:54 -07:00
Gian Merlino d046d5aa8d
Stalebot config adjustments. (#7936)
* Stalebot config adjustments.

- Different wordings for PRs and issues.
- Extend stale timeout for issues.
- Exempt "Proposal" label.
- Exempt issues in projects and issues with assignees.

* Fix copy/paste error.
2019-06-20 14:16:57 -07:00
Fangjin Yang ceade03213
Enable stalebot for issues (#7927)
* Enable stalebot for issues

* fix comments

* fix comments

* fix comments
2019-06-20 11:09:40 -07:00
Fokko Driesprong 48f20fe754 Add Spotbugs (#7894)
* Add Spotbugs

Exclude all the issues for now, so we can add them one by one.

(cherry picked from commit ceda4754dc8c703d1e0de85b48cd5f5409cfd5b7)

* Add additional rules to the list

* More rules

* More rules

* Add comments to the xml

* Move the spotbugs-exclude.xml to codestyle/
2019-06-20 21:06:52 +03:00
Clint Wylie 494b8ebe56 multi-value string column support for expressions (#7588)
* array support for expression language for multi-value string columns

* fix tests?

* fixes

* more tests

* fixes

* cleanup

* more better, more test

* ignore inspection

* license

* license fix

* inspection

* remove dumb import

* more better

* some comments

* add expr rewrite for arrayfn args for more magic, tests

* test stuff

* more tests

* fix test

* fix test

* castfunc can deal with arrays

* needs more empty array

* more tests, make cast to long array more forgiving

* refactor

* simplify ExprMacro Expr implementations with base classes in core

* oops

* more test

* use Shuttle for Parser.flatten, javadoc, cleanup

* fixes and more tests

* unused import

* fixes

* javadocs, cleanup, refactors

* fix imports

* more javadoc

* more javadoc

* more

* more javadocs, nonnullbydefault, minor refactor

* markdown fix

* adjustments

* more doc

* move initial filter out

* docs

* map empty arg lambda, apply function argument validation

* check function args at parse time instead of eval time

* more immutable

* more more immutable

* clarify grammar

* fix docs

* empty array is string test, we need a way to make arrays better maybe in the future, or define empty arrays as other types..
2019-06-19 13:57:37 -07:00
Vadim Ogievetsky 28eaa620a9 Web console: Data loader respects parse spec columns for data preview (#7922)
* small fixes in the data loader

* respect columns

* fix test
2019-06-19 12:48:31 -07:00
Fokko Driesprong 41f23b5120 Bump commons-compress from 1.16 to 1.18 (#7924) 2019-06-19 10:43:01 -07:00
Fokko Driesprong 0a6fbbbb80 Bump Apache Avro to 1.9.0 (#7772)
* Bump Apache Avro to 1.9.0

Apache Avro 1.9.0 brings a lot of new features:
* Deprecate Joda-Time in favor of Java8 JSR310 and setting it as default
* Remove support for Hadoop 1.x
* Move from Jackson 1.x to 2.9
* Add ZStandard Codec
* Lots of updates on the dependencies to fix CVE's
* Remove Jackson classes from public API
* Apache Avro is built by default with Java 8
* Apache Avro is compiled and tested with Java 11 to guarantee compatibility
* Apache Avro MapReduce is compiled and tested with Hadoop 3
* Apache Avro is now leaner, multiple dependencies were removed: guava, paranamer, commons-codec, and commons-logging
* Introduce JMH Performance Testing Framework
* Add Snappy support for C++ DataFile
* and many, many more!

* Add exclusions for Jackson
2019-06-19 03:31:18 -07:00
Himanshu 417fcef385 WorkerTaskManager to create disk files atomically and ignore task file corruption (#7917)
* WorkerTaskManager to create disk files atomically and ignore task file
corruptions

* fixing weird checkstyle lambda indentation issues
2019-06-18 09:18:43 -07:00
Xue Yu 20d1db9dff bump fastutil to 8.2.3 (#7920) 2019-06-18 09:17:34 -07:00
Clint Wylie 71997c16a2 switch links from druid.io to druid.apache.org (#7914)
* switch links from druid.io to druid.apache.org

* fix it
2019-06-18 09:06:27 -07:00
SandishKumarHN e80297efef Set Test timeout higher for robust performance (#7890) 2019-06-17 22:01:54 -07:00
Vadim Ogievetsky c44620a2e3 clean auto generated doc file also (#7911) 2019-06-17 18:01:29 -07:00
Vadim Ogievetsky 24dd4573da Added the web console to the quickstart tutorials and docs (#7863)
* added console to the quickstart tutorials

* feedback fixes

* feedback fixes

* more typo fixes

* moved reseting cluster section after load data

* update images

* stage -> step

* feedback fixes

* more feedback fixes
2019-06-17 18:00:54 -07:00
Vadim Ogievetsky df9cdcf13b Web console: added Query View metadata pane, more query details (#7905)
* refactor

* adding metadata hookup

* add scroll

* standerdize icons

* better loading state

* update snapshots

* fix lint

* do not mix up queryId and sqlQueryId
2019-06-17 13:14:42 -07:00
SandishKumarHN 01881e3a98 Use only com.google.errorprone.annotations.concurrent.GuardedBy, not javax.annotations.concurrent.GuardedBy (#7889) 2019-06-17 15:58:51 +02:00
Benedict Jin 4bceab0d7b
Bump jmh from 1.19 to 1.21 (#7876) 2019-06-17 19:27:09 +08:00
Benedict Jin fb7f8ec362
Bump RoaringBitmap from 0.8.0 to 0.8.6 (#7906) 2019-06-17 17:02:52 +08:00
litao91 daed1bc141 Fix HttpRemoteTaskRunnerConfig not loading (#7902) 2019-06-16 00:31:42 -07:00
Himanshu b3328b2785
endpoint to delete lookup tier and remove tier on last lookup deletion (#7852) 2019-06-15 17:55:50 -07:00
Jenny Zhu f603498e11 Web console: Add download path to SQL Query (#7898)
* adding download path to Query

* add more checker

* updated snap tests

* change after Vad's comments
2019-06-15 13:51:22 -07:00
Justin Borromeo 8e5003b01c Scan Doc Change (#7903) 2019-06-15 01:21:34 -07:00
Jihoon Son 3cd9a7507d Fix script for dependencies report for extensions (#7899) 2019-06-14 18:53:50 -07:00
Jihoon Son d00a9676b7 Set aws.region for unit tests automatically (#7868)
* Set aws.region for unit tests automatically

* Update README.template
2019-06-14 15:34:21 -07:00
Jihoon Son a648e1548d Add support of --exclude-extension argument for dependency report script (#7786) 2019-06-14 15:18:59 -07:00
mcbrewster 5e6d833fd9 Web-Console: tail log based on task status (#7888)
* clean up

* fix declarations

* spacing
2019-06-14 14:37:21 -07:00
Fokko Driesprong f581118f05 Remove Apache Pig from the tests (#7810)
* Remove Apache Pig from the tests

* Remove the Pig specific part

* Fix the Checkstyle issues

* Cleanup a bit

* Add an additional test

* Revert the abstract class
2019-06-14 14:18:58 -07:00
Sashidhar Thallam 3bee6adcf7 Use map.putIfAbsent() or map.computeIfAbsent() as appropriate instead of containsKey() + put() (#7764)
* https://github.com/apache/incubator-druid/issues/7316 Use Map.putIfAbsent() instead of containsKey() + put()

* fixing indentation

* Using map.computeIfAbsent() instead of map.putIfAbsent() where appropriate

* fixing checkstyle

* Changing the recommendation text

* Reverting auto changes made by IDE

* Implementing recommendation: A ConcurrentHashMap on which computeIfAbsent() is called should be assigned into variables of ConcurrentHashMap type, not ConcurrentMap

* Removing unused import
2019-06-14 17:59:36 +02:00
Vadim Ogievetsky 05346a9e0c allow semicolon with comment (#7892) 2019-06-14 07:53:19 -07:00
Xue Yu 456a3654ce add PolygonBound and missing extentions list doc (#7885) 2019-06-13 12:03:58 -07:00
mcbrewster 0763585208 Web-console: update supervisors table (#7799)
* add new columns

* fix syling

* fix spaces

* update snapshots

* fix Spelling

* fix capitalization

* reorder action dialog

* set color using state

* fix snapshots

* fix array

* update snapshots

* remove extra columns

* update snapshots

* update snapshots

* fixes

* update snapshots

* use cell

* fix spacing

* update snapshot
2019-06-11 13:06:35 -07:00
Clint Wylie 8117222da3 use right port for kafka tutorial, reinfoce that tutorials assume you are using micro-quickstart single-server configuration (#7862) 2019-06-11 08:50:52 -07:00
Xue Yu ce591d1457 Support var_pop, var_samp, stddev_pop and stddev_samp etc in sql (#7801)
* support var_pop, stddev_pop etc in sql

* fix sql compatible

* rebase on master

* update doc
2019-06-10 09:40:09 -07:00
Vadim Ogievetsky c612ddc0f4 Support more JODA time formats (#7857) 2019-06-09 20:08:29 -07:00
Vadim Ogievetsky c3d230b354 Web console: make the data loader more responsive when sampling data with many columns (#7856)
* Don't rerender data loader tables

* removed unused imports

* removed extra console.log

* fix typo
2019-06-09 10:44:13 -07:00
Vadim Ogievetsky 53b9fc4eb0 switch to use exInterlop (#7853) 2019-06-07 22:27:19 -07:00
Clint Wylie 3fbb0a5e00 Supervisor list api with states and health (#7839)
* allow optionally listing all supervisors with their state and health

* docs

* add state to full

* clean

* casing

* format

* spelling
2019-06-07 16:26:33 -07:00
Vadim Ogievetsky 248e075e24 Web console: Simplify task and supervisor detail displays (#7848)
* simplify detail display

* update tests
2019-06-07 11:02:45 -07:00
Surekha ea752ef562 Optimize overshadowed segments computation (#7595)
* Move the overshadowed segment computation to SQLMetadataSegmentManager's poll

* rename method in MetadataSegmentManager

* Fix tests

* PR comments

* PR comments

* PR comments

* fix indentation

* fix tests

*  fix test

*  add test for SegmentWithOvershadowedStatus serde format

* PR comments

* PR comments

* fix test

* remove snapshot updates outside poll

* PR comments

* PR comments

* PR comments

*  removed unused import
2019-06-07 19:15:54 +02:00
Vadim Ogievetsky 061b465d34 Web console: Normalize ingestion spec type (#7847)
* Normalize ingestion spec types

* also normalize on direct spec change
2019-06-07 08:32:40 -07:00
Vadim Ogievetsky df01335789 Web console: prevent the time parse view from going into a bad state (#7846)
* prevent the time parse view from going into a bad state

* add test
2019-06-07 08:32:10 -07:00
Vadim Ogievetsky ca5afd29b8 switch the web console to use react pure components (#7845) 2019-06-07 08:31:46 -07:00
Vadim Ogievetsky 0ca867efbf less aggressive default value (#7844) 2019-06-07 08:31:21 -07:00
Clint Wylie ee0d4ea589 add bloom filter fallback aggregator when types are unknown (#7719) 2019-06-06 14:39:32 -07:00