Commit Graph

76 Commits

Author SHA1 Message Date
Jihoon Son abcf624a2e
Bump up jackson-databind to 2.10.5.1 (#10655)
* Bump up jackson version to 2.10.5.1

* only jackson-databind

* license
2020-12-09 13:54:47 -08:00
frank chen c410648630
fix injection failure of StorageLocationSelectorStrategy objects (#10363)
* fix to allow customer storage location selector strategy

* add test cases to check instance of selector strategy

* update doc

* code format

* resolve code review comments

* inject StorageLocation

* fix CI

* fix mismatched license item reported by CI

* change property path from druid.segmentCache.locationSelectorStrategy.type to druid.segmentCache.locationSelector.strategy

* using a helper method to bind to correct property path
2020-12-08 09:48:31 -08:00
Vadim Ogievetsky e3f7217546
Web console: Improve the handling of extreme data (funky datasources, longs) (#10641)
* better API escape

* fix escaping issue, bigints

* update licenses

* fix align

* do not show Query with SQL if no SQL

* add prettify script

* update dev readme

* add ordering to the datasource list

* add ordering to supervisor table
2020-12-08 09:25:14 -08:00
Suneet Saldanha c94be8a945
Revert "Update google client libraries (#10536)" (#10599)
This reverts commit 4537016cad.
2020-12-03 20:14:52 +05:30
Vadim Ogievetsky 5b06c7a3a9
Web console: improve how code is imported, use API instance (#10597)
* fix imports

* clean up imports

* update DQT to fix escaping
2020-12-01 13:16:14 -08:00
Ayush Kulshrestha d0c2ede50c
Added CronScheduler support as a proof to clock drift while emitting metrics (#10448)
Co-authored-by: Ayush Kulshrestha <ayush.kulshrestha@miqdigital.com>
2020-11-25 12:31:38 +01:00
Nishant Bangarwa 4537016cad
Update google client libraries (#10536)
modify license.yaml

Update google oauth client version
2020-11-20 15:23:30 -08:00
Suneet Saldanha 6c8a77b7a9
Bump jetty to latest version (#10563)
This addresses CVE-2020-27216 which was flagged by the security vulnerability
job.
2020-11-09 08:51:36 -08:00
Vadim Ogievetsky f391e89653
Web console: refresh and tighten up the console styles 💅💫 (#10515)
* no hardcoded dark mode

* update blueprint

* style pass

* add open sans font

* fix public path

* update licenses

* add comment

* allow zero bsd

* add new lines

* update tests

* remove default mods

* use variables

* fix background

* fix query init style

* pad fracional part

* paginate braces

* update to latest blueprint, fix column tree

* Smart query limit -> Auto limit

* adjust gutter color for dark mode

* fix spelling

* update snapshot

Co-authored-by: asherbitter <ofir@imply.io>
2020-10-20 22:11:29 -07:00
Abhishek Agarwal d057c5149f
Fix the offset setting in GoogleStorage#get (#10449)
* Fix the offset in get of GCP object

* upgrade compute dependency

* fix version

* review comments

* missed
2020-10-01 08:38:58 -07:00
Vadim Ogievetsky 6c5c86d800
Web console: fix lookup edit dialog, allow column renaming (#10406)
* column rename

* update licenses file

* remove empty file

* update license file

* move comment
2020-09-20 14:10:05 -07:00
Igor Dvorzhak d0ee2e3a48
Upgrade ORC to 1.5.10 version (#10291) 2020-09-18 13:38:45 -07:00
Vadim Ogievetsky 3c8eacb2d4
Web console: improve query manager (convert to React hook) (#10360)
* Better query running

* update licenses

* update tests

* updated tests v2

* fade in cancel

* add exemplary tests

* update mkcomp

* fix inconsistent state update

* remove lastParsedQuery

* work if not a valid literal

* remove unused params

* fix licenses

* better state update

* get error message

* isEmpty tidy

* add tests around error message highlighting

* pull live query selector into a component

* add LiveQueryModeSelector tests

* update snapshots
2020-09-11 19:42:50 -07:00
Xavier Léauté 225490474d
Update Kafka dependencies to 2.6.0 (#10286)
* update Kafka dependencies to Kafka 2.6.0
* switch to Scala 2.13 build of Kafka
* update integration tests
* update Kafka tutorial
2020-08-15 07:56:40 -07:00
Vadim Ogievetsky 6d8799f2df
Update QueryView to use latest DruidQueryToolkit (#10201)
* Update to latest DruidQueryToolkit

* add THEN keyword

* do not crash on invalid JSON
2020-07-23 22:45:01 -07:00
Richard Startin e363b1cd20
Update RoaringBitmap to 0.9.0 (#9987) 2020-07-23 19:29:25 -07:00
Gian Merlino eeaf609fc0
Update Jetty to 9.4.30.v20200611. (#10098)
* Update Jetty to 9.4.30.v20200611.

This is the latest version currently available in the 9.4.x line.

* Various adjustments.

* Class name fixes.

* Remove unused HttpClientModule code.

* Add coverage suppressions.

* Another coverage suppression.

* Fix wildcards.
2020-07-07 14:24:02 -07:00
Clint Wylie ec1f443a5c
update avatica to handle additional character sets over jdbc (#10074)
* update avatica to handle additional character sets over jdbc

* update license yaml, fix test

* oops
2020-06-24 19:58:34 -07:00
Chi Cao Minh 67669b4ad4
Fix CVE-2020-13602 (#10024)
Upgrade postgres jdbc driver to latest version to address CVE, which was
fixed in 42.2.13.
2020-06-11 17:30:13 -07:00
Xavier Léauté 65280a6953
update kafka client version to 2.5.0 (#9902)
- remove dependency on deprecated internal Kafka classes
- keep LZ4 version in line with the version shipped with Kafka
2020-05-27 13:20:32 -07:00
Vadim Ogievetsky 63baa29ad1
Fix web console query view crashing on simple query (#9897)
* only parse full queries

* upgraded sql parser
2020-05-21 12:57:07 -07:00
Clint Wylie 6bc1d1b33f
fix license registry for com.nimbusds lang-tag (#9860) 2020-05-13 09:18:18 -07:00
Jihoon Son 9ab49b34db
Update notice; fix version of druid-query-toolkit (#9799) 2020-05-02 20:00:43 -07:00
Francesco Nidito e7e41e3a36
Adding support for autoscaling in GCE (#8987)
* Adding support for autoscaling in GCE

* adding extra google deps also in gce pom

* fix link in doc

* remove unused deps

* adding terms to spelling file

* version in pom 0.17.0-incubating-SNAPSHOT --> 0.18.0-SNAPSHOT

* GCEXyz -> GceXyz in naming for consistency

* add preconditions

* add VisibleForTesting annotation

* typos in comments

* use StringUtils.format instead of String.format

* use custom exception instead of exit

* factorize interval time between retries

* making literal value a constant

* iter all network interfaces

* use provided on google (non api) deps

* adding missing dep

* removing unneded this and use Objects methods instead o 3-way if in hash and comparison

* adding import

* adding retries around getRunningInstances and adding limit for operation end waiting

* refactor GceEnvironmentConfig.hashCode

* 0.18.0-SNAPSHOT -> 0.19.0-SNAPSHOT

* removing unused config

* adding tests to hash and equals

* adding nullable to waitForOperationEnd

* adding testTerminate

* adding unit tests for createComputeService

* increasing retries in unrelated integration-test to prevent sporadic failure (hopefully)

* reverting queryResponseTemplate change

* adding comment for Compute.Builder.build() returning null
2020-04-28 03:13:39 -07:00
Clint Wylie fc5383cd00
revert datasketches-java version to 1.1.0-incubating until new version is released (#9751)
* revert datasketches-java version to 1.1.0-incubating until fix is in place

* fix tests

* checkstyle
2020-04-24 12:52:12 -07:00
Jihoon Son 1b60148ec6
Missing license changes for sources in licenses.yaml (#9678) 2020-04-10 23:06:33 -07:00
Chi Cao Minh eb45981b60
Upgrade netty 4 to fix CVE-2020-11612 (#9651) 2020-04-09 13:26:14 -07:00
Maytas Monsereenusorn b95a1b9878
Fix NPE in RemoteTaskRunner event handler causes JVM shutdown (#9610)
* Fix NPE in RemoteTaskRunner event handler causes JVM shutdown

* address comments

* fix compile

* fix checkstyle

* fix lgtm

* fix merge

* fix test

* fix tests

* change scope

* address comments

* address comments
2020-04-07 14:53:51 -07:00
bolkedebruin 2d99966933
Add Apache Ranger Authorization (#9579) 2020-04-04 18:02:24 +02:00
Jonathan Wei dbaabdd247
Fix for [CVE-2020-1958]: Apache Druid LDAP injection vulnerability (#9600) 2020-04-01 14:52:01 -07:00
Himanshu 5604ac7963
druid extension for OpenID Connect auth using pac4j lib (#8992)
* druid pac4j security extension for OpenID Connect OAuth 2.0 authentication

* update version in druid-pac4j pom

* introducing unauthorized resource filter

* authenticated but authorized /unified-webconsole.html

* use httpReq.getRequestURI() for matching callback path

* add documentation

* minor doc addition

* licesne file updates

* make dependency analyze succeed

* fix doc build

* hopefully fixes doc build

* hopefully fixes license check build

* yet another try on fixing license build

* revert unintentional changes to website folder

* update version to 0.18.0-SNAPSHOT

* check session and its expiry on each request

* add crypto service

* code for encrypting the cookie

* update doc with cookiePassphrase

* update license yaml

* make sessionstore in Pac4jFilter private non static

* make Pac4jFilter fields final

* okta: use sha256 for hmac

* remove incubating

* add UTs for crypto util and session store impl

* use standard charsets

* add license header

* remove unused file

* add org.objenesis.objenesis to license.yaml

* a bit of nit changes  in CryptoService  and embedding EncryptionResult for clarity

* rename alg  to cipherAlgName

* take cipher alg name, mode and padding as input

* add java doc  for CryptoService  and make it more understandable

* another  UT for CryptoService

* cache pac4j Config

* use generics clearly in Pac4jSessionStore

* update cookiePassphrase doc to mention PasswordProvider

* mark stuff Nullable where appropriate in Pac4jSessionStore

* update doc to mention jdbc

* add error log on reaching callback resource

* javadoc  for Pac4jCallbackResource

* introduce NOOP_HTTP_ACTION_ADAPTER

* add correct module name in license file

* correct extensions folder name in licenses.yaml

* replace druid-kubernetes-extensions to druid-pac4j

* cache SecureRandom instance

* rename UnauthorizedResourceFilter to AuthenticationOnlyResourceFilter
2020-03-23 18:15:45 -07:00
Clint Wylie 8b9fe6f584
query laning and load shedding (#9407)
* prototype

* merge QueryScheduler and QueryManager

* everything in its right place

* adjustments

* docs

* fixes

* doc fixes

* use resilience4j instead of semaphore

* more tests

* simplify

* checkstyle

* spelling

* oops heh

* remove unused

* simplify

* concurrency tests

* add SqlResource tests, refactor error response

* add json config tests

* use LongAdder instead of AtomicLong

* remove test only stuffs from scheduler

* javadocs, etc

* style

* partial review stuffs

* adjust

* review stuffs

* more javadoc

* error response documentation

* spelling

* preserve user specified lane for NoSchedulingStrategy

* more test, why not

* doc adjustment

* style

* missed review for make a thing a constant

* fixes and tests

* fix test

* Update docs/configuration/index.md

Co-Authored-By: sthetland <steve.hetland@imply.io>

* doc update

Co-authored-by: sthetland <steve.hetland@imply.io>
2020-03-10 02:57:16 -07:00
zachjsh d771b42ed1
Move Azure extension into Core (#9394)
* Move Azure extension into Core

Moving the azure extension into Core.

* * Fix build failure

* * Add The MIT License (MIT) to list of compatible licenses

* * Address review comments

* * change reference to contrib azure to core azure

* * Fix spelling mistakes.
2020-02-25 17:49:16 -08:00
Chi Cao Minh 7fc99ee206
Add common optional dependencies for extensions (#9399)
* Add common optional dependencies for extensions

Include hadoop-aws and postgres JDBC connector jar to improve
out-of-the-box experience for extensions. The mysql JDBC connector jar
is not bundled as it is GPL.

* Update docs

* Fix typo
2020-02-25 00:04:00 -08:00
Fokko Driesprong 806dfe6de6
Bump Apache Avro to 1.9.2 (#9381)
* Bump Apache Avro 1.9.2

Bugfixes that where discovered in other projects

* Update missing license
2020-02-24 10:04:22 +01:00
zachjsh 5c202343c9
implement Azure InputSource reader and deprecate Azure FireHose (#9306)
* IMPLY-1946: Improve code quality and unit test coverage of the Azure extension

* Update unit tests to increase test coverage for the extension
* Clean up any messy code
* Enfore code coverage as part of tests.

* * Update azure extension pom to remove unnecessary things
* update jacoco thresholds

* * updgrade version of azure-storage library version uses to
  most upto-date version

* implement Azure InputSource reader and deprecate Azure FireHose

* implement azure InputSource reader
* deprecate Azure FireHose implementation

* * exclude common libraries that are included from druid core

* Implement more of Azure input source.

* * Add tests

* * Add more tests

* * deprecate azure firehose

* * added more tests

* * rollback fix for google cloud batch ingestion bug. Will be
  fixed in another PR.

* * Added javadocs for all azure related classes
* Addressed review comments

* * Remove dependency on org.apache.commons:commons-collections4
* Fix LGTM warnings
* Add com.google.inject.extensions:guice-assistedinject to licenses

* * rename classes as suggested in review comments

* * Address review comments

* * Address review comments

* * Address review comments
2020-02-11 17:41:58 -08:00
Alexander Saydakov 54bc4dc71b
use the latest datasketches-java-1.2.0-incubating (#9269)
Co-authored-by: Gian Merlino <gianmerlino@gmail.com>
2020-02-05 08:48:38 -08:00
Suneet Saldanha 33a97dfaae
Guicify druid sql module (#9279)
* Guicify druid sql module

Break up the SQLModule in to smaller modules and provide a binding that
modules can use to register schemas with druid sql.

* fix some tests

* address code review

* tests compile

* Working tests

* Add all the tests

* fix up licenses and dependencies

* add calcite dependency to druid-benchmarks

* tests pass

* rename the schemas
2020-02-04 11:33:48 -08:00
zachjsh 74ac9151c9
Fix / suppress netty CVEs CVE-2019-20445 and CVE-2019-20444 (#9300)
* Suppress netty 3 vulnerabilites and upgrade netty 4 version

* Upgrade netty 4 version to fix vulnerabilities CVE-2019-20445
  and CVE-2019-20444
* suppress these CVEs for netty 3

* * simplify suppression xml file
* update licenses file with new version of netty

* * fix type in licenses.yaml
2020-01-31 14:51:54 -08:00
Clint Wylie c6c8b80644 fix build by updating kafka client to 2.2.2 for CVE-2019-12399 (#9259)
* fix build by updating kafka client to 2.2.2 for CVE-2019-12399

* one kafka version to rule them all

* notice
2020-01-27 11:07:02 -08:00
Fokko Driesprong 12b84cfb33
Bump Jackson to 2.10.2 (#9173) 2020-01-17 11:39:32 +01:00
Fokko Driesprong 486c0fd149 Bump Apache Parquet to 1.11.0 (#9129)
* Bump Parquet to 1.11.0

* Update licenses.yaml

* Add parquet-format-structures
2020-01-16 16:24:25 -08:00
Jonathan Wei aa539177ec De-incubation cleanup in code, docs, packaging (#9108)
* De-incubation cleanup in code, docs, packaging

* remove unused docs script
2020-01-03 12:33:19 -05:00
Chi Cao Minh 3de7ab8523 DataSketches jars in core (#9003)
Having DataSketches jars in core will allow potential improvements, for
example:
- Provide an alternative implementation of HLL:
  https://datasketches.github.io/docs/HLL/HllSketchVsDruidHyperLogLogCollector.html
- Range partitioning for native parallel batch indexing without having
  the user load extensions on the classpath

Dev mailing list discussion:
https://lists.apache.org/thread.html/301410d71ff799cf616bf17c4ebcf9999fc30829f5fa62909f403e6c%40%3Cdev.druid.apache.org%3E
2019-12-10 14:02:34 -08:00
Chi Cao Minh af74acaa85 Address security vulnerabilities CVSS >= 7 (#8980)
* Address security vulnerabilities CVSS >= 7

Update dependencies to address security vulnerabilities with CVSS scores
of 7 or higher. A new Travis CI job is added to prevent new
high/critical security vulnerabilities from being added.

Updated dependencies:
- api-util 1.0.0 -> 1.0.3
- jackson 2.9.10 -> 2.10.1
- kafka 2.1.0 -> 2.1.1
- libthrift 0.10.0 -> 0.13.0
- protobuf 3.2.0 -> 3.11.0

The following high/critical security vulnerabilities are currently
suppressed (so that the new Travis CI job can be added now) and are left
as future work to fix:
- hibernate-validator:5.2.5
- jackson-mapper-asl:1.9.13
- libthrift:0.6.1
- netty:3.10.6
- nimbus-jose-jwt:4.41.1

* Rename EDL1 license file

* Fix inspection errors
2019-12-05 14:34:35 -08:00
Jonathan Wei 55ecaafff0 Add licenses.yaml entry for Wikipedia sample data (#8968) 2019-11-28 11:41:42 -08:00
Chi Cao Minh fba876b607 Update jackson to 2.9.10 (#8940)
Addresses security vulnerabilities:

- sonatype-2016-0397:
  https://github.com/FasterXML/jackson-core/issues/315

- sonatype-2017-0355:
  https://github.com/FasterXML/jackson-core/pull/322
2019-11-26 21:41:14 -08:00
Alexander Saydakov 4a9da3f3fc use the latest release of datasketches (#8647)
* use the latest release of datasketches

* added datasketches-memory dependency

* updated datasketches entries

* use datasketches-memory-1.2.0

* updated dependencies

* fixed tests
2019-11-25 19:45:51 -08:00
Jonathan Wei dc6178d1f2 Upgrade Calcite to 1.21 (#8566)
* Upgrade Calcite to 1.21

* Checkstyle, test fix'

* Exclude calcite yaml deps, update license.yaml

* Add method for exception chain handling

* Checkstyle

* PR comments, Add outer limit context flag

* Revert project settings change

* Update subquery test comment

* Checkstyle fix

* Fix test in sql compat mode

* Fix test

* Fix dependency analysis

* Address PR comments

* Checkstyle

* Adjust testSelectStarFromSelectSingleColumnWithLimitDescending
2019-11-20 21:22:55 -08:00
Chi Cao Minh 8365bdf62a Address security vulnerabilities (#8878)
* Address security vulnerabilities

Security vulnerabilities addressed by upgrading 3rd party libs:

- Upgrade avro-ipc to 1.9.1
  - sonatype-2019-0115
- Upgrade caffeine to 2.8.0
  - sonatype-2019-0282
- Upgrade commons-beanutils to 1.9.4
  - CVE-2014-0114
- Upgrade commons-codec to 1.13
  - sonatype-2012-0050
- Upgrade commons-compress to 1.19
  - CVE-2019-12402
  - sonatype-2018-0293
- Upgrade hadoop-common to 2.8.5
  - CVE-2018-11767
- Upgrade hadoop-mapreduce-client-core to 2.8.5
  - CVE-2017-3166
- Upgrade hibernate-validator to 5.2.5
  - CVE-2017-7536
- Upgrade httpclient to 4.5.10
  - sonatype-2017-0359
- Upgrade icu4j to 55.1
  - CVE-2014-8147
- Upgrade jackson-databind to 2.6.7.3:
  - CVE-2017-7525
- Upgrade jetty-http to 9.4.12:
  - CVE-2017-7657
  - CVE-2017-7658
  - CVE-2017-7656
  - CVE-2018-12545
- Upgrade log4j-core to 2.8.2
  - CVE-2017-5645:
- Upgrade netty to 3.10.6
  - CVE-2015-2156
- Upgrade netty-common to 4.1.42
  - CVE-2019-9518
- Upgrade netty-codec-http to 4.1.42
  - CVE-2019-16869
- Upgrade nimbus-jose-jwt to 4.41.1
  - CVE-2017-12972
  - CVE-2017-12974
- Upgrade plexus-utils to 3.0.24
  - CVE-2017-1000487
  - sonatype-2015-0173
  - sonatype-2016-0398
- Upgrade postgresql to 42.2.8
  - CVE-2018-10936

Note that if users are using JDBC lookups with postgres, they may need
to update the JDBC jar used by the lookup extension.

* Fix license for postgresql
2019-11-19 09:14:33 -08:00