Commit Graph

119 Commits

Author SHA1 Message Date
William Hyun 3abca73ee8
Upgrade ORC to 1.7.1 (#11919) 2021-11-15 09:13:03 -08:00
Karan Kumar 90640bb316
Support for hadoop 3 via maven profiles (#11794)
Add support for hadoop 3 profiles . Most of the details are captured in #11791 .
We use a combination of maven profiles and resource filtering to achieve this. Hadoop2 is supported by default and a new maven profile with the name hadoop3 is created. This will allow the user to choose the profile which is best suited for the use case.
2021-10-30 22:46:24 +05:30
Jihoon Son 07a232d7b4
Bump netty4 to 4.1.68; suppress CVE-2021-37136 and CVE-2021-37137 for netty3 (#11844)
* bump netty4 to 4.1.68

* suppress CVE-2021-37136 and CVE-2021-37137 for netty3

* license
2021-10-25 21:09:15 -07:00
Alexander Saydakov 8cf1cbc4a9
latest datasketches-java and datasketches-memory (#11773)
* latest datasketches-java and datasketches-memory

* updated versions of datasketches-java and datasketches-memory

Co-authored-by: AlexanderSaydakov <AlexanderSaydakov@users.noreply.github.com>
2021-10-19 23:42:30 -07:00
Xavier Léauté bc3b038712
Update Apache Kafka client libraries to 3.0.0 (#11735)
Release notes:
https://downloads.apache.org/kafka/3.0.0/RELEASE_NOTES.html
https://blogs.apache.org/kafka/entry/what-s-new-in-apache6
2021-10-05 10:23:19 -07:00
William Hyun 9bff6bd70e
Upgrade ORC to 1.7.0 (#11726)
* Upgrade ORC to 1.7.0

* address comments

* address comments

* Add import
2021-09-27 13:20:09 -07:00
dependabot[bot] 776ddf76f4
Bump parquet.version from 1.11.1 to 1.12.0 (#11346)
* Bump parquet.version from 1.11.1 to 1.12.0

Bumps `parquet.version` from 1.11.1 to 1.12.0.

Updates `parquet-column` from 1.11.1 to 1.12.0
- [Release notes](https://github.com/apache/parquet-mr/releases)
- [Changelog](https://github.com/apache/parquet-mr/blob/master/CHANGES.md)
- [Commits](https://github.com/apache/parquet-mr/compare/apache-parquet-1.11.1...apache-parquet-1.12.0)

Updates `parquet-avro` from 1.11.1 to 1.12.0
- [Release notes](https://github.com/apache/parquet-mr/releases)
- [Changelog](https://github.com/apache/parquet-mr/blob/master/CHANGES.md)
- [Commits](https://github.com/apache/parquet-mr/compare/apache-parquet-1.11.1...apache-parquet-1.12.0)

Updates `parquet-hadoop` from 1.11.1 to 1.12.0
- [Release notes](https://github.com/apache/parquet-mr/releases)
- [Changelog](https://github.com/apache/parquet-mr/blob/master/CHANGES.md)
- [Commits](https://github.com/apache/parquet-mr/compare/apache-parquet-1.11.1...apache-parquet-1.12.0)

---
updated-dependencies:
- dependency-name: org.apache.parquet:parquet-column
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.apache.parquet:parquet-avro
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.apache.parquet:parquet-hadoop
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update license

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Suneet Saldanha <suneet@apache.org>
2021-08-13 19:17:57 -07:00
dependabot[bot] 2e850b5655
Bump aws.sdk.version from 1.11.884 to 1.12.37 (#11367)
* Bump aws.sdk.version from 1.11.884 to 1.12.8

Bumps `aws.sdk.version` from 1.11.884 to 1.12.8.

Updates `aws-java-sdk-core` from 1.11.884 to 1.12.8
- [Release notes](https://github.com/aws/aws-sdk-java/releases)
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.11.884...1.12.8)

Updates `aws-java-sdk-ec2` from 1.11.884 to 1.12.8
- [Release notes](https://github.com/aws/aws-sdk-java/releases)
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.11.884...1.12.8)

Updates `aws-java-sdk-s3` from 1.11.884 to 1.12.8
- [Release notes](https://github.com/aws/aws-sdk-java/releases)
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.11.884...1.12.8)

Updates `aws-java-sdk-sts` from 1.11.884 to 1.12.8
- [Release notes](https://github.com/aws/aws-sdk-java/releases)
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.11.884...1.12.8)

Updates `aws-java-sdk-kinesis` from 1.11.884 to 1.12.8
- [Release notes](https://github.com/aws/aws-sdk-java/releases)
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.11.884...1.12.8)

Updates `aws-java-sdk-rds` from 1.11.884 to 1.12.8
- [Release notes](https://github.com/aws/aws-sdk-java/releases)
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.11.884...1.12.8)

---
updated-dependencies:
- dependency-name: com.amazonaws:aws-java-sdk-core
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: com.amazonaws:aws-java-sdk-ec2
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: com.amazonaws:aws-java-sdk-s3
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: com.amazonaws:aws-java-sdk-sts
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: com.amazonaws:aws-java-sdk-kinesis
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: com.amazonaws:aws-java-sdk-rds
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix license and bump to latest aws

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Suneet Saldanha <suneet@apache.org>
2021-08-01 00:12:57 -07:00
Dongjoon Hyun dbed4424b5
Upgrade ORC to 1.6.9 (#11518) 2021-07-31 23:33:03 -07:00
Xavier Léauté 4bca7f014e
update error-prone to 2.8.0 with fix for crashing check (#11494)
* error-prone 2.8.0 fixes https://github.com/google/error-prone/issues/2396
* fix for a few ignored return values
* fix unknown args in sub-modules
2021-07-29 09:13:46 -07:00
Suneet Saldanha 3f456fe305
Address CVE-2021-35515 CVE-2021-36090 (#11496)
* Address CVE-2021-35515 CVE-2021-36090

Bump commons-compress to deal with new CVEs

* fix licenses
2021-07-26 14:54:32 -07:00
Dongjoon Hyun 5037493e45
Bump commons-io to 2.11.0 (#11460)
* Bump commons-io to 2.11.0

* Address comments

* Remove try catch

* Fix checkstyle
2021-07-19 15:47:14 -07:00
Vadim Ogievetsky 377b5e708c
Web console: Data loading walkthrough fixes (#11416)
* fix quotes

* fix sql doc parsing

* prevent array-input from losing position while the user is typing

* make group filter click-to-filterable

* fix casing bug in exact table search

* do not sort columns in smaples

* can bypass transform step

* fixed string json parsing

* improve PartitionMessage

* better error messages

* feedback fixes

* tool to order dimensions in schema view
2021-07-10 07:56:50 -07:00
zachjsh 8037a54525
revert commons-io to 2.6 (#11392)
* * revert commons-io to 2.6

* * fix failing tests
2021-06-29 23:04:38 -07:00
Vadim Ogievetsky f56a5b9ba2
Web console: Better hotkeys and library upgrades (#11365)
* improve hotkeys

* fix test name

* refactor explain dialog

* explain tests

* small fixes

* update to popover2

* use resize sensor 2

* misc cleanup
2021-06-17 18:24:29 -07:00
Xavier Léauté a1c20d7457
update jackson dependencies to use bom (#11353)
Switching to the bom dependency declaration simplifies managing jackson
dependencies. It also removes the need to override individual library
versions for CVE fixes, since the bom takes care of that internally.

This change aligns our jackson dependency versions on 2.10.5(.x):
- updates jackson libraries from 2.10.2 to 2.10.5
- jackson-databind remains at 2.10.5.1 as defined in the bom

Release notes: https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.10
2021-06-16 18:37:30 -07:00
Xavier Léauté 712f2a5d00
upgrade error-prone to 2.7.1 and support checks with Java 11+ (#11363)
* upgrade error-prone to 2.7.1 and support checks with Java 11+

- upgrade error-prone to 2.7.1
- support running error-prone with Java 11 and above using -Xplugin
  instead of custom compiler
- add compiler arguments to ignore warnings/errors in Java 15/16
- introduce strictCompile property to enable strict profiles since we
  now need multiple strict profiles for Java 8
- properly exclude all generated source files from error-prone
- fix druid-processing overriding annotation processors from parent pom
- fix druid-core disabling most non-default checks
- align plugin and annotation errorprone versions
- fix / suppress additional issues found by error-prone:
  * fix bug in SeekableStreamSupervisor initializing ArrayList size with
    the taskGroupdId
  * fix missing @Override annotations
- remove outdated compiler plugin in benchmarks
- remove deleted ParameterPackage error-prone rule
- re-enable checks on benchmark module as well

* fix IntelliJ inspections

* disable LongFloatConversion due to bug in error-prone with JDK 8

* add comment about InsecureCrypto
2021-06-16 12:55:34 -07:00
John Gozde ac9b1f81b4
Web console: Remove support for IE11 and other older browsers (#11357)
* Use common browserlist and update to drop IE11

* Change TypeScript target to ES2016

* Update browserslist for "supports es6" support

* Show a warning if accessed from an unsupported browser

* Inline browser-update styles; detect SyntaxErrors too

* Better wording

* Upgrade to the latest Blueprint

* Refactor RunButton to be FC, use useHotkeys

* Remove dead license

* Update snapshots

* Address feedback

* Wording

Co-authored-by: Vadim Ogievetsky <vadimon@gmail.com>

Co-authored-by: Vadim Ogievetsky <vadimon@gmail.com>
2021-06-10 19:05:40 -07:00
dependabot[bot] 167044f715
Bump fastutil from 8.2.3 to 8.5.4 (#11347)
* Bump fastutil from 8.2.3 to 8.5.4

Bumps [fastutil](https://github.com/vigna/fastutil) from 8.2.3 to 8.5.4.
- [Release notes](https://github.com/vigna/fastutil/releases)
- [Changelog](https://github.com/vigna/fastutil/blob/master/CHANGES)
- [Commits](https://github.com/vigna/fastutil/compare/8.2.3...8.5.4)

---
updated-dependencies:
- dependency-name: it.unimi.dsi:fastutil
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* update licenses.yaml
* update maven dependency list for -core and -extra libraries to pass maven dependency checks

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2021-06-10 07:43:18 -07:00
dependabot[bot] be10a236d5
Bump commons-io from 2.6 to 2.9.0 (#11338)
* Bump commons-io from 2.6 to 2.9.0

Bumps commons-io from 2.6 to 2.9.0.

---
updated-dependencies:
- dependency-name: commons-io:commons-io
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* update licenses.yaml to reflect version bumps
* fix tests relying on specific log messages

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2021-06-08 10:02:59 -07:00
Vadim Ogievetsky 0c5d1c9725
Web console: add more query fixing auto suggestions (#11203)
* add more query fixing auto suggestions

* update query gen

* update toolkit

* update licenses

* fix funky quotes

* funky => fancy

* revert engine change

* separate web-console and website npm and node deps
2021-06-04 09:29:00 -07:00
Xavier Léauté b517c3339b
remove ZooKeeper 3.4 support + pass tests with Java 15 (#11073)
With this change, Druid will only support ZooKeeper 3.5.x and later.

In order to support Java 15 we need to switch to ZK 3.5.x client libraries and drop support for ZK 3.4.x
(see #10780 for the detailed reasons) 

* remove ZooKeeper 3.4.x compatibility
* exclude additional ZK 3.5.x netty dependencies to ensure we use our version
* keep ZooKeeper version used for integration tests in sync with client library version
* remove the need to specify ZK version at runtime for docker
* add support to run integration tests with JDK 15
* build and run unit tests with Java 15 in travis
2021-05-25 12:49:49 -07:00
Atul Mohan ac95f99d59
Upgrade jetty version (#11207) 2021-05-06 18:21:36 +08:00
Xavier Léauté 0296f20551
upgrade Apache Kafka to 2.8.0 (#11139)
* upgrade to Apache Kafka 2.8.0 (release notes:
  https://downloads.apache.org/kafka/2.8.0/RELEASE_NOTES.html)
* pass Kafka version as a Docker argument in integration tests
  to keep in sync with maven version
* fix use of internal Kafka APIs in integration tests
2021-04-24 08:27:07 -07:00
Vadim Ogievetsky 4caa221d72
Web console: Better inline docs (#11128)
* better highlight

* better highlighting

* add spec
2021-04-19 14:36:53 -07:00
John Gozde fdc3c2f362
Web console: update dev dependencies (#11119)
* Update some dev dependencies, prettify, tslint-fix

* Sort tsconfig keys for easy comparison

* Set noImplicitThis

* Slightly more accurate types

* Bump Jest and related

* Bump react to latest on v16

* Bump node-sass, sass-loader for node14 support

* Remove node-sass-chokidar (unused)

* More unused dependencies

* Fix blueprint imports

* Webpack 5

* Update webpack config for 'process' usage

* Update playwright-chromium

* Emit esnext modules for tree shaking

* Enable source maps in development

* Dedupe

* Bump babel and things

* npm audit fix

* Add .editorconfig file to match prettier settings

* Update licenses (tslib is 0BSD as of 1.11.2)

https://github.com/microsoft/tslib/pull/96

* Require node >= 10

* Use Node 10 to run e2e tests

* Use 'ws' transport mode for dev server (will be default in next version)

* Remove an 'any'

* No sourcemaps in prod

* Exclude .editorconfig from license checks

* Try nvm for setting node version
2021-04-16 20:15:19 -07:00
Sandeep 26d1074ade
[Security] Bump netty4.version from 4.1.48.Final to 4.1.63.Final (#11117) 2021-04-16 10:32:22 +08:00
Jonathan Wei d28d4e8ed3
Upgrade jetty to 9.4.39.v20210325 (#11076) 2021-04-06 20:20:40 -07:00
Himanshu 43638cc6f9
license.yaml fixes for code introduced related to AWS RDS token based password provider in PR #9518 (#10885)
* license.yaml fixes for code introduced related to AWS RDS token based password provider in PR #9518

* add notice for commons-dbcp in license file

* add version and update NOTICE file
2021-03-10 12:59:25 -08:00
Yi Yuan 36e86a2880
Add protobuf schema registry (#10839)
* dd_protobuf_schema_registry

* change licese

* delete some annotation

* nodify tests

* delete extra exception

* add licenses

* add descriptor and protoMessageType in ProtobufInputRowParser for adopt to old version

* seperate kafka-protobuf-provider

* modify protobuf.md

* refine protobuf.md

* add config and header

* bug fixed

Co-authored-by: yuanyi <yuanyi@freewheel.tv>
2021-03-09 15:15:51 -08:00
Atul Mohan 6040c30fcd
Upgrade jetty to latest version (#10937)
* Upgrade jetty

* Fix license
2021-03-04 08:28:50 -06:00
Abhishek Agarwal 7d9a61cf7f
Suppress CVE-2017-15288 and upgrade bcprov-ext-jdk15o (#10933) 2021-03-02 16:18:27 -08:00
spinatelli 99198c02af
Add config and header support for confluent schema registry. (#10314)
* Add config and header support for confluent schema registry. (porting code from https://github.com/apache/druid/pull/9096)

* Add Eclipse Public License 2.0 to license check

* Update licenses.yaml, revert changes to check-licenses.py and dependencies for integration-tests

* Add spelling exception and remove unused dependency

* Use non-deprecated getSchemaById() and remove duplicated license entry

* Update docs/ingestion/data-formats.md

Co-authored-by: Clint Wylie <cjwylie@gmail.com>

* Added check for schema being null, as per Confluent code

* Missing imports and whitespace

* Updated unit tests with AvroSchema

Co-authored-by: Sergio Spinatelli <sergio.spinatelli.extern@7-tv.de>
Co-authored-by: Sergio Spinatelli <sergio.spinatelli.extern@joyn.de>
Co-authored-by: Clint Wylie <cjwylie@gmail.com>
2021-02-27 14:25:35 -08:00
Alexander Saydakov f930cf14d6
Use the latest Apache DataSketches release 2.0.0 (#10917)
* use the latest Apache DataSketches release 2.0.0

* updated datasketches version

Co-authored-by: AlexanderSaydakov <AlexanderSaydakov@users.noreply.github.com>
2021-02-26 07:52:00 -06:00
Benedict Jin 32e801ceab
Bump Apache Parquet from 1.11.0 to 1.11.1 (#10889) 2021-02-17 12:18:17 +08:00
Jihoon Son a2b5e01142
Bump DataSketches memory to 1.3.0 (#10789) 2021-02-04 18:39:52 -08:00
Jonathan Wei a1a49811d9
Address CVE-2020-8570, suppress CVE-2020-8554 (#10826)
* Address CVE-2020-8570, suppress CVE-2020-8554

* Update licenses.yaml
2021-02-03 15:17:06 -08:00
Himadri Singh 1c1b396eaa
AWS Web Identity / IRSA Support (#10541)
* AWS Web Identity Support

required for AWS IRSA

* Update kinesis-ingestion.md

* disabling coverage tests

https://github.com/apache/druid/pull/10541#issuecomment-737558213

* exclude coverage

* Update licenses.yaml
2021-01-25 18:44:02 +05:30
Vadim Ogievetsky 2fc2938b01
Web console: fix bad results if there is not native bigint (#10741)
* fix bigint when it does not exist

* add test
2021-01-12 16:32:23 -08:00
Jonathan Wei c7f2d3fbb5
Update deps for CVE-2020-28168 and CVE-2020-28052 (#10733)
* Update deps for CVE-2020-28168 and CVE-2020-28052

* Make BC runtime scope
2021-01-07 20:31:44 -08:00
Himanshu c7b1212a43
AWS RDS token based password provider (#9518)
* refresh db pwd

* aws iam token password provider

* fix analyze-dependencies build

* fix doc build

* add  ut for BasicDataSourceExt

* more doc updates

* more  doc update

* moving aws  token password  provider to new extension

* remove duplicate changes

* make  all config inline

* extension docs

* refresh db  password  in SQL Firehose code path as well

* add ut

* fix build

* add new extension to distribution

* rds lib is not provided

* fix license build

* add version to license

* change parent version to 0.19.0-snapshot

* address review comments

* fix core/ code coverage

* Update server/src/main/java/org/apache/druid/metadata/BasicDataSourceExt.java

Co-authored-by: Clint Wylie <cjwylie@gmail.com>

* address review comments

* fix spellchecker

* remove inadvertant website file change

Co-authored-by: Clint Wylie <cjwylie@gmail.com>
2021-01-06 21:15:29 -08:00
Xavier Léauté b7a16d08a6
Update Apache Kafka to 2.7.0 (#10701)
- align scala versions to match Kafka
2020-12-22 13:56:00 -08:00
Himanshu ac1882bf74
kubernetes based discovery druid extension to run Druid on K8S without Zookeeper (#10544)
* honor zk enablement config in more places in druid code

* kubernetes based discovery module

* fix spotbugs check

* fix intellij checks error

* fix doc link to kubernetes.md from extension

* make spellchecker happy

* update license.yaml

* fix dependency check errors

* update extension coverage

* UTs for BaseNodeRoleWatcher

* fix forbidden-api check

* update k8s module coverage ignores

* add Bouncy Castle License being same as MIT License for license checking purposes

* further update licenses.yaml

* label/annotation pre-existence assumption

* address review comment
2020-12-14 21:10:31 -08:00
Jihoon Son abcf624a2e
Bump up jackson-databind to 2.10.5.1 (#10655)
* Bump up jackson version to 2.10.5.1

* only jackson-databind

* license
2020-12-09 13:54:47 -08:00
frank chen c410648630
fix injection failure of StorageLocationSelectorStrategy objects (#10363)
* fix to allow customer storage location selector strategy

* add test cases to check instance of selector strategy

* update doc

* code format

* resolve code review comments

* inject StorageLocation

* fix CI

* fix mismatched license item reported by CI

* change property path from druid.segmentCache.locationSelectorStrategy.type to druid.segmentCache.locationSelector.strategy

* using a helper method to bind to correct property path
2020-12-08 09:48:31 -08:00
Vadim Ogievetsky e3f7217546
Web console: Improve the handling of extreme data (funky datasources, longs) (#10641)
* better API escape

* fix escaping issue, bigints

* update licenses

* fix align

* do not show Query with SQL if no SQL

* add prettify script

* update dev readme

* add ordering to the datasource list

* add ordering to supervisor table
2020-12-08 09:25:14 -08:00
Suneet Saldanha c94be8a945
Revert "Update google client libraries (#10536)" (#10599)
This reverts commit 4537016cad.
2020-12-03 20:14:52 +05:30
Vadim Ogievetsky 5b06c7a3a9
Web console: improve how code is imported, use API instance (#10597)
* fix imports

* clean up imports

* update DQT to fix escaping
2020-12-01 13:16:14 -08:00
Ayush Kulshrestha d0c2ede50c
Added CronScheduler support as a proof to clock drift while emitting metrics (#10448)
Co-authored-by: Ayush Kulshrestha <ayush.kulshrestha@miqdigital.com>
2020-11-25 12:31:38 +01:00
Nishant Bangarwa 4537016cad
Update google client libraries (#10536)
modify license.yaml

Update google oauth client version
2020-11-20 15:23:30 -08:00