Commit Graph

1687 Commits

Author SHA1 Message Date
Clint Wylie e833a4700d
suppress hadoop3 cve that seem not applicable to us (#14252) 2023-05-10 23:08:05 -07:00
Tejaswini Bandlamudi 774073b2e7
Update Hadoop3 as default build version (#14005)
Hadoop 2 often causes red security scans on Druid distribution because of the dependencies it brings. We want to move away from Hadoop 2 and provide Hadoop 3 distribution available. Switch druid to building with Hadoop 3 by default. Druid will still be compatible with Hadoop 2 and users can build hadoop-2 compatible distribution using hadoop2 profile.
2023-04-26 12:52:51 +05:30
Tejaswini Bandlamudi cb302e1bd1
Use apache-jar-resource-bundle:1.5 instead of 1.5-SNAPSHOT (#14054) 2023-04-10 18:55:39 +05:30
Clint Wylie 1aef72aa7e
Bump up the version in pom to 27.0.0 in preparation of release (#14051) 2023-04-10 14:56:59 +05:30
Sandeep ccdf30e399
Bump Joda-Time version for current DateTimeZone data (#13999) 2023-03-29 20:15:49 +05:30
frankgrimes97 2f98675285
Tuple sketch SQL support (#13887)
This PR is a follow-up to #13819 so that the Tuple sketch functionality can be used in SQL for both ingestion using Multi-Stage Queries (MSQ) and also for analytic queries against Tuple sketch columns.
2023-03-28 18:47:12 +05:30
Abhishek Agarwal 139a058ba7
Use sonatype maven central for plugin repositories (#13961)
* Change search order of maven repositories

* Update pom.xml
2023-03-23 15:35:47 +05:30
abhagraw c52d15d65d
Fixing security vulnerability check errors (#13956)
* Fixing security vulnerability check errors

* Updating javax.el to jakarta.el

* Adding cron job trigger on changes to suppressions file
2023-03-23 11:10:06 +05:30
Benedict Jin cee2dfd768
Upgrade ZK from 3.5.9 to 3.5.10 to avoid data inconsistency risk (#13715) 2023-03-15 19:21:09 +05:30
Paul Rogers 4493275d88
Use Maven central repo rather than Apache (#13921)
* Use Maven central repo rather than Apache

* Disable snapshots
2023-03-13 10:49:32 -07:00
Elliott Freis d93fdb2632
Bump CycloneDX module to address POM errors (#13878)
* Bump CycloneDX module to address POM errors

* Including web-console in the PR

---------

Co-authored-by: Elliott Freis <elliottfreis@Elliott-Freis.earth.dynamic.blacklight.net>
2023-03-03 15:39:15 +05:30
Clint Wylie 08b5951cc5
merge druid-core, extendedset, and druid-hll into druid-processing to simplify everything (#13698)
* merge druid-core, extendedset, and druid-hll into druid-processing to simplify everything
* fix poms and license stuff
* mockito is evil
* allow reset of JvmUtils RuntimeInfo if tests used static injection to override
2023-02-17 14:27:41 -08:00
Adarsh Sanjeev e8330e95f5
Update Apache Kafka dependencies to 3.4.0 (#13802)
Release notes:
- https://downloads.apache.org/kafka/3.4.0/RELEASE_NOTES.html
2023-02-15 15:15:13 +05:30
Xavier Léauté 698670c88e
update core Apache Kafka dependencies to 3.3.2 (#13717)
Release notes:
- https://downloads.apache.org/kafka/3.3.2/RELEASE_NOTES.html
2023-01-27 21:00:01 -08:00
Dongjoon Hyun 2503095296
Publish SBOM artifacts (#13648) 2023-01-10 16:08:10 +05:30
abhagraw 74a76c74b1
Updating dependency check version (#13649) 2023-01-10 14:43:19 +05:30
Kashif Faraz 78ae0b7533
Upgrade to netty 4.1.86.Final to address CVEs (#13604)
This commit addresses the following CVEs:
- CVE-2021-43797
- CVE-2022-41881
2022-12-23 01:44:01 +05:30
Peter Stöckli df55768535
Add CodeQL workflow (#13477)
* workflower: Add CodeQL workflow

* add modified CodeQL build config
2022-12-21 09:24:39 +05:30
Jason Koch 6c44dd8175
perf: core/TextReader for faster json ingestion (#13545)
* perf: provide a custom utf8 specific buffered line iterator (benchmark)

Benchmark                         Mode  Cnt     Score     Error  Units
JsonLineReaderBenchmark.baseline  avgt   15  3459.871 ± 106.175  us/op

* perf: provide a custom utf8 specific buffered line iterator

Benchmark                         Mode  Cnt     Score    Error  Units
JsonLineReaderBenchmark.baseline  avgt   15  3022.053 ± 51.286  us/op

* perf: provide a custom utf8 specific buffered line iterator (more tests)

* perf: provide a custom utf8 specific buffered line iterator (pr feedback)

Ensure field visibility is as limited as possible

Null check for buffer in constructor

* perf: provide a custom utf8 specific buffered line iterator (pr feedback)

Remove additional 'finished' variable.

* perf: provide a custom utf8 specific buffered line iterator (more tests and bugfix)
2022-12-19 23:12:37 -08:00
Kashif Faraz 7cf761cee4
Prepare master branch for next release, 26.0.0 (#13401)
* Prepare master branch for next release, 26.0.0

* Use docker image for druid 24.0.1

* Fix version in druid-it-cases pom.xml
2022-11-22 15:31:01 +05:30
Paul Rogers 81d005f267
Druid Catalog basics (#13165)
Druid catalog basics

Catalog object model for tables, columns
Druid metadata DB storage (as an extension)
REST API to update the catalog (as an extension)
Integration tests
Model only: no planner integration yet
2022-11-12 15:30:22 -08:00
Didip Kerabat c875f4bd04
Upgrade curator to 5.4.0 (#13302) 2022-11-03 11:26:19 -07:00
Dr. Sizzles e5ad24ff9f
Support for middle manager less druid, tasks launch as k8s jobs (#13156)
* Support for middle manager less druid, tasks launch as k8s jobs

* Fixing forking task runner test

* Test cleanup, dependency cleanup, intellij inspections cleanup

* Changes per PR review

Add configuration option to disable http/https proxy for the k8s client
Update the docs to provide more detail about sidecar support

* Removing un-needed log lines

* Small changes per PR review

* Upon task completion we callback to the overlord to update the status / locaiton, for slower k8s clusters, this reduces locking time significantly

* Merge conflict fix

* Fixing tests and docs

* update tiny-cluster.yaml 

changed `enableTaskLevelLogPush` to `encapsulatedTask`

* Apply suggestions from code review

Co-authored-by: Abhishek Agarwal <1477457+abhishekagarwal87@users.noreply.github.com>

* Minor changes per PR request

* Cleanup, adding test to AbstractTask

* Add comment in peon.sh

* Bumping code coverage

* More tests to make code coverage happy

* Doh a duplicate dependnecy

* Integration test setup is weird for k8s, will do this in a different PR

* Reverting back all integration test changes, will do in anotbher PR

* use StringUtils.base64 instead of Base64

* Jdk is nasty, if i compress in jdk 11 in jdk 17 the decompressed result is different

Co-authored-by: Rahul Gidwani <r_gidwani@apple.com>
Co-authored-by: Abhishek Agarwal <1477457+abhishekagarwal87@users.noreply.github.com>
2022-11-02 19:44:47 -07:00
chi-chi weng 72c16097ac
Fix Apache Commons Text CVE-2022-42889 (#13226)
* Fix Apache Commons Text  CVE-2022-42889

Fix Apache Commons Text  CVE-2022-42889

https://nvd.nist.gov/vuln/detail/CVE-2022-42889

* Update license

Co-authored-by: Frank Chen <frank.chen021@outlook.com>
2022-10-26 10:04:32 +08:00
Frank Chen d30cf8c308
Dependency cleanup (#13194)
* Clean up dependency in extensions

* Bump protobuf/aws.sdk

* Bump aws-sdk to 1.12.317

* Fix CI

* Fix CI

* Update license

* Update license
2022-10-10 20:34:38 +08:00
Xavier Léauté eff7edb603
update core Apache Kafka dependencies to 3.3.1 (#13176)
Announcement:
- https://blogs.apache.org/kafka/entry/what-rsquo-s-new-in

Release notes:
- https://archive.apache.org/dist/kafka/3.3.0/RELEASE_NOTES.html
- https://downloads.apache.org/kafka/3.3.1/RELEASE_NOTES.html
2022-10-04 12:52:16 -07:00
AmatyaAvadhanula acafd0d1e0
Upgrade kafka version to 3.2.3 to fix CVE (#13142)
Upgrade to 3.2.3 to fix CVE: https://nvd.nist.gov/vuln/detail/CVE-2022-34917
2022-09-28 10:47:09 +05:30
Gian Merlino 5733360dfd
Update Snappy to 1.1.8.4. (#13081)
* Update Snappy to 1.1.8.4.

Prior to this, because snappy-java wasn't included in dependencyManagement,
we actually shipped multiple different versions for different extensions,
ranging from 1.1.7.1 to 1.1.8.4. Now, we standardize on 1.1.8.4.

Among other things, this enables the tests to pass on M1 Macs.

* Update snappy-java versions in licenses.yaml.
2022-09-14 15:13:47 -07:00
Adam Peck ee22663dd3
Add interpolation to JsonConfigurator (#13023)
* Add interpolation to JsonConfigurator

* Fix checkstyle

* Fix tests by removing common-text override

* Add back commons-text without version

* Remove unused hadoopDir configs

* Move some stuff to hopefully pass coverage
2022-09-07 12:48:01 +05:30
senthilkv 3d9aef225d
compressed big decimal - module (#10705)
Compressed Big Decimal is an extension which provides support for 
Mutable big decimal value that can be used to accumulate values 
without losing precision or reallocating memory. This type helps in 
absolute precision arithmetic on large numbers in applications, 
where greater level of accuracy is required, such as financial 
applications, currency based transactions. This helps avoid rounding 
issues where in potentially large amount of money can be lost.

Accumulation requires that the two numbers have the same scale, 
but does not require that they are of the same size. If the value 
being accumulated has a larger underlying array than this value 
(the result), then the higher order bits are dropped, similar to what 
happens when adding a long to an int and storing the result in an 
int. A compressed big decimal that holds its data with an embedded 
array.

Compressed big decimal is an absolute number based complex type 
based on big decimal in Java. This supports all the functionalities 
supported by Java Big Decimal. Java Big Decimal is not mutable in 
order to avoid big garbage collection issues. Compressed big decimal 
is needed to mutate the value in the accumulator.
2022-09-06 00:06:57 -07:00
Gian Merlino 48ceab2153
Add Java 17 information to documentation. (#12990)
The docs say Java 17 support is experimental, and give tips on running
successfully with Java 17.

This patch also removes java.base/jdk.internal.perf and
jdk.management/com.sun.management.internal from the list of required
exports and opens, because they were formerly needed for JvmMonitor,
which was rewritten in #12481 to use MXBeans instead.
2022-08-30 12:32:49 -07:00
Gian Merlino 9eb20e5e7c
Remove dependency on jvm-attach. (#12989)
This dependency was no longer needed after #12481, but remained because
it was used for a (now useless) test. This patch removes the test and
the dependency.
2022-08-29 14:18:33 -07:00
Abhishek Agarwal 618757352b
Bump up the version to 25.0.0 (#12975)
* Bump up the version to 25.0.0

* Fix the version in console
2022-08-29 11:27:38 +05:30
Adam Peck 21b73bde20
Update Curator to 5.3.0 (#12939)
* Update Curator to 5.3.0

* Update licenses.yaml

* Fix inspections + add tests.

* Fix checkstyle

* Another intellij inspection fix

* Update curator exclusions

* Cleanup new exhibitor references

* Remove unused dep and checkstyle fix
2022-08-26 18:23:40 -07:00
Paul Rogers cfed036091
Add the new integration test framework (#12368)
This commit is a first draft of the revised integration test framework which provides:
- A new directory, integration-tests-ex that holds the new integration test structure. (For now, the existing integration-tests is left unchanged.)
- Maven module druid-it-tools to hold code placed into the Docker image.
- Maven module druid-it-image to build the Druid-only test image from the tarball produced in distribution. (Dependencies live in their "official" image.)
- Maven module druid-it-cases that holds the revised tests and the framework itself. The framework includes file-based test configuration, test-specific clients, test initialization and updated versions of some of the common test support classes.

The integration test setup is primarily a huge mass of details. This approach refactors many of those details: from how the image is built and configured to how the Docker Compose scripts are structured to test configuration. An extensive set of "readme" files explains those details. Rather than repeat that material here, please consult those files for explanations.
2022-08-24 17:03:23 +05:30
Gian Merlino d7d15ba51f
Add druid-multi-stage-query extension. (#12918)
* Add druid-multi-stage-query extension.

* Adjustments from CI.

* Task ID validation.

* Various changes from code review.

* Remove unnecessary code.

* LGTM-related.
2022-08-23 18:44:01 -07:00
Xavier Léauté 752e42a312
fix running integration tests on macos aarch64 (#12913)
* add osx-aarch_64 netty-transport-native-kqueue native dependency
* align docker-java dependency versions using bom and update to 3.2.13
2022-08-17 18:03:24 +02:00
dependabot[bot] f70f7b4b89
Bump postgresql from 42.3.3 to 42.4.1 (#12871)
* Bump postgresql from 42.3.3 to 42.4.1

Bumps [postgresql](https://github.com/pgjdbc/pgjdbc) from 42.3.3 to 42.4.1.
- [Release notes](https://github.com/pgjdbc/pgjdbc/releases)
- [Changelog](https://github.com/pgjdbc/pgjdbc/blob/master/CHANGELOG.md)
- [Commits](https://github.com/pgjdbc/pgjdbc/compare/REL42.3.3...REL42.4.1)

---
updated-dependencies:
- dependency-name: org.postgresql:postgresql
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* update licenses.yaml

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2022-08-16 23:25:39 +02:00
Paul Rogers 4706a4c572
Docker build for the revised ITs (#12707)
* Docker build for the revised ITs

* Fix POM versions

* Update comments from review suggestions
2022-08-10 14:17:33 +05:30
Gian Merlino ef6811ef88
Improved Java 17 support and Java runtime docs. (#12839)
* Improved Java 17 support and Java runtime docs.

1) Add a "Java runtime" doc page with information about supported
   Java versions, garbage collection, and strong encapsulation..

2) Update asm and equalsverifier to versions that support Java 17.

3) Add additional "--add-opens" lines to surefire configuration, so
   tests can pass successfully under Java 17.

4) Switch openjdk15 tests to openjdk17.

5) Update FrameFile to specifically mention Java runtime incompatibility
   as the cause of not being able to use Memory.map.

6) Update SegmentLoadDropHandler to log an error for Errors too, not
   just Exceptions. This is important because an IllegalAccessError is
   encountered when the correct "--add-opens" line is not provided,
   which would otherwise be silently ignored.

7) Update example configs to use druid.indexer.runner.javaOptsArray
   instead of druid.indexer.runner.javaOpts. (The latter is deprecated.)

* Adjustments.

* Use run-java in more places.

* Add run-java.

* Update .gitignore.

* Exclude hadoop-client-api.

Brought in when building on Java 17.

* Swap one more usage of java.

* Fix the run-java script.

* Fix flag.

* Include link to Temurin.

* Spelling.

* Update examples/bin/run-java

Co-authored-by: Xavier Léauté <xl+github@xvrl.net>

Co-authored-by: Xavier Léauté <xl+github@xvrl.net>
2022-08-03 23:16:05 -07:00
Karan Kumar 3290b49754
Log4j bump to 2.18 due to [LOG4J2-3419] (#12847)
* Log4j bump to 2.18 due to [LOG4J2-3419]

* Fixing license issues
2022-08-02 23:25:40 -07:00
PJ Fanning 188b5b0027
Upgrade to jetty 9.4.48.v20220622 due to CVEs (#12801)
* Upgrade to jetty 9.4.48.v20220622 due to CVEs

* Update licenses.yaml
2022-07-26 10:11:48 +08:00
Kashif Faraz 9e5f0109fd
Fix CVE-2022-2048 (jetty) and CVE-2022-31159 (aws-java-sdk-s3) (#12807)
Changes:
- Upgrade aws sdk version from `1.12.37` to `1.12.264`
- Upgrade jetty version from `9.4.41.v20210516` to `9.4.47.v20220610`
2022-07-21 13:08:18 +05:30
Paul Rogers ee15c238cc
Clone Calcite planner to access validator (#12708)
Done in preparation for the "single-pass" planner.
2022-07-14 18:10:33 -07:00
Gian Merlino 9c925b4f09
Frame format for data transfer and short-term storage. (#12745)
* Frame format for data transfer and short-term storage.

As we move towards query execution plans that involve more transfer
of data between servers, it's important to have a data format that
provides for doing this more efficiently than the options available to
us today.

This patch adds:

- Columnar frames, which support fast querying.
- Row-based frames, which support fast sorting via memory comparison
  and fast whole-row copies via memory copying.
- Frame files, a container format that can be stored on disk or
  transferred between servers.

The idea is we should use row-based frames when data is expected to
be sorted, and columnar frames when data is expected to be queried.

The code in this patch is not used in production yet. Therefore, the
patch involves minimal changes outside of the org.apache.druid.frame
package.  The main ones are adjustments to SqlBenchmark to add benchmarks
for queries on frames, and the addition of a "forEach" method to Sequence.

* Fixes based on tests, static analysis.

* Additional fixes.

* Skip DS mapping tests on JDK 14+

* Better JDK checking in tests.

* Fix imports.

* Additional comment.

* Adjustments from code review.

* Update test case.
2022-07-08 20:42:06 -07:00
Jianhuan Liu 4574dea5e9
Use MXBeans to get GC metrics #12476 (#12481)
* jvm gc to mxbeans

* add zgc and shenandoah #12476

* remove tryCreateGcCounter

* separate the space collector

* blend GcGenerationCollector into GcCollector

* add jdk surefire argLine
2022-07-08 14:32:06 +08:00
PJ Fanning 059aba781a
issue-12628: upgrade jetty to 9.4.41.v20210516 due to CVE (#12629)
* upgrade jetty to 9.4.41.v20210516 due to cve

* Update licenses.yaml
2022-07-07 00:20:01 +08:00
imply-cheddar e3128e3fa3
Poison stupid pool (#12646)
* Poison StupidPool and fix resource leaks

There are various resource leaks from test setup as well as some
corners in query processing.  We poison the StupidPool to start failing
tests when the leaks come and fix any issues uncovered from that so
that we can start from a clean baseline.

Unfortunately, because of how poisoning works,
we can only fail future checkouts from the same pool,
which means that there is a natural race between a
leak happening -> GC occurs -> leak detected -> pool poisoned.

This race means that, depending on interleaving of tests,
if the very last time that an object is checked out
from the pool leaks, then it won't get caught.
At some point in the future, something will catch it,
 however and from that point on it will be deterministic.

* Remove various things left over from iterations

* Clean up FilterAnalysis and add javadoc on StupidPool

* Revert changes to .idea/misc.xml that accidentally got pushed

* Style and test branches

* Stylistic woes
2022-07-03 14:36:22 -07:00
Rohan Garg c09b5a2294
Fix skipTests build flag (#12716)
* fix skipTests

* Skip console UTs with skipTests

* Use skipTests in skip-tests profile
2022-06-29 21:59:26 -07:00
Rui Chen 068bea6334
deps: upgrade mysql-connector-java to v5.1.49 (#12704) 2022-06-29 23:15:46 +08:00
Paul Rogers f83fab699e
Add IT-related changes pulled out of PR #12368 (#12673)
This commit contains changes made to the existing ITs to support the new ITs.

Changes:
- Make the "custom node role" code usable by the new ITs. 
- Use flag `-DskipITs` to skips the integration tests but runs unit tests.
- Use flag `-DskipUTs` skips unit tests but runs the "new" integration tests.
- Expand the existing Druid profile, `-P skip-tests` to skip both ITs and UTs.
2022-06-26 02:13:59 +05:30
Dr. Sizzles 7291c92f4f
Adding zstandard compression library (#12408)
* Adding zstandard compression library

* 1. Took @clintropolis's advice to have ZStandard decompressor use the byte array when the buffers are not direct.
2. Cleaned up checkstyle issues.

* Fixing zstandard version to latest stable version in pom's and updating license files

* Removing zstd from benchmarks and adding to processing (poms)

* fix the intellij inspection issue

* Removing the prefix v for the version in the license check for ztsd

* Fixing license checks

Co-authored-by: Rahul Gidwani <r_gidwani@apple.com>
2022-05-28 17:01:44 -07:00
Abhishek Agarwal 32fe4d1324
Use a different repository to download sigar artifacts. (#12561) 2022-05-24 14:42:51 +05:30
Clint Wylie 2d8dbb53e0
update to latest lz4 1.8.0 (#12557) 2022-05-21 16:02:20 +08:00
Xavier Léauté ec41dfb535
upgrade core Apache Kafka dependencies to 3.2.0 (#12538)
Announcement: https://blogs.apache.org/kafka/entry/what-s-new-in-apache8
Release notes: https://downloads.apache.org/kafka/3.2.0/RELEASE_NOTES.html
2022-05-19 09:04:52 -07:00
Gian Merlino 4631cff2a9
Free ByteBuffers in tests and fix some bugs. (#12521)
* Ensure ByteBuffers allocated in tests get freed.

Many tests had problems where a direct ByteBuffer would be allocated
and then not freed. This is bad because it causes flaky tests.

To fix this:

1) Add ByteBufferUtils.allocateDirect(size), which returns a ResourceHolder.
   This makes it easy to free the direct buffer. Currently, it's only used
   in tests, because production code seems OK.

2) Update all usages of ByteBuffer.allocateDirect (off-heap) in tests either
   to ByteBuffer.allocate (on-heap, which are garbaged collected), or to
   ByteBufferUtils.allocateDirect (wherever it seemed like there was a good
   reason for the buffer to be off-heap). Make sure to close all direct
   holders when done.

* Changes based on CI results.

* A different approach.

* Roll back BitmapOperationTest stuff.

* Try additional surefire memory.

* Revert "Roll back BitmapOperationTest stuff."

This reverts commit 49f846d9e3.

* Add TestBufferPool.

* Revert Xmx change in tests.

* Better behaved NestedQueryPushDownTest. Exit tests on OOME.

* Fix TestBufferPool.

* Remove T1C from ARM tests.

* Somewhat safer.

* Fix tests.

* Fix style stuff.

* Additional debugging.

* Reset null / expr configs better.

* ExpressionLambdaAggregatorFactory thread-safety.

* Alter forkNode to try to get better info when a JVM crashes.

* Fix buffer retention in ExpressionLambdaAggregatorFactory.

* Remove unused import.
2022-05-19 07:42:29 -07:00
Kashif Faraz 7ab2170802
Use datasketches version 3.2.0 (#12509)
Changes:
- Use apache datasketches version 3.2.0.
- Remove unsafe reflection-based usage of datasketch internals added in #12022
2022-05-13 11:28:15 +05:30
Abhishek Radhakrishnan 9177515be2
Add IPAddress java library as dependency and migrate IPv4 functions to use the new library. (#11634)
* Add ipaddress library as dependency.

* IPv4 functions to use the inet.ipaddr package.

* Remove unused imports.

* Add new function.

* Minor rename.

* Add more unit tests.

* IPv4 address expr utils unit tests and address options.

* Adjust the IPv4Util functions.

* Move the UTs a bit around.

* Javadoc comments.

* Add license info for IPAddress.

* Fix groupId, artifact and version in license.yaml.

* Remove redundant subnet in messages - fixes UT.

* Remove unused commons-net dependency for /processing project.

* Make class and methods public so it can be accessed.

* Add initial version of benchmark

* Add subnetutils package for benchmarks.

* Auto generate ip addresses.

* Add more v4 address representations in setup to avoid bias.

* Use ThreadLocalRandom to avoid forbidden API usage.

* Adjust IPv4AddressBenchmark to adhere to codestyle rules.

* Update ipaddress library to latest 5.3.4

* Add ipaddress package dependency to benchmarks project.
2022-05-11 22:06:20 -07:00
aggarwalakshay dd8781f5b0
Upgrade dependency-check-maven to 7.0.4 (#12441) 2022-05-01 22:45:58 +08:00
Gian Merlino 72d15ab321
JvmMonitor: Handle more generation and collector scenarios. (#12469)
* JvmMonitor: Handle more generation and collector scenarios.

ZGC on Java 11 only has a generation 1 (there is no 0). This causes
a NullPointerException when trying to extract the spacesCount for
generation 0. In addition, ZGC on Java 15 has a collector number 2
but no spaces in generation 2, which breaks the assumption that
collectors always have same-numbered spaces.

This patch adjusts things to be more robust, enabling the JvmMonitor
to work properly for ZGC on both Java 11 and 15.

* Test adjustments.

* Improve surefire arglines.

* Need a placeholder
2022-04-27 11:18:40 -07:00
Abhishek Agarwal 2fe053c5cb
Bump up the versions (#12480) 2022-04-27 14:28:20 +05:30
PJ Fanning 5099f5aa70
update httpclient due to cve (#12422)
https://github.com/apache/druid/issues/12421
2022-04-21 10:12:19 +08:00
Jihoon Son d2a6754692
Bump PostgreSQL JDBC driver to 42.3.3 (CVE-2022-21724) (#12410)
* Bump PostgreSQL JDBC driver to 42.3.3 (CVE-2022-21724)

* update license file
2022-04-09 15:38:26 +05:30
aggarwalakshay 7d5666109c
upgrade surefire 3.0.0-M6 (#12395)
* upgrade surefire 3.0.0-M6

* increasing memory
2022-04-04 23:56:15 -07:00
Xavier Léauté c3377bf744
upgrade maven-pmd-plugin to fix warning (#12349)
we sometimes see warnings similar to the one mentioned
https://issues.apache.org/jira/browse/MPMD-325

Upgrading the plugin should hopefully reduce occurrence of those.
2022-03-19 10:18:26 -07:00
dependabot[bot] 4ed1abca94
Bump slf4j.version from 1.7.12 to 1.7.36 (#11594)
Bump slf4j.version from 1.7.12 to 1.7.36

- [Release notes](Release notes: https://www.slf4j.org/news.html)

Updates `jcl-over-slf4j` from 1.7.12 to 1.7.36
- [Commits](https://github.com/qos-ch/slf4j/compare/v_1.7.12...v_1.7.36)

Updates `slf4j-simple` from 1.7.12 to 1.7.36
- [Commits](https://github.com/qos-ch/slf4j/compare/v_1.7.12...v_1.7.36)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Suneet Saldanha <suneet@apache.org>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2022-03-18 13:45:44 -07:00
Xavier Léauté 6f0e5f25fa
update surefire plugin to 3.0.0-M4 (#12342)
stay on surefire 3.0.0-M4 until we can upgrade to 3.0.0-M6
with a fix for https://issues.apache.org/jira/browse/SUREFIRE-1815
causing issues in RetryUtilsTest.
2022-03-18 08:20:28 -07:00
dependabot[bot] a5dfb911de
Bump maven-site-plugin from 3.1 to 3.11.0 (#12310)
Bumps [maven-site-plugin](https://github.com/apache/maven-site-plugin) from 3.1 to 3.11.0.
- [Release notes](https://github.com/apache/maven-site-plugin/releases)
- [Commits](https://github.com/apache/maven-site-plugin/compare/maven-site-plugin-3.1...maven-site-plugin-3.11.0)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-site-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-17 15:17:29 +08:00
Xavier Léauté 5d02a91faa
upgrade Error Prone to 2.11 (requires Java 11) (#12306)
The latest version of Error Prone now requires Java 11. Upgrading means we can
remove a lot of the maven profile complexity required to run checks with Java 8.
This also requires switching our strict build to use Java 11.

* update error-prone to 2.11
* remove need for specific maven profiles for Java 8 and Java 15
* fix additional Error Prone warnings with Java 11
* update strict build to use Java 11
2022-03-14 19:40:48 -07:00
dependabot[bot] a1cdee2a3a
Bump jersey.version from 1.19.3 to 1.19.4 (#12290)
* Bump jersey.version from 1.19.3 to 1.19.4

Bumps `jersey.version` from 1.19.3 to 1.19.4.

Updates `jersey-client` from 1.19.3 to 1.19.4

Updates `jersey-core` from 1.19.3 to 1.19.4

Updates `jersey-grizzly2` from 1.19.3 to 1.19.4

Updates `jersey-guice` from 1.19.3 to 1.19.4

Updates `jersey-server` from 1.19.3 to 1.19.4

Updates `jersey-servlet` from 1.19.3 to 1.19.4

Updates `jersey-json` from 1.19.3 to 1.19.4

Updates `jersey-test-framework-core` from 1.19.3 to 1.19.4

Updates `jersey-test-framework-grizzly2` from 1.19.3 to 1.19.4

---
updated-dependencies:
- dependency-name: com.sun.jersey:jersey-client
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: com.sun.jersey:jersey-core
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: com.sun.jersey:jersey-grizzly2
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: com.sun.jersey.contribs:jersey-guice
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.sun.jersey:jersey-server
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.sun.jersey:jersey-servlet
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.sun.jersey:jersey-json
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.sun.jersey.jersey-test-framework:jersey-test-framework-core
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: com.sun.jersey.jersey-test-framework:jersey-test-framework-grizzly2
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update licenses.yaml

* Update licenses.yaml

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Clint Wylie <cwylie@apache.org>
2022-03-04 09:57:20 +08:00
Alexander Saydakov 50038d9344
latest datasketches-java-3.1.0 (#12224)
These changes are to use the latest datasketches-java-3.1.0 and also to restore support for quantile and HLL4 sketches to be able to grow larger than a given buffer in a buffer aggregator and move to heap in rare cases. This was discussed in #11544.

Co-authored-by: AlexanderSaydakov <AlexanderSaydakov@users.noreply.github.com>
2022-03-01 17:14:42 -08:00
Xavier Léauté d105519558
Replace use of PowerMock with Mockito (#12282)
Mockito now supports all our needs and plays much better with recent Java versions.
Migrating to Mockito also simplifies running the kind of tests that required PowerMock in the past. 

* replace all uses of powermock with mockito-inline
* upgrade mockito to 4.3.1 and fix use of deprecated methods
* import mockito bom to align all our mockito dependencies
* add powermock to forbidden-apis to avoid accidentally reintroducing it in the future
2022-02-27 22:47:09 -08:00
Xavier Léauté 1434197ee1
update airline dependency to 2.x (#12270)
* upgrade Airline to Airline 2
  https://github.com/airlift/airline is no longer maintained, updating to
  https://github.com/rvesse/airline (Airline 2) to use an actively
  maintained version, while minimizing breaking changes.

  Note, this is a backwards incompatible change, and extensions relying on
  the CliCommandCreator extension point will also need to be updated.

* fix dependency checks where jakarta.inject is now resolved first instead
  of javax.inject, due to Airline 2 using jakarta
2022-02-27 15:19:28 -08:00
Xavier Léauté 009dd9e09a
upgrade core Apache Kafka dependencies to 3.1.0 (#12203)
Announcement: https://blogs.apache.org/kafka/entry/what-s-new-in-apache7
Release notes: https://dist.apache.org/repos/dist/release/kafka/3.1.0/RELEASE_NOTES.html

* upgrade core Apache Kafka dependencies to 3.1.0
* fix use of private Kafka APIs
* remove deprecated test rules
* remove mock calls that weren't verified in the first place
* remove the need for powermock in KafkaLookupExtractorFactoryTest
* align curator-test version with curator itself
* update easymock to 4.3.0
2022-02-23 18:42:51 -08:00
Laksh Singla 4add2510ed
Add syntax support for PARTITIONED BY/CLUSTERED BY in INSERT queries (#12163)
This PR aims to add parser changes for supporting PARTITIONED BY and CLUSTERED BY as proposed in the issue #11929.
2022-02-08 16:23:15 +05:30
Uwe Schindler 1f7dd6d86c
Forbiddenapis: Split the guava16-only signatures file from main signatures file (#12170) 2022-01-19 17:50:28 -08:00
Benedict Jin b55f7a25fe
Fix forbiddenapis causing travis failing (#12158)
* Fix forbiddenapis causing travis failing

* Use failOnUnresolvableSignatures instead
2022-01-15 16:13:37 -08:00
Ivan Vankovich 6a93872586
OpenTelemetry emitter extension (#12015)
* Add OpenTelemetry emitter extension

* Fix build

* Fix checkstyle

* Add used undeclared dependencies

* Ignore unused declared dependencies
2022-01-15 12:18:04 +08:00
Frank Chen fe71fc414f
Update log4j2 to 2.17.1 (#12106)
Signed-off-by: frank chen <frank.chen021@outlook.com>
2021-12-30 19:18:16 -06:00
Frank Chen f345759360
Update to 2.17.0 (#12081) 2021-12-19 20:27:08 -08:00
Gian Merlino d917e0433e
Update to log4j 2.16.0. (#12061)
* Update to log4j 2.16.0.

* Update licenses.yaml
2021-12-13 19:06:00 -08:00
Xavier Léauté 19316018b8
update log4j to 2.15.0 to address security vulnerabilities (#12051) 2021-12-09 22:34:54 -08:00
Xavier Léauté ffc5ade506
Remove use of deprecated PMD ruleset (#12044)
* Remove use of deprecated PMD ruleset

This fixes annoying warnings we were getting during build.

- Use a custom PMD ruleset, since the built-in one uses deprecated rules.
- UnnecessaryImport replaces most of the deprecated rules
- Update maven-pmd-plugin to 3.15
- Exclude ancient asm version from caliper, since this was causing
  incompatibility warnings with PMD and could also affect our tests runs
  in unexpected ways
2021-12-09 13:04:27 -08:00
Xavier Léauté 0565f0e6a1
fix build warnings for forbidden-apis (#12034)
* replace deprecated forbidden-apis config failOnUnresolvableSignatures
with ignoreSignaturesOfMissingClasses which avoids warnings for
classes not present in a particular sub-module

* fix incorrect signature for Files.createTempDirectory
2021-12-07 22:21:01 -08:00
Clint Wylie 7237dc837c
complex typed expressions (#11853)
* complex typed expressions

* add built-in hll collector expressions to get coverage on druid-processing, more types, more better

* rampage!!!

* more javadoc

* adjustments

* oops

* lol

* remove unused dependency

* contradiction?

* more test
2021-11-08 00:33:06 -08:00
Karan Kumar 90640bb316
Support for hadoop 3 via maven profiles (#11794)
Add support for hadoop 3 profiles . Most of the details are captured in #11791 .
We use a combination of maven profiles and resource filtering to achieve this. Hadoop2 is supported by default and a new maven profile with the name hadoop3 is created. This will allow the user to choose the profile which is best suited for the use case.
2021-10-30 22:46:24 +05:30
Jihoon Son 07a232d7b4
Bump netty4 to 4.1.68; suppress CVE-2021-37136 and CVE-2021-37137 for netty3 (#11844)
* bump netty4 to 4.1.68

* suppress CVE-2021-37136 and CVE-2021-37137 for netty3

* license
2021-10-25 21:09:15 -07:00
Alexander Saydakov 8cf1cbc4a9
latest datasketches-java and datasketches-memory (#11773)
* latest datasketches-java and datasketches-memory

* updated versions of datasketches-java and datasketches-memory

Co-authored-by: AlexanderSaydakov <AlexanderSaydakov@users.noreply.github.com>
2021-10-19 23:42:30 -07:00
Xavier Léauté bc3b038712
Update Apache Kafka client libraries to 3.0.0 (#11735)
Release notes:
https://downloads.apache.org/kafka/3.0.0/RELEASE_NOTES.html
https://blogs.apache.org/kafka/entry/what-s-new-in-apache6
2021-10-05 10:23:19 -07:00
William Hyun 9bff6bd70e
Upgrade ORC to 1.7.0 (#11726)
* Upgrade ORC to 1.7.0

* address comments

* address comments

* Add import
2021-09-27 13:20:09 -07:00
Clint Wylie 1370fcf9bf
dependency check with inhert instead of aggregate (#11709) 2021-09-15 04:18:59 -07:00
Clint Wylie fe1d8c206a
bump version to 0.23.0-SNAPSHOT (#11670) 2021-09-08 15:56:04 -07:00
Jihoon Son 2a658acad4
Put sleep in an extension (#11632)
* Put sleep in an extension

* dependency
2021-08-25 01:27:45 -07:00
dependabot[bot] 590583699f
Bump maven-enforcer-plugin from 3.0.0-M3 to 3.0.0 (#11543)
Bumps [maven-enforcer-plugin](https://github.com/apache/maven-enforcer) from 3.0.0-M3 to 3.0.0.
- [Release notes](https://github.com/apache/maven-enforcer/releases)
- [Commits](https://github.com/apache/maven-enforcer/compare/enforcer-3.0.0-M3...enforcer-3.0.0)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-enforcer-plugin
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-04 08:43:49 -07:00
dependabot[bot] cf674c833c
Bump maven-resources-plugin from 3.1.0 to 3.2.0 (#11525)
Bumps [maven-resources-plugin](https://github.com/apache/maven-resources-plugin) from 3.1.0 to 3.2.0.
- [Release notes](https://github.com/apache/maven-resources-plugin/releases)
- [Commits](https://github.com/apache/maven-resources-plugin/compare/maven-resources-plugin-3.1.0...maven-resources-plugin-3.2.0)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-resources-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-08-02 09:38:34 -07:00
dependabot[bot] 2e850b5655
Bump aws.sdk.version from 1.11.884 to 1.12.37 (#11367)
* Bump aws.sdk.version from 1.11.884 to 1.12.8

Bumps `aws.sdk.version` from 1.11.884 to 1.12.8.

Updates `aws-java-sdk-core` from 1.11.884 to 1.12.8
- [Release notes](https://github.com/aws/aws-sdk-java/releases)
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.11.884...1.12.8)

Updates `aws-java-sdk-ec2` from 1.11.884 to 1.12.8
- [Release notes](https://github.com/aws/aws-sdk-java/releases)
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.11.884...1.12.8)

Updates `aws-java-sdk-s3` from 1.11.884 to 1.12.8
- [Release notes](https://github.com/aws/aws-sdk-java/releases)
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.11.884...1.12.8)

Updates `aws-java-sdk-sts` from 1.11.884 to 1.12.8
- [Release notes](https://github.com/aws/aws-sdk-java/releases)
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.11.884...1.12.8)

Updates `aws-java-sdk-kinesis` from 1.11.884 to 1.12.8
- [Release notes](https://github.com/aws/aws-sdk-java/releases)
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.11.884...1.12.8)

Updates `aws-java-sdk-rds` from 1.11.884 to 1.12.8
- [Release notes](https://github.com/aws/aws-sdk-java/releases)
- [Changelog](https://github.com/aws/aws-sdk-java/blob/master/CHANGELOG.md)
- [Commits](https://github.com/aws/aws-sdk-java/compare/1.11.884...1.12.8)

---
updated-dependencies:
- dependency-name: com.amazonaws:aws-java-sdk-core
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: com.amazonaws:aws-java-sdk-ec2
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: com.amazonaws:aws-java-sdk-s3
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: com.amazonaws:aws-java-sdk-sts
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: com.amazonaws:aws-java-sdk-kinesis
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: com.amazonaws:aws-java-sdk-rds
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix license and bump to latest aws

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Suneet Saldanha <suneet@apache.org>
2021-08-01 00:12:57 -07:00
Xavier Léauté 4bca7f014e
update error-prone to 2.8.0 with fix for crashing check (#11494)
* error-prone 2.8.0 fixes https://github.com/google/error-prone/issues/2396
* fix for a few ignored return values
* fix unknown args in sub-modules
2021-07-29 09:13:46 -07:00
Suneet Saldanha 3f456fe305
Address CVE-2021-35515 CVE-2021-36090 (#11496)
* Address CVE-2021-35515 CVE-2021-36090

Bump commons-compress to deal with new CVEs

* fix licenses
2021-07-26 14:54:32 -07:00
Dongjoon Hyun 5037493e45
Bump commons-io to 2.11.0 (#11460)
* Bump commons-io to 2.11.0

* Address comments

* Remove try catch

* Fix checkstyle
2021-07-19 15:47:14 -07:00
Clint Wylie 63fcd77c38
support using mariadb connector with mysql extensions (#11402)
* support using mariadb connector with mysql extensions

* cleanup and more tests

* fix test

* javadocs, more tests, etc

* style and more test

* more test more better

* missing pom

* more pom
2021-07-08 12:25:37 -07:00
zachjsh 8037a54525
revert commons-io to 2.6 (#11392)
* * revert commons-io to 2.6

* * fix failing tests
2021-06-29 23:04:38 -07:00
Xavier Léauté 3ad6a3d74f
switch to netty-bom instead of individual dependencies (#11356) 2021-06-29 12:52:12 -07:00
Xavier Léauté a1c20d7457
update jackson dependencies to use bom (#11353)
Switching to the bom dependency declaration simplifies managing jackson
dependencies. It also removes the need to override individual library
versions for CVE fixes, since the bom takes care of that internally.

This change aligns our jackson dependency versions on 2.10.5(.x):
- updates jackson libraries from 2.10.2 to 2.10.5
- jackson-databind remains at 2.10.5.1 as defined in the bom

Release notes: https://github.com/FasterXML/jackson/wiki/Jackson-Release-2.10
2021-06-16 18:37:30 -07:00
Xavier Léauté 712f2a5d00
upgrade error-prone to 2.7.1 and support checks with Java 11+ (#11363)
* upgrade error-prone to 2.7.1 and support checks with Java 11+

- upgrade error-prone to 2.7.1
- support running error-prone with Java 11 and above using -Xplugin
  instead of custom compiler
- add compiler arguments to ignore warnings/errors in Java 15/16
- introduce strictCompile property to enable strict profiles since we
  now need multiple strict profiles for Java 8
- properly exclude all generated source files from error-prone
- fix druid-processing overriding annotation processors from parent pom
- fix druid-core disabling most non-default checks
- align plugin and annotation errorprone versions
- fix / suppress additional issues found by error-prone:
  * fix bug in SeekableStreamSupervisor initializing ArrayList size with
    the taskGroupdId
  * fix missing @Override annotations
- remove outdated compiler plugin in benchmarks
- remove deleted ParameterPackage error-prone rule
- re-enable checks on benchmark module as well

* fix IntelliJ inspections

* disable LongFloatConversion due to bug in error-prone with JDK 8

* add comment about InsecureCrypto
2021-06-16 12:55:34 -07:00
Xavier Léauté 6edc017fad
update jacoco for Java 15/16 support (#11361) 2021-06-12 13:57:41 +08:00
dependabot[bot] 167044f715
Bump fastutil from 8.2.3 to 8.5.4 (#11347)
* Bump fastutil from 8.2.3 to 8.5.4

Bumps [fastutil](https://github.com/vigna/fastutil) from 8.2.3 to 8.5.4.
- [Release notes](https://github.com/vigna/fastutil/releases)
- [Changelog](https://github.com/vigna/fastutil/blob/master/CHANGES)
- [Commits](https://github.com/vigna/fastutil/compare/8.2.3...8.5.4)

---
updated-dependencies:
- dependency-name: it.unimi.dsi:fastutil
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* update licenses.yaml
* update maven dependency list for -core and -extra libraries to pass maven dependency checks

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2021-06-10 07:43:18 -07:00
dependabot[bot] 1a27ed0b86
Bump junit from 4.12 to 4.13.2 (#11344)
Bumps [junit](https://github.com/junit-team/junit4) from 4.12 to 4.13.2.
- [Release notes](https://github.com/junit-team/junit4/releases)
- [Changelog](https://github.com/junit-team/junit4/blob/main/doc/ReleaseNotes4.12.md)
- [Commits](https://github.com/junit-team/junit4/compare/r4.12...r4.13.2)

---
updated-dependencies:
- dependency-name: junit:junit
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2021-06-08 18:18:39 -07:00
dependabot[bot] be10a236d5
Bump commons-io from 2.6 to 2.9.0 (#11338)
* Bump commons-io from 2.6 to 2.9.0

Bumps commons-io from 2.6 to 2.9.0.

---
updated-dependencies:
- dependency-name: commons-io:commons-io
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* update licenses.yaml to reflect version bumps
* fix tests relying on specific log messages

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2021-06-08 10:02:59 -07:00
Xavier Léauté dbec8da7ea
enable dependabot for dependency updates (#11079)
* enable dependabot for dependency updates
* exclude dependabot config from license checks
2021-06-07 13:56:32 -07:00
Vadim Ogievetsky 0c5d1c9725
Web console: add more query fixing auto suggestions (#11203)
* add more query fixing auto suggestions

* update query gen

* update toolkit

* update licenses

* fix funky quotes

* funky => fancy

* revert engine change

* separate web-console and website npm and node deps
2021-06-04 09:29:00 -07:00
Xavier Léauté b517c3339b
remove ZooKeeper 3.4 support + pass tests with Java 15 (#11073)
With this change, Druid will only support ZooKeeper 3.5.x and later.

In order to support Java 15 we need to switch to ZK 3.5.x client libraries and drop support for ZK 3.4.x
(see #10780 for the detailed reasons) 

* remove ZooKeeper 3.4.x compatibility
* exclude additional ZK 3.5.x netty dependencies to ensure we use our version
* keep ZooKeeper version used for integration tests in sync with client library version
* remove the need to specify ZK version at runtime for docker
* add support to run integration tests with JDK 15
* build and run unit tests with Java 15 in travis
2021-05-25 12:49:49 -07:00
Atul Mohan ac95f99d59
Upgrade jetty version (#11207) 2021-05-06 18:21:36 +08:00
Xavier Léauté 0296f20551
upgrade Apache Kafka to 2.8.0 (#11139)
* upgrade to Apache Kafka 2.8.0 (release notes:
  https://downloads.apache.org/kafka/2.8.0/RELEASE_NOTES.html)
* pass Kafka version as a Docker argument in integration tests
  to keep in sync with maven version
* fix use of internal Kafka APIs in integration tests
2021-04-24 08:27:07 -07:00
Vadim Ogievetsky 4caa221d72
Web console: Better inline docs (#11128)
* better highlight

* better highlighting

* add spec
2021-04-19 14:36:53 -07:00
John Gozde fdc3c2f362
Web console: update dev dependencies (#11119)
* Update some dev dependencies, prettify, tslint-fix

* Sort tsconfig keys for easy comparison

* Set noImplicitThis

* Slightly more accurate types

* Bump Jest and related

* Bump react to latest on v16

* Bump node-sass, sass-loader for node14 support

* Remove node-sass-chokidar (unused)

* More unused dependencies

* Fix blueprint imports

* Webpack 5

* Update webpack config for 'process' usage

* Update playwright-chromium

* Emit esnext modules for tree shaking

* Enable source maps in development

* Dedupe

* Bump babel and things

* npm audit fix

* Add .editorconfig file to match prettier settings

* Update licenses (tslib is 0BSD as of 1.11.2)

https://github.com/microsoft/tslib/pull/96

* Require node >= 10

* Use Node 10 to run e2e tests

* Use 'ws' transport mode for dev server (will be default in next version)

* Remove an 'any'

* No sourcemaps in prod

* Exclude .editorconfig from license checks

* Try nvm for setting node version
2021-04-16 20:15:19 -07:00
Sandeep 26d1074ade
[Security] Bump netty4.version from 4.1.48.Final to 4.1.63.Final (#11117) 2021-04-16 10:32:22 +08:00
Jonathan Wei d28d4e8ed3
Upgrade jetty to 9.4.39.v20210325 (#11076) 2021-04-06 20:20:40 -07:00
Clint Wylie 071b6f5685
bump to latest of same version node and npm versions, bump frontend-maven-plugin (#11057) 2021-03-31 16:42:18 -07:00
Xavier Léauté 68781a0d20
update testing frameworks for Java 15 support (#10984)
* update jacoco to 0.8.6
* update easymock to 4.2
* update equalsverifier to 3.5.5
* update mockito to 3.8.0
* update powermock to 2.0.9
* update assertj-core to 3.19.0
* update testng to 7.3.0
  - fix DTD url security for testng 7.x
  - fix backwards incompatibility in testng 7.x
2021-03-12 20:18:13 -08:00
Xavier Léauté d26e1bc70d
update code check plugins for Java 15 support (#10978)
* update maven-forbidden-api plugin to 3.1
* update maven-pmd-plugin to 3.14
* update spotbugs to 4.2.2
* fixes validation failures newly caught by those updates
  - fix SpotBugs NP_NONNULL_PARAM_VIOLATION
  - fix PMD UnnecessaryFullyQualifiedName
2021-03-11 07:31:41 -08:00
Xavier Léauté 7a68cd8b86
use maven enforcer to check maven version (#10977)
* removes a warning about prerequisites only being allowed for plugins
* update maven enforcer plugin to the latest version (3.0.0-M3)
2021-03-11 07:30:10 -08:00
Tianxin Zhao a57c28e9ce
prometheus metric exporter (#10412)
* prometheus-emitter

* use existing jetty server to expose prometheus collection endpoint

* unused variables

* better variable names

* removed unused dependencies

* more metric definitions

* reorganize

* use prometheus HTTPServer instead of hooking into Jetty server

* temporary empty help string

* temporary non-empty help.  fix incorrect dimension value in JSON (also updated statsd json)

* added full help text.  added metric conversion factor for timers that are not using seconds. Correct metric dimension name in documentation

* added documentation for prometheus emitter

* safety for invalid labelNames

* fix travis checks

* Unit test and better sanitization of metrics names and label values

* add precondition to check namespace against regex

* use precompiled regex

* remove static imports. fix metric types

* better docs. fix possible NPE in PrometheusEmitterConfig. Guard against multiple calls to PrometheusEmitter.start()

* Update regex for label-value replacements to allow internal numeric values.  Additional tests

* Adds missing license header
updates website/.spelling to add words used in prometheus-emitter docs.
updates docs/operations/metrics.md to correct the spelling of
bufferPoolName

* fixes version in extensions-contrib/prometheus-emitter

* fix style guide errors

* update import ordering

* add another word to website/.spelling

* remove unthrown declared exception

* remove unused import

* Pushgateway strategy for metrics

* typo

* Format fix and nullable strategy

* Update pom file for prometheus-emitter

* code review comments. Counter to gauge for cache metrics, periodical task to pushGateway

* Syntax fix

* Dimension label regex include numeric character back, fix previous commit

* bump prometheus-emitter pom dev version

* Remove scheduled task inside poen that push metrics

* Fix checkstyle

* Unit test coverage

* Unit test coverage

* Spelling

* Doc fix

* spelling

Co-authored-by: Michael Schiff <michael.schiff@tubemogul.com>
Co-authored-by: Michael Schiff <schiff.michael@gmail.com>
Co-authored-by: Tianxin Zhao <tianxin.zhao@tubemogul.com>
Co-authored-by: Tianxin Zhao <tizhao@adobe.com>
2021-03-09 14:37:31 -08:00
Clint Wylie 96889cdebc
add avro + kafka + schema registry integration test (#10929)
* add avro + schema registry integration test

* style

* retry init

* maybe this

* oops heh

* this will fix it

* review stuffs

* fix comment
2021-03-08 08:12:12 -08:00
zhangyue19921010 bddacbb1c3
Dynamic auto scale Kafka-Stream ingest tasks (#10524)
* druid task auto scale based on kafka lag

* fix kafkaSupervisorIOConfig and KinesisSupervisorIOConfig

* druid task auto scale based on kafka lag

* fix kafkaSupervisorIOConfig and KinesisSupervisorIOConfig

* test dynamic auto scale done

* auto scale tasks tested on prd cluster

* auto scale tasks tested on prd cluster

* modify code style to solve 29055.10 29055.9 29055.17 29055.18 29055.19 29055.20

* rename test fiel function

* change codes and add docs based on capistrant reviewed

* midify test docs

* modify docs

* modify docs

* modify docs

* merge from master

* Extract the autoScale logic out of SeekableStreamSupervisor to minimize putting more stuff inside there &&  Make autoscaling algorithm configurable and scalable.

* fix ci failed

* revert msic.xml

* add uts to test autoscaler create && scale out/in and kafka ingest with scale enable

* add more uts

* fix inner class check

* add IT for kafka ingestion with autoscaler

* add new IT in groups=kafka-index named testKafkaIndexDataWithWithAutoscaler

* review change

* code review

* remove unused imports

* fix NLP

* fix docs and UTs

* revert misc.xml

* use jackson to build autoScaleConfig with default values

* add uts

* use jackson to init AutoScalerConfig in IOConfig instead of Map<>

* autoscalerConfig interface and provide a defaultAutoScalerConfig

* modify uts

* modify docs

* fix checkstyle

* revert misc.xml

* modify uts

* reviewed code change

* reviewed code change

* code reviewed

* code review

* log changed

* do StringUtils.encodeForFormat when create allocationExec

* code review && limit taskCountMax to partitionNumbers

* modify docs

* code review

Co-authored-by: yuezhang <yuezhang@freewheel.tv>
2021-03-06 14:36:52 +05:30
Atul Mohan 6040c30fcd
Upgrade jetty to latest version (#10937)
* Upgrade jetty

* Fix license
2021-03-04 08:28:50 -06:00
Alexander Saydakov f930cf14d6
Use the latest Apache DataSketches release 2.0.0 (#10917)
* use the latest Apache DataSketches release 2.0.0

* updated datasketches version

Co-authored-by: AlexanderSaydakov <AlexanderSaydakov@users.noreply.github.com>
2021-02-26 07:52:00 -06:00
Suneet Saldanha bc7004006f
Update dependency-check plugin (#10883)
* Use dependency-check aggregate

* oops
2021-02-16 19:22:04 -08:00
Will Xu c8d2654605
Use native git for git-commit-id-plugin to speed up build (#10881)
* Segment timeline doesn't show results older than 3 months

* Adoption testing patch for web segment timeline view and also refactoring default time config

* Changing git-commit-id-plugin to use native git, shaving off 15% off build time

Co-authored-by: dev <dev@dev.minitoken.com>
2021-02-12 09:31:07 -08:00
Jihoon Son a2b5e01142
Bump DataSketches memory to 1.3.0 (#10789) 2021-02-04 18:39:52 -08:00
Himadri Singh 1c1b396eaa
AWS Web Identity / IRSA Support (#10541)
* AWS Web Identity Support

required for AWS IRSA

* Update kinesis-ingestion.md

* disabling coverage tests

https://github.com/apache/druid/pull/10541#issuecomment-737558213

* exclude coverage

* Update licenses.yaml
2021-01-25 18:44:02 +05:30
Jihoon Son 95065bdf1a
Bump dev version to 0.22.0-SNAPSHOT (#10759) 2021-01-15 13:16:23 -08:00
Himanshu c7b1212a43
AWS RDS token based password provider (#9518)
* refresh db pwd

* aws iam token password provider

* fix analyze-dependencies build

* fix doc build

* add  ut for BasicDataSourceExt

* more doc updates

* more  doc update

* moving aws  token password  provider to new extension

* remove duplicate changes

* make  all config inline

* extension docs

* refresh db  password  in SQL Firehose code path as well

* add ut

* fix build

* add new extension to distribution

* rds lib is not provided

* fix license build

* add version to license

* change parent version to 0.19.0-snapshot

* address review comments

* fix core/ code coverage

* Update server/src/main/java/org/apache/druid/metadata/BasicDataSourceExt.java

Co-authored-by: Clint Wylie <cjwylie@gmail.com>

* address review comments

* fix spellchecker

* remove inadvertant website file change

Co-authored-by: Clint Wylie <cjwylie@gmail.com>
2021-01-06 21:15:29 -08:00
Xavier Léauté b7a16d08a6
Update Apache Kafka to 2.7.0 (#10701)
- align scala versions to match Kafka
2020-12-22 13:56:00 -08:00
Himanshu ac1882bf74
kubernetes based discovery druid extension to run Druid on K8S without Zookeeper (#10544)
* honor zk enablement config in more places in druid code

* kubernetes based discovery module

* fix spotbugs check

* fix intellij checks error

* fix doc link to kubernetes.md from extension

* make spellchecker happy

* update license.yaml

* fix dependency check errors

* update extension coverage

* UTs for BaseNodeRoleWatcher

* fix forbidden-api check

* update k8s module coverage ignores

* add Bouncy Castle License being same as MIT License for license checking purposes

* further update licenses.yaml

* label/annotation pre-existence assumption

* address review comment
2020-12-14 21:10:31 -08:00
Jihoon Son abcf624a2e
Bump up jackson-databind to 2.10.5.1 (#10655)
* Bump up jackson version to 2.10.5.1

* only jackson-databind

* license
2020-12-09 13:54:47 -08:00
Suneet Saldanha c94be8a945
Revert "Update google client libraries (#10536)" (#10599)
This reverts commit 4537016cad.
2020-12-03 20:14:52 +05:30
Ayush Kulshrestha d0c2ede50c
Added CronScheduler support as a proof to clock drift while emitting metrics (#10448)
Co-authored-by: Ayush Kulshrestha <ayush.kulshrestha@miqdigital.com>
2020-11-25 12:31:38 +01:00
Nishant Bangarwa 4537016cad
Update google client libraries (#10536)
modify license.yaml

Update google oauth client version
2020-11-20 15:23:30 -08:00
Suneet Saldanha 6c8a77b7a9
Bump jetty to latest version (#10563)
This addresses CVE-2020-27216 which was flagged by the security vulnerability
job.
2020-11-09 08:51:36 -08:00
Jonathan Wei 65c0d64676
Update version to 0.21.0-SNAPSHOT (#10450)
* [maven-release-plugin] prepare release druid-0.21.0

* [maven-release-plugin] prepare for next development iteration

* Update web-console versions
2020-10-03 16:08:34 -07:00
Abhishek Agarwal d057c5149f
Fix the offset setting in GoogleStorage#get (#10449)
* Fix the offset in get of GCP object

* upgrade compute dependency

* fix version

* review comments

* missed
2020-10-01 08:38:58 -07:00
Igor Dvorzhak d0ee2e3a48
Upgrade ORC to 1.5.10 version (#10291) 2020-09-18 13:38:45 -07:00
Xavier Léauté 225490474d
Update Kafka dependencies to 2.6.0 (#10286)
* update Kafka dependencies to Kafka 2.6.0
* switch to Scala 2.13 build of Kafka
* update integration tests
* update Kafka tutorial
2020-08-15 07:56:40 -07:00
Richard Startin e363b1cd20
Update RoaringBitmap to 0.9.0 (#9987) 2020-07-23 19:29:25 -07:00
Gian Merlino eeaf609fc0
Update Jetty to 9.4.30.v20200611. (#10098)
* Update Jetty to 9.4.30.v20200611.

This is the latest version currently available in the 9.4.x line.

* Various adjustments.

* Class name fixes.

* Remove unused HttpClientModule code.

* Add coverage suppressions.

* Another coverage suppression.

* Fix wildcards.
2020-07-07 14:24:02 -07:00
Clint Wylie c86e7ce30b
bump version to 0.20.0-SNAPSHOT (#10124) 2020-07-06 15:08:32 -07:00
frank chen 60c6bd5b4c
support Aliyun OSS service as deep storage (#9898)
* init commit, all tests passed

* fix format

Signed-off-by: frank chen <frank.chen021@outlook.com>

* data stored successfully

* modify config path

* add doc

* add aliyun-oss extension to project

* remove descriptor deletion code to avoid warning message output by aliyun client

* fix warnings reported by lgtm-com

* fix ci warnings

Signed-off-by: frank chen <frank.chen021@outlook.com>

* fix errors reported by intellj inspection check

Signed-off-by: frank chen <frank.chen021@outlook.com>

* fix doc spelling check

Signed-off-by: frank chen <frank.chen021@outlook.com>

* fix dependency warnings reported by ci

Signed-off-by: frank chen <frank.chen021@outlook.com>

* fix warnings reported by CI

Signed-off-by: frank chen <frank.chen021@outlook.com>

* add package configuration to support showing extension info

Signed-off-by: frank chen <frank.chen021@outlook.com>

* add IT test cases and fix bugs

Signed-off-by: frank chen <frank.chen021@outlook.com>

* 1. code review comments adopted
2. change schema from 'aliyun-oss' to 'oss'

Signed-off-by: frank chen <frank.chen021@outlook.com>

* add license info

Signed-off-by: frank chen <frank.chen021@outlook.com>

* fix doc

Signed-off-by: frank chen <frank.chen021@outlook.com>

* exclude execution of IT testcases of OSS extension from CI

Signed-off-by: frank chen <frank.chen021@outlook.com>

* put the extensions under contrib group and add to distribution

* fix names in test cases

* add unit test to cover OssInputSource

* fix names in test cases

* fix dependency problem reported by CI

Signed-off-by: frank chen <frank.chen021@outlook.com>
2020-07-01 22:20:53 -07:00
Jihoon Son 657f8ee80f
Fix RetryQueryRunner to actually do the job (#10082)
* Fix RetryQueryRunner to actually do the job

* more javadoc

* fix test and checkstyle

* don't combine for testing

* address comments

* fix unit tests

* always initialize response context in cachingClusteredClient

* fix subquery

* address comments

* fix test

* query id for builders

* make queryId optional in the builders and ClusterQueryResult

* fix test

* suppress tests and unused methods

* exclude groupBy builder

* fix jacoco exclusion

* add tests for builders

* address comments

* don't truncate
2020-07-01 14:02:21 -07:00
Suneet Saldanha 15a0b4ffe2
Filter http requests by http method (#10085)
* Filter http requests by http method

Add a config that allows a user which http methods to allow against their
Druid server.

Druid will only accept http requests with the method: GET, PUT, POST, DELETE
and OPTIONS.
If a Druid admin wants to allow other methods, they can do so by using the
ServerConfig#allowedHttpMethods config.

If a Druid user would like to disallow OPTIONS, this can be done by changing
the AuthConfig#allowUnauthenticatedHttpOptions config

* Exclude OPTIONS from always supported HTTP methods

Add HEAD as an allowed method for web console e2e tests

* fix docs

* fix security IT

* Actually fix the web console e2e tests

* Ignore icode coverage for nitialization classes

* code review
2020-06-29 16:59:31 -07:00
Clint Wylie ec1f443a5c
update avatica to handle additional character sets over jdbc (#10074)
* update avatica to handle additional character sets over jdbc

* update license yaml, fix test

* oops
2020-06-24 19:58:34 -07:00