Commit Graph

1558 Commits

Author SHA1 Message Date
Abhishek Agarwal ae5fc18946 [maven-release-plugin] prepare for next development iteration 2022-09-10 04:55:09 +00:00
Abhishek Agarwal 09991dcf17 [maven-release-plugin] prepare release druid-24.0.0-rc2 2022-09-10 04:54:58 +00:00
Abhishek Agarwal a6f5a74650 [maven-release-plugin] prepare for next development iteration 2022-09-08 14:18:57 +00:00
Abhishek Agarwal 2319f10cb6 [maven-release-plugin] prepare release druid-24.0.0-rc1 2022-09-08 14:18:46 +00:00
Abhishek Agarwal 8b321eae14 Update version to 24.0.0 2022-08-25 09:55:55 +00:00
Paul Rogers cfed036091
Add the new integration test framework (#12368)
This commit is a first draft of the revised integration test framework which provides:
- A new directory, integration-tests-ex that holds the new integration test structure. (For now, the existing integration-tests is left unchanged.)
- Maven module druid-it-tools to hold code placed into the Docker image.
- Maven module druid-it-image to build the Druid-only test image from the tarball produced in distribution. (Dependencies live in their "official" image.)
- Maven module druid-it-cases that holds the revised tests and the framework itself. The framework includes file-based test configuration, test-specific clients, test initialization and updated versions of some of the common test support classes.

The integration test setup is primarily a huge mass of details. This approach refactors many of those details: from how the image is built and configured to how the Docker Compose scripts are structured to test configuration. An extensive set of "readme" files explains those details. Rather than repeat that material here, please consult those files for explanations.
2022-08-24 17:03:23 +05:30
Gian Merlino d7d15ba51f
Add druid-multi-stage-query extension. (#12918)
* Add druid-multi-stage-query extension.

* Adjustments from CI.

* Task ID validation.

* Various changes from code review.

* Remove unnecessary code.

* LGTM-related.
2022-08-23 18:44:01 -07:00
Xavier Léauté 752e42a312
fix running integration tests on macos aarch64 (#12913)
* add osx-aarch_64 netty-transport-native-kqueue native dependency
* align docker-java dependency versions using bom and update to 3.2.13
2022-08-17 18:03:24 +02:00
dependabot[bot] f70f7b4b89
Bump postgresql from 42.3.3 to 42.4.1 (#12871)
* Bump postgresql from 42.3.3 to 42.4.1

Bumps [postgresql](https://github.com/pgjdbc/pgjdbc) from 42.3.3 to 42.4.1.
- [Release notes](https://github.com/pgjdbc/pgjdbc/releases)
- [Changelog](https://github.com/pgjdbc/pgjdbc/blob/master/CHANGELOG.md)
- [Commits](https://github.com/pgjdbc/pgjdbc/compare/REL42.3.3...REL42.4.1)

---
updated-dependencies:
- dependency-name: org.postgresql:postgresql
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* update licenses.yaml

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2022-08-16 23:25:39 +02:00
Paul Rogers 4706a4c572
Docker build for the revised ITs (#12707)
* Docker build for the revised ITs

* Fix POM versions

* Update comments from review suggestions
2022-08-10 14:17:33 +05:30
Gian Merlino ef6811ef88
Improved Java 17 support and Java runtime docs. (#12839)
* Improved Java 17 support and Java runtime docs.

1) Add a "Java runtime" doc page with information about supported
   Java versions, garbage collection, and strong encapsulation..

2) Update asm and equalsverifier to versions that support Java 17.

3) Add additional "--add-opens" lines to surefire configuration, so
   tests can pass successfully under Java 17.

4) Switch openjdk15 tests to openjdk17.

5) Update FrameFile to specifically mention Java runtime incompatibility
   as the cause of not being able to use Memory.map.

6) Update SegmentLoadDropHandler to log an error for Errors too, not
   just Exceptions. This is important because an IllegalAccessError is
   encountered when the correct "--add-opens" line is not provided,
   which would otherwise be silently ignored.

7) Update example configs to use druid.indexer.runner.javaOptsArray
   instead of druid.indexer.runner.javaOpts. (The latter is deprecated.)

* Adjustments.

* Use run-java in more places.

* Add run-java.

* Update .gitignore.

* Exclude hadoop-client-api.

Brought in when building on Java 17.

* Swap one more usage of java.

* Fix the run-java script.

* Fix flag.

* Include link to Temurin.

* Spelling.

* Update examples/bin/run-java

Co-authored-by: Xavier Léauté <xl+github@xvrl.net>

Co-authored-by: Xavier Léauté <xl+github@xvrl.net>
2022-08-03 23:16:05 -07:00
Karan Kumar 3290b49754
Log4j bump to 2.18 due to [LOG4J2-3419] (#12847)
* Log4j bump to 2.18 due to [LOG4J2-3419]

* Fixing license issues
2022-08-02 23:25:40 -07:00
PJ Fanning 188b5b0027
Upgrade to jetty 9.4.48.v20220622 due to CVEs (#12801)
* Upgrade to jetty 9.4.48.v20220622 due to CVEs

* Update licenses.yaml
2022-07-26 10:11:48 +08:00
Kashif Faraz 9e5f0109fd
Fix CVE-2022-2048 (jetty) and CVE-2022-31159 (aws-java-sdk-s3) (#12807)
Changes:
- Upgrade aws sdk version from `1.12.37` to `1.12.264`
- Upgrade jetty version from `9.4.41.v20210516` to `9.4.47.v20220610`
2022-07-21 13:08:18 +05:30
Paul Rogers ee15c238cc
Clone Calcite planner to access validator (#12708)
Done in preparation for the "single-pass" planner.
2022-07-14 18:10:33 -07:00
Gian Merlino 9c925b4f09
Frame format for data transfer and short-term storage. (#12745)
* Frame format for data transfer and short-term storage.

As we move towards query execution plans that involve more transfer
of data between servers, it's important to have a data format that
provides for doing this more efficiently than the options available to
us today.

This patch adds:

- Columnar frames, which support fast querying.
- Row-based frames, which support fast sorting via memory comparison
  and fast whole-row copies via memory copying.
- Frame files, a container format that can be stored on disk or
  transferred between servers.

The idea is we should use row-based frames when data is expected to
be sorted, and columnar frames when data is expected to be queried.

The code in this patch is not used in production yet. Therefore, the
patch involves minimal changes outside of the org.apache.druid.frame
package.  The main ones are adjustments to SqlBenchmark to add benchmarks
for queries on frames, and the addition of a "forEach" method to Sequence.

* Fixes based on tests, static analysis.

* Additional fixes.

* Skip DS mapping tests on JDK 14+

* Better JDK checking in tests.

* Fix imports.

* Additional comment.

* Adjustments from code review.

* Update test case.
2022-07-08 20:42:06 -07:00
Jianhuan Liu 4574dea5e9
Use MXBeans to get GC metrics #12476 (#12481)
* jvm gc to mxbeans

* add zgc and shenandoah #12476

* remove tryCreateGcCounter

* separate the space collector

* blend GcGenerationCollector into GcCollector

* add jdk surefire argLine
2022-07-08 14:32:06 +08:00
PJ Fanning 059aba781a
issue-12628: upgrade jetty to 9.4.41.v20210516 due to CVE (#12629)
* upgrade jetty to 9.4.41.v20210516 due to cve

* Update licenses.yaml
2022-07-07 00:20:01 +08:00
imply-cheddar e3128e3fa3
Poison stupid pool (#12646)
* Poison StupidPool and fix resource leaks

There are various resource leaks from test setup as well as some
corners in query processing.  We poison the StupidPool to start failing
tests when the leaks come and fix any issues uncovered from that so
that we can start from a clean baseline.

Unfortunately, because of how poisoning works,
we can only fail future checkouts from the same pool,
which means that there is a natural race between a
leak happening -> GC occurs -> leak detected -> pool poisoned.

This race means that, depending on interleaving of tests,
if the very last time that an object is checked out
from the pool leaks, then it won't get caught.
At some point in the future, something will catch it,
 however and from that point on it will be deterministic.

* Remove various things left over from iterations

* Clean up FilterAnalysis and add javadoc on StupidPool

* Revert changes to .idea/misc.xml that accidentally got pushed

* Style and test branches

* Stylistic woes
2022-07-03 14:36:22 -07:00
Rohan Garg c09b5a2294
Fix skipTests build flag (#12716)
* fix skipTests

* Skip console UTs with skipTests

* Use skipTests in skip-tests profile
2022-06-29 21:59:26 -07:00
Rui Chen 068bea6334
deps: upgrade mysql-connector-java to v5.1.49 (#12704) 2022-06-29 23:15:46 +08:00
Paul Rogers f83fab699e
Add IT-related changes pulled out of PR #12368 (#12673)
This commit contains changes made to the existing ITs to support the new ITs.

Changes:
- Make the "custom node role" code usable by the new ITs. 
- Use flag `-DskipITs` to skips the integration tests but runs unit tests.
- Use flag `-DskipUTs` skips unit tests but runs the "new" integration tests.
- Expand the existing Druid profile, `-P skip-tests` to skip both ITs and UTs.
2022-06-26 02:13:59 +05:30
Dr. Sizzles 7291c92f4f
Adding zstandard compression library (#12408)
* Adding zstandard compression library

* 1. Took @clintropolis's advice to have ZStandard decompressor use the byte array when the buffers are not direct.
2. Cleaned up checkstyle issues.

* Fixing zstandard version to latest stable version in pom's and updating license files

* Removing zstd from benchmarks and adding to processing (poms)

* fix the intellij inspection issue

* Removing the prefix v for the version in the license check for ztsd

* Fixing license checks

Co-authored-by: Rahul Gidwani <r_gidwani@apple.com>
2022-05-28 17:01:44 -07:00
Abhishek Agarwal 32fe4d1324
Use a different repository to download sigar artifacts. (#12561) 2022-05-24 14:42:51 +05:30
Clint Wylie 2d8dbb53e0
update to latest lz4 1.8.0 (#12557) 2022-05-21 16:02:20 +08:00
Xavier Léauté ec41dfb535
upgrade core Apache Kafka dependencies to 3.2.0 (#12538)
Announcement: https://blogs.apache.org/kafka/entry/what-s-new-in-apache8
Release notes: https://downloads.apache.org/kafka/3.2.0/RELEASE_NOTES.html
2022-05-19 09:04:52 -07:00
Gian Merlino 4631cff2a9
Free ByteBuffers in tests and fix some bugs. (#12521)
* Ensure ByteBuffers allocated in tests get freed.

Many tests had problems where a direct ByteBuffer would be allocated
and then not freed. This is bad because it causes flaky tests.

To fix this:

1) Add ByteBufferUtils.allocateDirect(size), which returns a ResourceHolder.
   This makes it easy to free the direct buffer. Currently, it's only used
   in tests, because production code seems OK.

2) Update all usages of ByteBuffer.allocateDirect (off-heap) in tests either
   to ByteBuffer.allocate (on-heap, which are garbaged collected), or to
   ByteBufferUtils.allocateDirect (wherever it seemed like there was a good
   reason for the buffer to be off-heap). Make sure to close all direct
   holders when done.

* Changes based on CI results.

* A different approach.

* Roll back BitmapOperationTest stuff.

* Try additional surefire memory.

* Revert "Roll back BitmapOperationTest stuff."

This reverts commit 49f846d9e3.

* Add TestBufferPool.

* Revert Xmx change in tests.

* Better behaved NestedQueryPushDownTest. Exit tests on OOME.

* Fix TestBufferPool.

* Remove T1C from ARM tests.

* Somewhat safer.

* Fix tests.

* Fix style stuff.

* Additional debugging.

* Reset null / expr configs better.

* ExpressionLambdaAggregatorFactory thread-safety.

* Alter forkNode to try to get better info when a JVM crashes.

* Fix buffer retention in ExpressionLambdaAggregatorFactory.

* Remove unused import.
2022-05-19 07:42:29 -07:00
Kashif Faraz 7ab2170802
Use datasketches version 3.2.0 (#12509)
Changes:
- Use apache datasketches version 3.2.0.
- Remove unsafe reflection-based usage of datasketch internals added in #12022
2022-05-13 11:28:15 +05:30
Abhishek Radhakrishnan 9177515be2
Add IPAddress java library as dependency and migrate IPv4 functions to use the new library. (#11634)
* Add ipaddress library as dependency.

* IPv4 functions to use the inet.ipaddr package.

* Remove unused imports.

* Add new function.

* Minor rename.

* Add more unit tests.

* IPv4 address expr utils unit tests and address options.

* Adjust the IPv4Util functions.

* Move the UTs a bit around.

* Javadoc comments.

* Add license info for IPAddress.

* Fix groupId, artifact and version in license.yaml.

* Remove redundant subnet in messages - fixes UT.

* Remove unused commons-net dependency for /processing project.

* Make class and methods public so it can be accessed.

* Add initial version of benchmark

* Add subnetutils package for benchmarks.

* Auto generate ip addresses.

* Add more v4 address representations in setup to avoid bias.

* Use ThreadLocalRandom to avoid forbidden API usage.

* Adjust IPv4AddressBenchmark to adhere to codestyle rules.

* Update ipaddress library to latest 5.3.4

* Add ipaddress package dependency to benchmarks project.
2022-05-11 22:06:20 -07:00
aggarwalakshay dd8781f5b0
Upgrade dependency-check-maven to 7.0.4 (#12441) 2022-05-01 22:45:58 +08:00
Gian Merlino 72d15ab321
JvmMonitor: Handle more generation and collector scenarios. (#12469)
* JvmMonitor: Handle more generation and collector scenarios.

ZGC on Java 11 only has a generation 1 (there is no 0). This causes
a NullPointerException when trying to extract the spacesCount for
generation 0. In addition, ZGC on Java 15 has a collector number 2
but no spaces in generation 2, which breaks the assumption that
collectors always have same-numbered spaces.

This patch adjusts things to be more robust, enabling the JvmMonitor
to work properly for ZGC on both Java 11 and 15.

* Test adjustments.

* Improve surefire arglines.

* Need a placeholder
2022-04-27 11:18:40 -07:00
Abhishek Agarwal 2fe053c5cb
Bump up the versions (#12480) 2022-04-27 14:28:20 +05:30
PJ Fanning 5099f5aa70
update httpclient due to cve (#12422)
https://github.com/apache/druid/issues/12421
2022-04-21 10:12:19 +08:00
Jihoon Son d2a6754692
Bump PostgreSQL JDBC driver to 42.3.3 (CVE-2022-21724) (#12410)
* Bump PostgreSQL JDBC driver to 42.3.3 (CVE-2022-21724)

* update license file
2022-04-09 15:38:26 +05:30
aggarwalakshay 7d5666109c
upgrade surefire 3.0.0-M6 (#12395)
* upgrade surefire 3.0.0-M6

* increasing memory
2022-04-04 23:56:15 -07:00
Xavier Léauté c3377bf744
upgrade maven-pmd-plugin to fix warning (#12349)
we sometimes see warnings similar to the one mentioned
https://issues.apache.org/jira/browse/MPMD-325

Upgrading the plugin should hopefully reduce occurrence of those.
2022-03-19 10:18:26 -07:00
dependabot[bot] 4ed1abca94
Bump slf4j.version from 1.7.12 to 1.7.36 (#11594)
Bump slf4j.version from 1.7.12 to 1.7.36

- [Release notes](Release notes: https://www.slf4j.org/news.html)

Updates `jcl-over-slf4j` from 1.7.12 to 1.7.36
- [Commits](https://github.com/qos-ch/slf4j/compare/v_1.7.12...v_1.7.36)

Updates `slf4j-simple` from 1.7.12 to 1.7.36
- [Commits](https://github.com/qos-ch/slf4j/compare/v_1.7.12...v_1.7.36)

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Suneet Saldanha <suneet@apache.org>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2022-03-18 13:45:44 -07:00
Xavier Léauté 6f0e5f25fa
update surefire plugin to 3.0.0-M4 (#12342)
stay on surefire 3.0.0-M4 until we can upgrade to 3.0.0-M6
with a fix for https://issues.apache.org/jira/browse/SUREFIRE-1815
causing issues in RetryUtilsTest.
2022-03-18 08:20:28 -07:00
dependabot[bot] a5dfb911de
Bump maven-site-plugin from 3.1 to 3.11.0 (#12310)
Bumps [maven-site-plugin](https://github.com/apache/maven-site-plugin) from 3.1 to 3.11.0.
- [Release notes](https://github.com/apache/maven-site-plugin/releases)
- [Commits](https://github.com/apache/maven-site-plugin/compare/maven-site-plugin-3.1...maven-site-plugin-3.11.0)

---
updated-dependencies:
- dependency-name: org.apache.maven.plugins:maven-site-plugin
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2022-03-17 15:17:29 +08:00
Xavier Léauté 5d02a91faa
upgrade Error Prone to 2.11 (requires Java 11) (#12306)
The latest version of Error Prone now requires Java 11. Upgrading means we can
remove a lot of the maven profile complexity required to run checks with Java 8.
This also requires switching our strict build to use Java 11.

* update error-prone to 2.11
* remove need for specific maven profiles for Java 8 and Java 15
* fix additional Error Prone warnings with Java 11
* update strict build to use Java 11
2022-03-14 19:40:48 -07:00
dependabot[bot] a1cdee2a3a
Bump jersey.version from 1.19.3 to 1.19.4 (#12290)
* Bump jersey.version from 1.19.3 to 1.19.4

Bumps `jersey.version` from 1.19.3 to 1.19.4.

Updates `jersey-client` from 1.19.3 to 1.19.4

Updates `jersey-core` from 1.19.3 to 1.19.4

Updates `jersey-grizzly2` from 1.19.3 to 1.19.4

Updates `jersey-guice` from 1.19.3 to 1.19.4

Updates `jersey-server` from 1.19.3 to 1.19.4

Updates `jersey-servlet` from 1.19.3 to 1.19.4

Updates `jersey-json` from 1.19.3 to 1.19.4

Updates `jersey-test-framework-core` from 1.19.3 to 1.19.4

Updates `jersey-test-framework-grizzly2` from 1.19.3 to 1.19.4

---
updated-dependencies:
- dependency-name: com.sun.jersey:jersey-client
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: com.sun.jersey:jersey-core
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: com.sun.jersey:jersey-grizzly2
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: com.sun.jersey.contribs:jersey-guice
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.sun.jersey:jersey-server
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.sun.jersey:jersey-servlet
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.sun.jersey:jersey-json
  dependency-type: direct:production
  update-type: version-update:semver-patch
- dependency-name: com.sun.jersey.jersey-test-framework:jersey-test-framework-core
  dependency-type: direct:development
  update-type: version-update:semver-patch
- dependency-name: com.sun.jersey.jersey-test-framework:jersey-test-framework-grizzly2
  dependency-type: direct:development
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* Update licenses.yaml

* Update licenses.yaml

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Clint Wylie <cwylie@apache.org>
2022-03-04 09:57:20 +08:00
Alexander Saydakov 50038d9344
latest datasketches-java-3.1.0 (#12224)
These changes are to use the latest datasketches-java-3.1.0 and also to restore support for quantile and HLL4 sketches to be able to grow larger than a given buffer in a buffer aggregator and move to heap in rare cases. This was discussed in #11544.

Co-authored-by: AlexanderSaydakov <AlexanderSaydakov@users.noreply.github.com>
2022-03-01 17:14:42 -08:00
Xavier Léauté d105519558
Replace use of PowerMock with Mockito (#12282)
Mockito now supports all our needs and plays much better with recent Java versions.
Migrating to Mockito also simplifies running the kind of tests that required PowerMock in the past. 

* replace all uses of powermock with mockito-inline
* upgrade mockito to 4.3.1 and fix use of deprecated methods
* import mockito bom to align all our mockito dependencies
* add powermock to forbidden-apis to avoid accidentally reintroducing it in the future
2022-02-27 22:47:09 -08:00
Xavier Léauté 1434197ee1
update airline dependency to 2.x (#12270)
* upgrade Airline to Airline 2
  https://github.com/airlift/airline is no longer maintained, updating to
  https://github.com/rvesse/airline (Airline 2) to use an actively
  maintained version, while minimizing breaking changes.

  Note, this is a backwards incompatible change, and extensions relying on
  the CliCommandCreator extension point will also need to be updated.

* fix dependency checks where jakarta.inject is now resolved first instead
  of javax.inject, due to Airline 2 using jakarta
2022-02-27 15:19:28 -08:00
Xavier Léauté 009dd9e09a
upgrade core Apache Kafka dependencies to 3.1.0 (#12203)
Announcement: https://blogs.apache.org/kafka/entry/what-s-new-in-apache7
Release notes: https://dist.apache.org/repos/dist/release/kafka/3.1.0/RELEASE_NOTES.html

* upgrade core Apache Kafka dependencies to 3.1.0
* fix use of private Kafka APIs
* remove deprecated test rules
* remove mock calls that weren't verified in the first place
* remove the need for powermock in KafkaLookupExtractorFactoryTest
* align curator-test version with curator itself
* update easymock to 4.3.0
2022-02-23 18:42:51 -08:00
Laksh Singla 4add2510ed
Add syntax support for PARTITIONED BY/CLUSTERED BY in INSERT queries (#12163)
This PR aims to add parser changes for supporting PARTITIONED BY and CLUSTERED BY as proposed in the issue #11929.
2022-02-08 16:23:15 +05:30
Uwe Schindler 1f7dd6d86c
Forbiddenapis: Split the guava16-only signatures file from main signatures file (#12170) 2022-01-19 17:50:28 -08:00
Benedict Jin b55f7a25fe
Fix forbiddenapis causing travis failing (#12158)
* Fix forbiddenapis causing travis failing

* Use failOnUnresolvableSignatures instead
2022-01-15 16:13:37 -08:00
Ivan Vankovich 6a93872586
OpenTelemetry emitter extension (#12015)
* Add OpenTelemetry emitter extension

* Fix build

* Fix checkstyle

* Add used undeclared dependencies

* Ignore unused declared dependencies
2022-01-15 12:18:04 +08:00
Frank Chen fe71fc414f
Update log4j2 to 2.17.1 (#12106)
Signed-off-by: frank chen <frank.chen021@outlook.com>
2021-12-30 19:18:16 -06:00