Commit Graph

1677 Commits

Author SHA1 Message Date
George Shiqi Wu fc929e6d25
upgrade shade (#15042)
Upgrade maven shade plugin to try to fix build failures

Sometimes we get maven shade errors in our integ tests becasue we don't run clean in between runs to clear the cache in order to speed them up. This can lead to the below error.
Error: Failed to execute goal org.apache.maven.plugins:maven-shade-plugin:3.2.4:shade (opentelemetry-extension) on project opentelemetry-emitter: Error creating shaded jar: duplicate entry: META-INF/services/org.apache.druid.opentelemetry.shaded.io.grpc.NameResolverProvider

See: https://issues.apache.org/jira/projects/MSHADE/issues/MSHADE-425?filter=allissues
An example run that failed: https://github.com/apache/druid/actions/runs/6301662092/job/17117142375?pr=14887

According to the ticket this is fixed by updating shade to 3.4.1.

When I updated to 3.4.1 I kept running into a different issue during static checks. (Caused by: java.lang.NoClassDefFoundError: com/github/rvesse/airline/parser/errors/ParseException)

I had to add the createDependencyReducedPom: false to get the build to pass.

The dependency reduced pom feature was added in 3.3.0 which we were not using before so setting it explicitly to false should not be a issue. https://issues.apache.org/jira/browse/MSHADE-36)
2023-09-27 18:19:21 +05:30
Xavier Léauté 22abc10f24
update RoaringBitmap to 0.9.49 (#15006)
* update RoaringBitmap to 0.9.49

update RoaringBitmap from 0.9.0 to 0.9.49

Many optimizations and improvements have gone into recent releases of
RoaringBitmap. It seems worthwhile to incorporate those.

* implement workaround for BatchIterator interface change

* add test case for BatchIteratorAdapter.advanceIfNeeded
2023-09-20 15:52:27 -07:00
Tejaswini Bandlamudi dec6a0aa14
Update google client apis to latest version (#14414)
Currently Druid is using google apis client 1.26.0 version and google-oauth-client-1.26.0.jar in particular is bringing following CVEs CVE-2020-7692, CVE-2021-22573. Despite the CVEs being false positives, they're causing red security scans on Druid distribution. Hence updating the version to latest version with these CVE fixes.
2023-09-11 12:27:23 +05:30
Zoltan Haindrich 54336e2a3e
Imporve on incremental compilation (#14860)
This patch fixes a few issues toward #14858

1. some phony classes were added to enable maven to track the compilation of those classes
2. cyclonedx 2.7.9 seem to handle incremental compilation better; it had a PR relating to that
3. needed to update root pom to 25
4. update antlr to 4.5.3 older one didn't really worked incrementally; 4.5.3 works much better
2023-08-24 16:06:16 +05:30
Tejaswini Bandlamudi d87056e708
Upgrade guava version to 31.1-jre (#14767)
Currently, Druid is using Guava 16.0.1 version. This upgrade to 31.1-jre fixes the following issues.

CVE-2018-10237 (Unbounded memory allocation in Google Guava 11.0 through 24.x before 24.1.1 allows remote attackers to conduct denial of service attacks against servers that depend on this library and deserialize attacker-provided data because the AtomicDoubleArray class (when serialized with Java serialization) and the CompoundOrdering class (when serialized with GWT serialization) perform eager allocation without appropriate checks on what a client has sent and whether the data size is reasonable). We don't use Java or GWT serializations. Despite being false positive they're causing red security scans on Druid distribution.
Latest version of google-client-api is incompatible with the existing Guava version. This PR unblocks Update google client apis to latest version #14414
2023-08-22 12:09:53 +05:30
dependabot[bot] 1e14df4c49
Bump com.ibm.icu:icu4j from 55.1 to 73.2 (#14853)
* Bump com.ibm.icu:icu4j from 55.1 to 73.2

Bumps [com.ibm.icu:icu4j](https://github.com/unicode-org/icu) from 55.1 to 73.2.
- [Release notes](https://github.com/unicode-org/icu/releases)
- [Commits](https://github.com/unicode-org/icu/commits)

---
updated-dependencies:
- dependency-name: com.ibm.icu:icu4j
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

* update licenses.yaml

* update Unicode/ICU license

* fix license check for unicode/icu

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2023-08-18 09:10:39 -04:00
Lucas Capistrant 9c124f2cde
Add a configurable bufferPeriod between when a segment is marked unused and deleted by KillUnusedSegments duty (#12599)
* Add new configurable buffer period to create gap between mark unused and kill of segment

* Changes after testing

* fixes and improvements

* changes after initial self review

* self review changes

* update sql statement that was lacking last_used

* shore up some code in SqlMetadataConnector after self review

* fix derby compatibility and improve testing/docs

* fix checkstyle violations

* Fixes post merge with master

* add some unit tests to improve coverage

* ignore test coverage on new UpdateTools cli tool

* another attempt to ignore UpdateTables in coverage check

* change column name to used_flag_last_updated

* fix a method signature after column name switch

* update docs spelling

* Update spelling dictionary

* Fixing up docs/spelling and integrating altering tasks table with my alteration code

* Update NULL values for used_flag_last_updated in the background

* Remove logic to allow segs with null used_flag_last_updated to be killed regardless of bufferPeriod

* remove unneeded things now that the new column is automatically updated

* Test new background row updater method

* fix broken tests

* fix create table statement

* cleanup DDL formatting

* Revert adding columns to entry table by default

* fix compilation issues after merge with master

* discovered and fixed metastore inserts that were breaking integration tests

* fixup forgotten insert by using pattern of sharing now timestamp across columns

* fix issue introduced by merge

* fixup after merge with master

* add some directions to docs in the case of segment table validation issues
2023-08-17 19:32:51 -05:00
dependabot[bot] 2cc3bd6383
Bump joda-time:joda-time from 2.12.4 to 2.12.5 (#14855)
* Bump joda-time:joda-time from 2.12.4 to 2.12.5

Bumps [joda-time:joda-time](https://github.com/JodaOrg/joda-time) from 2.12.4 to 2.12.5.
- [Release notes](https://github.com/JodaOrg/joda-time/releases)
- [Changelog](https://github.com/JodaOrg/joda-time/blob/main/RELEASE-NOTES.txt)
- [Commits](https://github.com/JodaOrg/joda-time/compare/v2.12.4...v2.12.5)

---
updated-dependencies:
- dependency-name: joda-time:joda-time
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

* update licenses.yaml

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2023-08-17 11:24:22 -07:00
dependabot[bot] 2a7fbf2ab4
Bump org.apache.directory.api:api-util from 1.0.3 to 2.1.3 (#14852)
Bumps [org.apache.directory.api:api-util](https://github.com/apache/directory-ldap-api) from 1.0.3 to 2.1.3.
- [Commits](https://github.com/apache/directory-ldap-api/compare/1.0.3...2.1.3)

---
updated-dependencies:
- dependency-name: org.apache.directory.api:api-util
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-17 08:56:34 -07:00
dependabot[bot] 97c3773012
Bump commons-cli:commons-cli from 1.3.1 to 1.5.0 (#14837)
* Bump commons-cli:commons-cli from 1.3.1 to 1.5.0

Bumps commons-cli:commons-cli from 1.3.1 to 1.5.0.

---
updated-dependencies:
- dependency-name: commons-cli:commons-cli
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* update licenses.yaml

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2023-08-16 07:37:56 -07:00
dependabot[bot] 8be7751dbd
Bump org.tukaani:xz from 1.8 to 1.9 (#14839)
* Bump org.tukaani:xz from 1.8 to 1.9

Bumps org.tukaani:xz from 1.8 to 1.9.

---
updated-dependencies:
- dependency-name: org.tukaani:xz
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* update licenses.yaml

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2023-08-16 07:37:29 -07:00
dependabot[bot] e55fe67535
Bump apache.curator.version from 5.4.0 to 5.5.0 (#14843)
* Bump apache.curator.version from 5.4.0 to 5.5.0

Bumps `apache.curator.version` from 5.4.0 to 5.5.0.

Updates `org.apache.curator:curator-client` from 5.4.0 to 5.5.0
- [Commits](https://github.com/apache/curator/compare/apache-curator-5.4.0...apache-curator-5.5.0)

Updates `org.apache.curator:curator-framework` from 5.4.0 to 5.5.0
- [Commits](https://github.com/apache/curator/compare/apache-curator-5.4.0...apache-curator-5.5.0)

Updates `org.apache.curator:curator-recipes` from 5.4.0 to 5.5.0
- [Commits](https://github.com/apache/curator/compare/apache-curator-5.4.0...apache-curator-5.5.0)

Updates `org.apache.curator:curator-x-discovery` from 5.4.0 to 5.5.0
- [Commits](https://github.com/apache/curator/compare/apache-curator-5.4.0...apache-curator-5.5.0)

Updates `org.apache.curator:curator-test` from 5.4.0 to 5.5.0
- [Commits](https://github.com/apache/curator/compare/apache-curator-5.4.0...apache-curator-5.5.0)

---
updated-dependencies:
- dependency-name: org.apache.curator:curator-client
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.apache.curator:curator-framework
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.apache.curator:curator-recipes
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.apache.curator:curator-x-discovery
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: org.apache.curator:curator-test
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* update licenses.yaml

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2023-08-16 07:36:58 -07:00
dependabot[bot] 9be0f64f50
Bump org.apache.commons:commons-compress from 1.21 to 1.23.0 (#14820)
* Bump org.apache.commons:commons-compress from 1.21 to 1.23.0

Bumps org.apache.commons:commons-compress from 1.21 to 1.23.0.

---
updated-dependencies:
- dependency-name: org.apache.commons:commons-compress
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* update licenses.yaml

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2023-08-15 20:08:54 -04:00
dependabot[bot] aeeeed3b35
Bump protobuf.version from 3.21.7 to 3.24.0 (#14823)
* Bump protobuf.version from 3.21.7 to 3.24.0

Bumps `protobuf.version` from 3.21.7 to 3.24.0.

Updates `com.google.protobuf:protobuf-java` from 3.21.7 to 3.24.0
- [Release notes](https://github.com/protocolbuffers/protobuf/releases)
- [Changelog](https://github.com/protocolbuffers/protobuf/blob/main/protobuf_release.bzl)
- [Commits](https://github.com/protocolbuffers/protobuf/compare/v3.21.7...v3.24.0)

Updates `com.google.protobuf:protobuf-java-util` from 3.21.7 to 3.24.0

---
updated-dependencies:
- dependency-name: com.google.protobuf:protobuf-java
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: com.google.protobuf:protobuf-java-util
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* update licenses.yaml

* fix licenses.yaml

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2023-08-15 12:15:55 -07:00
dependabot[bot] 8abdaa239b
Bump dropwizard.metrics.version from 4.0.0 to 4.2.19 (#14824)
* Bump dropwizard.metrics.version from 4.0.0 to 4.2.19

Bumps `dropwizard.metrics.version` from 4.0.0 to 4.2.19.

Updates `io.dropwizard.metrics:metrics-core` from 4.0.0 to 4.2.19
- [Release notes](https://github.com/dropwizard/metrics/releases)
- [Commits](https://github.com/dropwizard/metrics/compare/v4.0.0...v4.2.19)

Updates `io.dropwizard.metrics:metrics-jmx` from 4.0.0 to 4.2.19
- [Release notes](https://github.com/dropwizard/metrics/releases)
- [Commits](https://github.com/dropwizard/metrics/compare/v4.0.0...v4.2.19)

---
updated-dependencies:
- dependency-name: io.dropwizard.metrics:metrics-core
  dependency-type: direct:production
  update-type: version-update:semver-minor
- dependency-name: io.dropwizard.metrics:metrics-jmx
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* update licenses.yaml

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2023-08-15 12:15:29 -07:00
dependabot[bot] 2fdf5b195f
Bump org.assertj:assertj-core from 3.19.0 to 3.24.2 (#14815)
Bumps org.assertj:assertj-core from 3.19.0 to 3.24.2.

---
updated-dependencies:
- dependency-name: org.assertj:assertj-core
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-15 07:39:48 -07:00
dependabot[bot] 0967048dca
Bump org.scala-lang:scala-library from 2.13.9 to 2.13.11 (#14826)
Bumps [org.scala-lang:scala-library](https://github.com/scala/scala) from 2.13.9 to 2.13.11.
- [Release notes](https://github.com/scala/scala/releases)
- [Commits](https://github.com/scala/scala/compare/v2.13.9...v2.13.11)

---
updated-dependencies:
- dependency-name: org.scala-lang:scala-library
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-08-15 07:38:34 -07:00
dependabot[bot] 5f54ae7d27
Bump org.apache.maven.plugins:maven-surefire-plugin (#14813)
Bumps [org.apache.maven.plugins:maven-surefire-plugin](https://github.com/apache/maven-surefire) from 3.0.0-M7 to 3.1.2.
- [Release notes](https://github.com/apache/maven-surefire/releases)
- [Commits](https://github.com/apache/maven-surefire/compare/surefire-3.0.0-M7...surefire-3.1.2)
2023-08-14 23:07:53 -07:00
dependabot[bot] d5d483fdc9
Bump org.apache.rat:apache-rat-plugin from 0.12 to 0.15 (#14817) 2023-08-14 23:07:03 -07:00
Rishabh Singh 0dc305f9e4
Upgrade hibernate validator version to fix CVE-2019-10219 (#14757) 2023-08-14 11:50:51 +05:30
dependabot[bot] e2d2afce46
Bump postgresql from 42.4.1 to 42.6.0 (#13959)
* Bump postgresql from 42.4.1 to 42.6.0

Bumps [postgresql](https://github.com/pgjdbc/pgjdbc) from 42.4.1 to 42.6.0.
- [Release notes](https://github.com/pgjdbc/pgjdbc/releases)
- [Changelog](https://github.com/pgjdbc/pgjdbc/blob/master/CHANGELOG.md)
- [Commits](https://github.com/pgjdbc/pgjdbc/compare/REL42.4.1...REL42.6.0)

---
updated-dependencies:
- dependency-name: org.postgresql:postgresql
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

* update licenses.yaml

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2023-08-12 19:17:00 -04:00
Soumyava afe22907a5
Calcite upgrade 1.35 (#14510)
* Update to Calcite 1.35.0
* Update from.ftl for Calcite 1.35.0.
* Fixed tests in Calcite upgrade by doing the following:
1. Added a new rule, CoreRules.PROJECT_FILTER_TRANSPOSE_WHOLE_PROJECT_EXPRESSIONS, to Base rules
2. Refactored the CorrelateUnnestRule
3. Updated CorrelateUnnestRel accordingly
4. Fixed a case with selector filters on the left where Calcite was eliding the virtual column
5. Additional test cases for fixes in 2,3,4
6. Update to StringListAggregator to fail a query if separators are not propagated appropriately
* Refactored for testcases to pass after the upgrade, introduced 2 new data sources for handling filters and select projects
* Added a literalSqlAggregator as the upgraded Calcite involved changes to subquery remove rule. This corrected plans for 2 queries with joins and subqueries by replacing an useless literal dimension with a post agg. Additionally a test with COUNT DISTINCT and FILTER which was failing with Calcite 1.21 is added here which passes with 1.35
* Updated to latest avatica and updated code as SqlUnknownTimeStamp is now used in Calcite which needs to be resolved to a timestamp literal
* Added a wrapper segment ref to use for unnest and filter segment reference
2023-08-11 12:47:16 -07:00
Tejaswini Bandlamudi a45b25fa1d
Removes support for Hadoop 2 (#14763)
Removing Hadoop 2 support as discussed in https://lists.apache.org/list?dev@druid.apache.org:lte=1M:hadoop
2023-08-09 17:47:52 +05:30
Tejaswini Bandlamudi 550a66d71e
Upgrade jackson-databind to 2.12.7 (#14770)
The current version of jackson-databind is flagged for vulnerabilities CVE-2020-28491 (Although cbor format is not used in druid), CVE-2020-36518 (Seems genuine as deeply nested json in can cause resource exhaustion). Updating the dependency to the latest version 2.12.7 to fix these vulnerabilities.
2023-08-09 12:22:16 +05:30
Tejaswini Bandlamudi d0403f00fd
upgrade org.mozilla:rhino (#14765) 2023-08-08 12:17:59 +05:30
Xavier Léauté c1c2435aee
upgrade core Apache Kafka dependencies to 3.5.1 (#14721)
Release notes: https://downloads.apache.org/kafka/3.5.1/RELEASE_NOTES.html
Announcement: https://lists.apache.org/thread/p7jyv3ys7b6jowcb6lys7821qcbcpb07

Release notes: https://downloads.apache.org/kafka/3.5.0/RELEASE_NOTES.html
Announcement: https://lists.apache.org/thread/s6x3zvkrv32v5y8yb6hh31h57spdbylk
2023-08-02 01:08:40 -07:00
Pranav 8a10b46dd8
Adding the PropertyNamingStrategies from jackson for fixing hadoop ingestion (#14671) 2023-08-01 20:02:43 +05:30
dependabot[bot] e99bab2fd3
Bump org.xerial.snappy:snappy-java from 1.1.10.1 to 1.1.10.3 (#14641)
Bumps [org.xerial.snappy:snappy-java](https://github.com/xerial/snappy-java) from 1.1.10.1 to 1.1.10.3.
- [Release notes](https://github.com/xerial/snappy-java/releases)
- [Commits](https://github.com/xerial/snappy-java/compare/v1.1.10.1...v1.1.10.3)

---
updated-dependencies:
- dependency-name: org.xerial.snappy:snappy-java
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
2023-07-26 06:27:55 -07:00
AmatyaAvadhanula 0412f40d36
Prepare master branch for next release, 28.0.0 (#14595)
* Prepare master branch for next release, 28.0.0
2023-07-18 09:22:30 +05:30
Atul Mohan 03d6d395a0
Extension to read and ingest iceberg data files (#14329)
This adds a new contrib extension: druid-iceberg-extensions which can be used to ingest data stored in Apache Iceberg format. It adds a new input source of type iceberg that connects to a catalog and retrieves the data files associated with an iceberg table and provides these data file paths to either an S3 or HDFS input source depending on the warehouse location.

Two important dependencies associated with Apache Iceberg tables are:

Catalog : This extension supports reading from either a Hive Metastore catalog or a Local file-based catalog. Support for AWS Glue is not available yet.
Warehouse : This extension supports reading data files from either HDFS or S3. Adapters for other cloud object locations should be easy to add by extending the AbstractInputSourceAdapter.
2023-07-18 08:59:57 +05:30
Sam Rash 0dcb19f7e3
Add Continuous Profiling to Unit Tests (#14506)
Uses a custom continusou jfr profiler.

Modifies the github actions for tests to do profiling only in the case
of jdk17, as the profiler requires jdk17+ to use the JFR streaming API
plus a few other language features in the code.

Continuous Profiling service is provided to the Apache Druid project
free of charge by Imply and any committer can request free access to
the UI.
2023-07-12 17:50:38 -07:00
Gian Merlino 63ee69b4e8
Claim full support for Java 17. (#14384)
* Claim full support for Java 17.

No production code has changed, except the startup scripts.

Changes:

1) Allow Java 17 without DRUID_SKIP_JAVA_CHECK.

2) Include the full list of opens and exports on both Java 11 and 17.

3) Document that Java 17 is both supported and preferred.

4) Switch some tests from Java 11 to 17 to get better coverage on the
   preferred version.

* Doc update.

* Update errorprone.

* Update docker_build_containers.sh.

* Update errorprone in licenses.yaml.

* Add some more run-javas.

* Additional run-javas.

* Update errorprone.

* Suppress new errorprone error.

* Add exports and opens in ForkingTaskRunner for Java 11+.

Test, doc changes.

* Additional errorprone updates.

* Update for errorprone.

* Restore old fomatting in LdapCredentialsValidator.

* Copy bin/ too.

* Fix Java 15, 17 build line in docker_build_containers.sh.

* Update busybox image.

* One more java command.

* Fix interpolation.

* IT commandline refinements.

* Switch to busybox 1.34.1-glibc.

* POM adjustments, build and test one IT on 17.

* Additional debugging.

* Fix silly thing.

* Adjust command line.

* Add exports and opens one more place.

* Additional harmonization of strong encapsulation parameters.
2023-07-07 12:52:35 -07:00
Jan Werner 95115d722a
CVE fixes - update of multiple dependencies. (#14519)
Apache Druid brings multiple direct and transitive dependencies that are affected by plethora of CVEs.
This PR attempts to update all the dependencies that did not require code refactoring.
This PR modifies pom files, license file and OWASP Dependency Check suppression file.
2023-07-07 20:27:30 +05:30
Clint Wylie 277aaa5c57
remove druid.processing.columnCache.sizeBytes and CachingIndexed, combine string column implementations (#14500)
* combine string column implementations
changes:
* generic indexed, front-coded, and auto string columns now all share the same column and index supplier implementations
* remove CachingIndexed implementation, which I think is largely no longer needed by the switch of many things to directly using ByteBuffer, avoiding the cost of creating Strings
* remove ColumnConfig.columnCacheSizeBytes since CachingIndexed was the only user
2023-07-02 19:37:15 -07:00
Tejaswini Bandlamudi baa64e6d8a
update hadoop version to 3.3.6 (#14489) 2023-06-28 15:03:10 +05:30
Tejaswini Bandlamudi 72cf91fbc0
Upgrade Avro to latest version (#14440)
Upgraded Avro to 1.11.1
2023-06-24 14:51:30 +05:30
Clint Wylie 9b1779734b
fix website mvn build (#14458)
changes:
* fix website mvn build
* remove the i18n/en.json file add to gitignore
* add spellcheck to mvn test phase
2023-06-22 12:14:23 -07:00
Hardik Bajaj 1ea9158a50
Added new SysMonitorOshi v0 using Oshi library (#14359)
Added a new monitor SysMonitorOshi to replace SysMonitor. The new monitor has a wider support for different machine architectures including ARM instances. Please switch to SysMonitorOshi as SysMonitor is now deprecated and will be removed in future releases.
2023-06-20 20:57:58 +05:30
Alexander Saydakov f6169d437b
use the latest datasketches-java-4.1.0 (#14430)
Co-authored-by: AlexanderSaydakov <AlexanderSaydakov@users.noreply.github.com>
2023-06-14 16:03:56 -07:00
Alexander Saydakov 4131c0df13
use the latest datasketches-java-4.0.0 (#14334)
* use the latest datasketches-java-4.0.0

* updated versions of datasketches

* adjusted expectation

* fixed the expectations

---------

Co-authored-by: AlexanderSaydakov <AlexanderSaydakov@users.noreply.github.com>
2023-05-27 22:19:18 -07:00
Clint Wylie e833a4700d
suppress hadoop3 cve that seem not applicable to us (#14252) 2023-05-10 23:08:05 -07:00
Tejaswini Bandlamudi 774073b2e7
Update Hadoop3 as default build version (#14005)
Hadoop 2 often causes red security scans on Druid distribution because of the dependencies it brings. We want to move away from Hadoop 2 and provide Hadoop 3 distribution available. Switch druid to building with Hadoop 3 by default. Druid will still be compatible with Hadoop 2 and users can build hadoop-2 compatible distribution using hadoop2 profile.
2023-04-26 12:52:51 +05:30
Tejaswini Bandlamudi cb302e1bd1
Use apache-jar-resource-bundle:1.5 instead of 1.5-SNAPSHOT (#14054) 2023-04-10 18:55:39 +05:30
Clint Wylie 1aef72aa7e
Bump up the version in pom to 27.0.0 in preparation of release (#14051) 2023-04-10 14:56:59 +05:30
Sandeep ccdf30e399
Bump Joda-Time version for current DateTimeZone data (#13999) 2023-03-29 20:15:49 +05:30
frankgrimes97 2f98675285
Tuple sketch SQL support (#13887)
This PR is a follow-up to #13819 so that the Tuple sketch functionality can be used in SQL for both ingestion using Multi-Stage Queries (MSQ) and also for analytic queries against Tuple sketch columns.
2023-03-28 18:47:12 +05:30
Abhishek Agarwal 139a058ba7
Use sonatype maven central for plugin repositories (#13961)
* Change search order of maven repositories

* Update pom.xml
2023-03-23 15:35:47 +05:30
abhagraw c52d15d65d
Fixing security vulnerability check errors (#13956)
* Fixing security vulnerability check errors

* Updating javax.el to jakarta.el

* Adding cron job trigger on changes to suppressions file
2023-03-23 11:10:06 +05:30
Benedict Jin cee2dfd768
Upgrade ZK from 3.5.9 to 3.5.10 to avoid data inconsistency risk (#13715) 2023-03-15 19:21:09 +05:30
Paul Rogers 4493275d88
Use Maven central repo rather than Apache (#13921)
* Use Maven central repo rather than Apache

* Disable snapshots
2023-03-13 10:49:32 -07:00
Elliott Freis d93fdb2632
Bump CycloneDX module to address POM errors (#13878)
* Bump CycloneDX module to address POM errors

* Including web-console in the PR

---------

Co-authored-by: Elliott Freis <elliottfreis@Elliott-Freis.earth.dynamic.blacklight.net>
2023-03-03 15:39:15 +05:30
Clint Wylie 08b5951cc5
merge druid-core, extendedset, and druid-hll into druid-processing to simplify everything (#13698)
* merge druid-core, extendedset, and druid-hll into druid-processing to simplify everything
* fix poms and license stuff
* mockito is evil
* allow reset of JvmUtils RuntimeInfo if tests used static injection to override
2023-02-17 14:27:41 -08:00
Adarsh Sanjeev e8330e95f5
Update Apache Kafka dependencies to 3.4.0 (#13802)
Release notes:
- https://downloads.apache.org/kafka/3.4.0/RELEASE_NOTES.html
2023-02-15 15:15:13 +05:30
Xavier Léauté 698670c88e
update core Apache Kafka dependencies to 3.3.2 (#13717)
Release notes:
- https://downloads.apache.org/kafka/3.3.2/RELEASE_NOTES.html
2023-01-27 21:00:01 -08:00
Dongjoon Hyun 2503095296
Publish SBOM artifacts (#13648) 2023-01-10 16:08:10 +05:30
abhagraw 74a76c74b1
Updating dependency check version (#13649) 2023-01-10 14:43:19 +05:30
Kashif Faraz 78ae0b7533
Upgrade to netty 4.1.86.Final to address CVEs (#13604)
This commit addresses the following CVEs:
- CVE-2021-43797
- CVE-2022-41881
2022-12-23 01:44:01 +05:30
Peter Stöckli df55768535
Add CodeQL workflow (#13477)
* workflower: Add CodeQL workflow

* add modified CodeQL build config
2022-12-21 09:24:39 +05:30
Jason Koch 6c44dd8175
perf: core/TextReader for faster json ingestion (#13545)
* perf: provide a custom utf8 specific buffered line iterator (benchmark)

Benchmark                         Mode  Cnt     Score     Error  Units
JsonLineReaderBenchmark.baseline  avgt   15  3459.871 ± 106.175  us/op

* perf: provide a custom utf8 specific buffered line iterator

Benchmark                         Mode  Cnt     Score    Error  Units
JsonLineReaderBenchmark.baseline  avgt   15  3022.053 ± 51.286  us/op

* perf: provide a custom utf8 specific buffered line iterator (more tests)

* perf: provide a custom utf8 specific buffered line iterator (pr feedback)

Ensure field visibility is as limited as possible

Null check for buffer in constructor

* perf: provide a custom utf8 specific buffered line iterator (pr feedback)

Remove additional 'finished' variable.

* perf: provide a custom utf8 specific buffered line iterator (more tests and bugfix)
2022-12-19 23:12:37 -08:00
Kashif Faraz 7cf761cee4
Prepare master branch for next release, 26.0.0 (#13401)
* Prepare master branch for next release, 26.0.0

* Use docker image for druid 24.0.1

* Fix version in druid-it-cases pom.xml
2022-11-22 15:31:01 +05:30
Paul Rogers 81d005f267
Druid Catalog basics (#13165)
Druid catalog basics

Catalog object model for tables, columns
Druid metadata DB storage (as an extension)
REST API to update the catalog (as an extension)
Integration tests
Model only: no planner integration yet
2022-11-12 15:30:22 -08:00
Didip Kerabat c875f4bd04
Upgrade curator to 5.4.0 (#13302) 2022-11-03 11:26:19 -07:00
Dr. Sizzles e5ad24ff9f
Support for middle manager less druid, tasks launch as k8s jobs (#13156)
* Support for middle manager less druid, tasks launch as k8s jobs

* Fixing forking task runner test

* Test cleanup, dependency cleanup, intellij inspections cleanup

* Changes per PR review

Add configuration option to disable http/https proxy for the k8s client
Update the docs to provide more detail about sidecar support

* Removing un-needed log lines

* Small changes per PR review

* Upon task completion we callback to the overlord to update the status / locaiton, for slower k8s clusters, this reduces locking time significantly

* Merge conflict fix

* Fixing tests and docs

* update tiny-cluster.yaml 

changed `enableTaskLevelLogPush` to `encapsulatedTask`

* Apply suggestions from code review

Co-authored-by: Abhishek Agarwal <1477457+abhishekagarwal87@users.noreply.github.com>

* Minor changes per PR request

* Cleanup, adding test to AbstractTask

* Add comment in peon.sh

* Bumping code coverage

* More tests to make code coverage happy

* Doh a duplicate dependnecy

* Integration test setup is weird for k8s, will do this in a different PR

* Reverting back all integration test changes, will do in anotbher PR

* use StringUtils.base64 instead of Base64

* Jdk is nasty, if i compress in jdk 11 in jdk 17 the decompressed result is different

Co-authored-by: Rahul Gidwani <r_gidwani@apple.com>
Co-authored-by: Abhishek Agarwal <1477457+abhishekagarwal87@users.noreply.github.com>
2022-11-02 19:44:47 -07:00
chi-chi weng 72c16097ac
Fix Apache Commons Text CVE-2022-42889 (#13226)
* Fix Apache Commons Text  CVE-2022-42889

Fix Apache Commons Text  CVE-2022-42889

https://nvd.nist.gov/vuln/detail/CVE-2022-42889

* Update license

Co-authored-by: Frank Chen <frank.chen021@outlook.com>
2022-10-26 10:04:32 +08:00
Frank Chen d30cf8c308
Dependency cleanup (#13194)
* Clean up dependency in extensions

* Bump protobuf/aws.sdk

* Bump aws-sdk to 1.12.317

* Fix CI

* Fix CI

* Update license

* Update license
2022-10-10 20:34:38 +08:00
Xavier Léauté eff7edb603
update core Apache Kafka dependencies to 3.3.1 (#13176)
Announcement:
- https://blogs.apache.org/kafka/entry/what-rsquo-s-new-in

Release notes:
- https://archive.apache.org/dist/kafka/3.3.0/RELEASE_NOTES.html
- https://downloads.apache.org/kafka/3.3.1/RELEASE_NOTES.html
2022-10-04 12:52:16 -07:00
AmatyaAvadhanula acafd0d1e0
Upgrade kafka version to 3.2.3 to fix CVE (#13142)
Upgrade to 3.2.3 to fix CVE: https://nvd.nist.gov/vuln/detail/CVE-2022-34917
2022-09-28 10:47:09 +05:30
Gian Merlino 5733360dfd
Update Snappy to 1.1.8.4. (#13081)
* Update Snappy to 1.1.8.4.

Prior to this, because snappy-java wasn't included in dependencyManagement,
we actually shipped multiple different versions for different extensions,
ranging from 1.1.7.1 to 1.1.8.4. Now, we standardize on 1.1.8.4.

Among other things, this enables the tests to pass on M1 Macs.

* Update snappy-java versions in licenses.yaml.
2022-09-14 15:13:47 -07:00
Adam Peck ee22663dd3
Add interpolation to JsonConfigurator (#13023)
* Add interpolation to JsonConfigurator

* Fix checkstyle

* Fix tests by removing common-text override

* Add back commons-text without version

* Remove unused hadoopDir configs

* Move some stuff to hopefully pass coverage
2022-09-07 12:48:01 +05:30
senthilkv 3d9aef225d
compressed big decimal - module (#10705)
Compressed Big Decimal is an extension which provides support for 
Mutable big decimal value that can be used to accumulate values 
without losing precision or reallocating memory. This type helps in 
absolute precision arithmetic on large numbers in applications, 
where greater level of accuracy is required, such as financial 
applications, currency based transactions. This helps avoid rounding 
issues where in potentially large amount of money can be lost.

Accumulation requires that the two numbers have the same scale, 
but does not require that they are of the same size. If the value 
being accumulated has a larger underlying array than this value 
(the result), then the higher order bits are dropped, similar to what 
happens when adding a long to an int and storing the result in an 
int. A compressed big decimal that holds its data with an embedded 
array.

Compressed big decimal is an absolute number based complex type 
based on big decimal in Java. This supports all the functionalities 
supported by Java Big Decimal. Java Big Decimal is not mutable in 
order to avoid big garbage collection issues. Compressed big decimal 
is needed to mutate the value in the accumulator.
2022-09-06 00:06:57 -07:00
Gian Merlino 48ceab2153
Add Java 17 information to documentation. (#12990)
The docs say Java 17 support is experimental, and give tips on running
successfully with Java 17.

This patch also removes java.base/jdk.internal.perf and
jdk.management/com.sun.management.internal from the list of required
exports and opens, because they were formerly needed for JvmMonitor,
which was rewritten in #12481 to use MXBeans instead.
2022-08-30 12:32:49 -07:00
Gian Merlino 9eb20e5e7c
Remove dependency on jvm-attach. (#12989)
This dependency was no longer needed after #12481, but remained because
it was used for a (now useless) test. This patch removes the test and
the dependency.
2022-08-29 14:18:33 -07:00
Abhishek Agarwal 618757352b
Bump up the version to 25.0.0 (#12975)
* Bump up the version to 25.0.0

* Fix the version in console
2022-08-29 11:27:38 +05:30
Adam Peck 21b73bde20
Update Curator to 5.3.0 (#12939)
* Update Curator to 5.3.0

* Update licenses.yaml

* Fix inspections + add tests.

* Fix checkstyle

* Another intellij inspection fix

* Update curator exclusions

* Cleanup new exhibitor references

* Remove unused dep and checkstyle fix
2022-08-26 18:23:40 -07:00
Paul Rogers cfed036091
Add the new integration test framework (#12368)
This commit is a first draft of the revised integration test framework which provides:
- A new directory, integration-tests-ex that holds the new integration test structure. (For now, the existing integration-tests is left unchanged.)
- Maven module druid-it-tools to hold code placed into the Docker image.
- Maven module druid-it-image to build the Druid-only test image from the tarball produced in distribution. (Dependencies live in their "official" image.)
- Maven module druid-it-cases that holds the revised tests and the framework itself. The framework includes file-based test configuration, test-specific clients, test initialization and updated versions of some of the common test support classes.

The integration test setup is primarily a huge mass of details. This approach refactors many of those details: from how the image is built and configured to how the Docker Compose scripts are structured to test configuration. An extensive set of "readme" files explains those details. Rather than repeat that material here, please consult those files for explanations.
2022-08-24 17:03:23 +05:30
Gian Merlino d7d15ba51f
Add druid-multi-stage-query extension. (#12918)
* Add druid-multi-stage-query extension.

* Adjustments from CI.

* Task ID validation.

* Various changes from code review.

* Remove unnecessary code.

* LGTM-related.
2022-08-23 18:44:01 -07:00
Xavier Léauté 752e42a312
fix running integration tests on macos aarch64 (#12913)
* add osx-aarch_64 netty-transport-native-kqueue native dependency
* align docker-java dependency versions using bom and update to 3.2.13
2022-08-17 18:03:24 +02:00
dependabot[bot] f70f7b4b89
Bump postgresql from 42.3.3 to 42.4.1 (#12871)
* Bump postgresql from 42.3.3 to 42.4.1

Bumps [postgresql](https://github.com/pgjdbc/pgjdbc) from 42.3.3 to 42.4.1.
- [Release notes](https://github.com/pgjdbc/pgjdbc/releases)
- [Changelog](https://github.com/pgjdbc/pgjdbc/blob/master/CHANGELOG.md)
- [Commits](https://github.com/pgjdbc/pgjdbc/compare/REL42.3.3...REL42.4.1)

---
updated-dependencies:
- dependency-name: org.postgresql:postgresql
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <support@github.com>

* update licenses.yaml

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2022-08-16 23:25:39 +02:00
Paul Rogers 4706a4c572
Docker build for the revised ITs (#12707)
* Docker build for the revised ITs

* Fix POM versions

* Update comments from review suggestions
2022-08-10 14:17:33 +05:30
Gian Merlino ef6811ef88
Improved Java 17 support and Java runtime docs. (#12839)
* Improved Java 17 support and Java runtime docs.

1) Add a "Java runtime" doc page with information about supported
   Java versions, garbage collection, and strong encapsulation..

2) Update asm and equalsverifier to versions that support Java 17.

3) Add additional "--add-opens" lines to surefire configuration, so
   tests can pass successfully under Java 17.

4) Switch openjdk15 tests to openjdk17.

5) Update FrameFile to specifically mention Java runtime incompatibility
   as the cause of not being able to use Memory.map.

6) Update SegmentLoadDropHandler to log an error for Errors too, not
   just Exceptions. This is important because an IllegalAccessError is
   encountered when the correct "--add-opens" line is not provided,
   which would otherwise be silently ignored.

7) Update example configs to use druid.indexer.runner.javaOptsArray
   instead of druid.indexer.runner.javaOpts. (The latter is deprecated.)

* Adjustments.

* Use run-java in more places.

* Add run-java.

* Update .gitignore.

* Exclude hadoop-client-api.

Brought in when building on Java 17.

* Swap one more usage of java.

* Fix the run-java script.

* Fix flag.

* Include link to Temurin.

* Spelling.

* Update examples/bin/run-java

Co-authored-by: Xavier Léauté <xl+github@xvrl.net>

Co-authored-by: Xavier Léauté <xl+github@xvrl.net>
2022-08-03 23:16:05 -07:00
Karan Kumar 3290b49754
Log4j bump to 2.18 due to [LOG4J2-3419] (#12847)
* Log4j bump to 2.18 due to [LOG4J2-3419]

* Fixing license issues
2022-08-02 23:25:40 -07:00
PJ Fanning 188b5b0027
Upgrade to jetty 9.4.48.v20220622 due to CVEs (#12801)
* Upgrade to jetty 9.4.48.v20220622 due to CVEs

* Update licenses.yaml
2022-07-26 10:11:48 +08:00
Kashif Faraz 9e5f0109fd
Fix CVE-2022-2048 (jetty) and CVE-2022-31159 (aws-java-sdk-s3) (#12807)
Changes:
- Upgrade aws sdk version from `1.12.37` to `1.12.264`
- Upgrade jetty version from `9.4.41.v20210516` to `9.4.47.v20220610`
2022-07-21 13:08:18 +05:30
Paul Rogers ee15c238cc
Clone Calcite planner to access validator (#12708)
Done in preparation for the "single-pass" planner.
2022-07-14 18:10:33 -07:00
Gian Merlino 9c925b4f09
Frame format for data transfer and short-term storage. (#12745)
* Frame format for data transfer and short-term storage.

As we move towards query execution plans that involve more transfer
of data between servers, it's important to have a data format that
provides for doing this more efficiently than the options available to
us today.

This patch adds:

- Columnar frames, which support fast querying.
- Row-based frames, which support fast sorting via memory comparison
  and fast whole-row copies via memory copying.
- Frame files, a container format that can be stored on disk or
  transferred between servers.

The idea is we should use row-based frames when data is expected to
be sorted, and columnar frames when data is expected to be queried.

The code in this patch is not used in production yet. Therefore, the
patch involves minimal changes outside of the org.apache.druid.frame
package.  The main ones are adjustments to SqlBenchmark to add benchmarks
for queries on frames, and the addition of a "forEach" method to Sequence.

* Fixes based on tests, static analysis.

* Additional fixes.

* Skip DS mapping tests on JDK 14+

* Better JDK checking in tests.

* Fix imports.

* Additional comment.

* Adjustments from code review.

* Update test case.
2022-07-08 20:42:06 -07:00
Jianhuan Liu 4574dea5e9
Use MXBeans to get GC metrics #12476 (#12481)
* jvm gc to mxbeans

* add zgc and shenandoah #12476

* remove tryCreateGcCounter

* separate the space collector

* blend GcGenerationCollector into GcCollector

* add jdk surefire argLine
2022-07-08 14:32:06 +08:00
PJ Fanning 059aba781a
issue-12628: upgrade jetty to 9.4.41.v20210516 due to CVE (#12629)
* upgrade jetty to 9.4.41.v20210516 due to cve

* Update licenses.yaml
2022-07-07 00:20:01 +08:00
imply-cheddar e3128e3fa3
Poison stupid pool (#12646)
* Poison StupidPool and fix resource leaks

There are various resource leaks from test setup as well as some
corners in query processing.  We poison the StupidPool to start failing
tests when the leaks come and fix any issues uncovered from that so
that we can start from a clean baseline.

Unfortunately, because of how poisoning works,
we can only fail future checkouts from the same pool,
which means that there is a natural race between a
leak happening -> GC occurs -> leak detected -> pool poisoned.

This race means that, depending on interleaving of tests,
if the very last time that an object is checked out
from the pool leaks, then it won't get caught.
At some point in the future, something will catch it,
 however and from that point on it will be deterministic.

* Remove various things left over from iterations

* Clean up FilterAnalysis and add javadoc on StupidPool

* Revert changes to .idea/misc.xml that accidentally got pushed

* Style and test branches

* Stylistic woes
2022-07-03 14:36:22 -07:00
Rohan Garg c09b5a2294
Fix skipTests build flag (#12716)
* fix skipTests

* Skip console UTs with skipTests

* Use skipTests in skip-tests profile
2022-06-29 21:59:26 -07:00
Rui Chen 068bea6334
deps: upgrade mysql-connector-java to v5.1.49 (#12704) 2022-06-29 23:15:46 +08:00
Paul Rogers f83fab699e
Add IT-related changes pulled out of PR #12368 (#12673)
This commit contains changes made to the existing ITs to support the new ITs.

Changes:
- Make the "custom node role" code usable by the new ITs. 
- Use flag `-DskipITs` to skips the integration tests but runs unit tests.
- Use flag `-DskipUTs` skips unit tests but runs the "new" integration tests.
- Expand the existing Druid profile, `-P skip-tests` to skip both ITs and UTs.
2022-06-26 02:13:59 +05:30
Dr. Sizzles 7291c92f4f
Adding zstandard compression library (#12408)
* Adding zstandard compression library

* 1. Took @clintropolis's advice to have ZStandard decompressor use the byte array when the buffers are not direct.
2. Cleaned up checkstyle issues.

* Fixing zstandard version to latest stable version in pom's and updating license files

* Removing zstd from benchmarks and adding to processing (poms)

* fix the intellij inspection issue

* Removing the prefix v for the version in the license check for ztsd

* Fixing license checks

Co-authored-by: Rahul Gidwani <r_gidwani@apple.com>
2022-05-28 17:01:44 -07:00
Abhishek Agarwal 32fe4d1324
Use a different repository to download sigar artifacts. (#12561) 2022-05-24 14:42:51 +05:30
Clint Wylie 2d8dbb53e0
update to latest lz4 1.8.0 (#12557) 2022-05-21 16:02:20 +08:00
Xavier Léauté ec41dfb535
upgrade core Apache Kafka dependencies to 3.2.0 (#12538)
Announcement: https://blogs.apache.org/kafka/entry/what-s-new-in-apache8
Release notes: https://downloads.apache.org/kafka/3.2.0/RELEASE_NOTES.html
2022-05-19 09:04:52 -07:00
Gian Merlino 4631cff2a9
Free ByteBuffers in tests and fix some bugs. (#12521)
* Ensure ByteBuffers allocated in tests get freed.

Many tests had problems where a direct ByteBuffer would be allocated
and then not freed. This is bad because it causes flaky tests.

To fix this:

1) Add ByteBufferUtils.allocateDirect(size), which returns a ResourceHolder.
   This makes it easy to free the direct buffer. Currently, it's only used
   in tests, because production code seems OK.

2) Update all usages of ByteBuffer.allocateDirect (off-heap) in tests either
   to ByteBuffer.allocate (on-heap, which are garbaged collected), or to
   ByteBufferUtils.allocateDirect (wherever it seemed like there was a good
   reason for the buffer to be off-heap). Make sure to close all direct
   holders when done.

* Changes based on CI results.

* A different approach.

* Roll back BitmapOperationTest stuff.

* Try additional surefire memory.

* Revert "Roll back BitmapOperationTest stuff."

This reverts commit 49f846d9e3.

* Add TestBufferPool.

* Revert Xmx change in tests.

* Better behaved NestedQueryPushDownTest. Exit tests on OOME.

* Fix TestBufferPool.

* Remove T1C from ARM tests.

* Somewhat safer.

* Fix tests.

* Fix style stuff.

* Additional debugging.

* Reset null / expr configs better.

* ExpressionLambdaAggregatorFactory thread-safety.

* Alter forkNode to try to get better info when a JVM crashes.

* Fix buffer retention in ExpressionLambdaAggregatorFactory.

* Remove unused import.
2022-05-19 07:42:29 -07:00
Kashif Faraz 7ab2170802
Use datasketches version 3.2.0 (#12509)
Changes:
- Use apache datasketches version 3.2.0.
- Remove unsafe reflection-based usage of datasketch internals added in #12022
2022-05-13 11:28:15 +05:30
Abhishek Radhakrishnan 9177515be2
Add IPAddress java library as dependency and migrate IPv4 functions to use the new library. (#11634)
* Add ipaddress library as dependency.

* IPv4 functions to use the inet.ipaddr package.

* Remove unused imports.

* Add new function.

* Minor rename.

* Add more unit tests.

* IPv4 address expr utils unit tests and address options.

* Adjust the IPv4Util functions.

* Move the UTs a bit around.

* Javadoc comments.

* Add license info for IPAddress.

* Fix groupId, artifact and version in license.yaml.

* Remove redundant subnet in messages - fixes UT.

* Remove unused commons-net dependency for /processing project.

* Make class and methods public so it can be accessed.

* Add initial version of benchmark

* Add subnetutils package for benchmarks.

* Auto generate ip addresses.

* Add more v4 address representations in setup to avoid bias.

* Use ThreadLocalRandom to avoid forbidden API usage.

* Adjust IPv4AddressBenchmark to adhere to codestyle rules.

* Update ipaddress library to latest 5.3.4

* Add ipaddress package dependency to benchmarks project.
2022-05-11 22:06:20 -07:00
aggarwalakshay dd8781f5b0
Upgrade dependency-check-maven to 7.0.4 (#12441) 2022-05-01 22:45:58 +08:00
Gian Merlino 72d15ab321
JvmMonitor: Handle more generation and collector scenarios. (#12469)
* JvmMonitor: Handle more generation and collector scenarios.

ZGC on Java 11 only has a generation 1 (there is no 0). This causes
a NullPointerException when trying to extract the spacesCount for
generation 0. In addition, ZGC on Java 15 has a collector number 2
but no spaces in generation 2, which breaks the assumption that
collectors always have same-numbered spaces.

This patch adjusts things to be more robust, enabling the JvmMonitor
to work properly for ZGC on both Java 11 and 15.

* Test adjustments.

* Improve surefire arglines.

* Need a placeholder
2022-04-27 11:18:40 -07:00