e43bb74c3a
Add MSQ Durable Storage Connector for Google Cloud Storage and change current Google Cloud Storage client library ( #15398 )
...
The PR addresses 2 things:
Add MSQ durable storage connector for GCS
Change GCS client library from the old Google API Client Library to the recommended Google Cloud Client Library. Ref: https://cloud.google.com/apis/docs/client-libraries-explained
2023-12-14 07:34:49 +05:30
5f86072456
Prepare master for Druid 29 ( #15121 )
...
Prepare master for Druid 29
2023-10-11 10:33:45 +05:30
dec6a0aa14
Update google client apis to latest version ( #14414 )
...
Currently Druid is using google apis client 1.26.0 version and google-oauth-client-1.26.0.jar in particular is bringing following CVEs CVE-2020-7692, CVE-2021-22573. Despite the CVEs being false positives, they're causing red security scans on Druid distribution. Hence updating the version to latest version with these CVE fixes.
2023-09-11 12:27:23 +05:30
e55fe67535
Bump apache.curator.version from 5.4.0 to 5.5.0 ( #14843 )
...
* Bump apache.curator.version from 5.4.0 to 5.5.0
Bumps `apache.curator.version` from 5.4.0 to 5.5.0.
Updates `org.apache.curator:curator-client` from 5.4.0 to 5.5.0
- [Commits](https://github.com/apache/curator/compare/apache-curator-5.4.0...apache-curator-5.5.0 )
Updates `org.apache.curator:curator-framework` from 5.4.0 to 5.5.0
- [Commits](https://github.com/apache/curator/compare/apache-curator-5.4.0...apache-curator-5.5.0 )
Updates `org.apache.curator:curator-recipes` from 5.4.0 to 5.5.0
- [Commits](https://github.com/apache/curator/compare/apache-curator-5.4.0...apache-curator-5.5.0 )
Updates `org.apache.curator:curator-x-discovery` from 5.4.0 to 5.5.0
- [Commits](https://github.com/apache/curator/compare/apache-curator-5.4.0...apache-curator-5.5.0 )
Updates `org.apache.curator:curator-test` from 5.4.0 to 5.5.0
- [Commits](https://github.com/apache/curator/compare/apache-curator-5.4.0...apache-curator-5.5.0 )
---
updated-dependencies:
- dependency-name: org.apache.curator:curator-client
dependency-type: direct:production
update-type: version-update:semver-minor
- dependency-name: org.apache.curator:curator-framework
dependency-type: direct:production
update-type: version-update:semver-minor
- dependency-name: org.apache.curator:curator-recipes
dependency-type: direct:production
update-type: version-update:semver-minor
- dependency-name: org.apache.curator:curator-x-discovery
dependency-type: direct:production
update-type: version-update:semver-minor
- dependency-name: org.apache.curator:curator-test
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
* update licenses.yaml
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Xavier Léauté <xvrl@apache.org>
2023-08-16 07:36:58 -07:00
0412f40d36
Prepare master branch for next release, 28.0.0 ( #14595 )
...
* Prepare master branch for next release, 28.0.0
2023-07-18 09:22:30 +05:30
95115d722a
CVE fixes - update of multiple dependencies. ( #14519 )
...
Apache Druid brings multiple direct and transitive dependencies that are affected by plethora of CVEs.
This PR attempts to update all the dependencies that did not require code refactoring.
This PR modifies pom files, license file and OWASP Dependency Check suppression file.
2023-07-07 20:27:30 +05:30
1aef72aa7e
Bump up the version in pom to 27.0.0 in preparation of release ( #14051 )
2023-04-10 14:56:59 +05:30
08b5951cc5
merge druid-core, extendedset, and druid-hll into druid-processing to simplify everything ( #13698 )
...
* merge druid-core, extendedset, and druid-hll into druid-processing to simplify everything
* fix poms and license stuff
* mockito is evil
* allow reset of JvmUtils RuntimeInfo if tests used static injection to override
2023-02-17 14:27:41 -08:00
9ffaba9c7f
Fix MySQL drivers setup for Revised ITs ( #13800 )
...
* download both mysql drivers and use org.mariadb.jdbc.Driver for now
* use com.mysql.jdbc.Driver
2023-02-15 11:03:25 +05:30
5ef689fc3f
Cloud deep storage tests in new IT framework (S3, GCS, Azure) ( #13535 )
...
* MSQ s3 deep storage tests
* Fix license check
* Getting config values from env variables
* Added s3TestUtils
* Merged AbstractITSQLBasedIngestionTest with AbstractITBatchIndexTest
* Fixing license issues
* Fixing checkstyle errors
* Fix spotbug errors
* Update s3util name in other files
* GCS and Azure deep storage tests
* Fix license and checkstyle errors
* Fix dependency error
* fix intellij check errors
* Copy credentials file in all containers
* Refactor and gcs file upload fix
* Fixing dependency check errors and codeQL warnings
* Fixing checkstyle errors
* Fixing intellij inspection errors
* Removing unrequired exceptions
* Addressing comments
2023-01-11 09:43:44 +05:30
56076d33fb
Worker retry for MSQ task ( #13353 )
...
* Initial commit.
* Fixing error message in retry exceeded exception
* Cleaning up some code
* Adding some test cases.
* Adding java docs.
* Finishing up state test cases.
* Adding some more java docs and fixing spot bugs, intellij inspections
* Fixing intellij inspections and added tests
* Documenting error codes
* Migrate current integration batch tests to equivalent MSQ tests (#13374 )
* Migrate current integration batch tests to equivalent MSQ tests using new IT framework
* Fix build issues
* Trigger Build
* Adding more tests and addressing comments
* fixBuildIssues
* fix dependency issues
* Parameterized the test and addressed comments
* Addressing comments
* fixing checkstyle errors
* Adressing comments
* Adding ITTest which kills the worker abruptly
* Review comments phase one
* Adding doc changes
* Adjusting for single threaded execution.
* Adding Sequential Merge PR state handling
* Merge things
* Fixing checkstyle.
* Adding new context param for fault tolerance.
Adding stale task handling in sketchFetcher.
Adding UT's.
* Merge things
* Merge things
* Adding parameterized tests
Created separate module for faultToleranceTests
* Adding missed files
* Review comments and fixing tests.
* Documentation things.
* Fixing IT
* Controller impl fix.
* Fixing racy WorkerSketchFetcherTest.java exception handling.
Co-authored-by: abhagraw <99210446+abhagraw@users.noreply.github.com>
Co-authored-by: Karan Kumar <cryptoe@karans-mbp.lan>
2023-01-11 07:38:29 +05:30
365474ff1d
New IT Framework - InputSource and InputFormat Tests ( #13597 )
...
* New IT Framework - InputSource and InputFormat Tests
* Fixing checkstyle errors
* Updating InputSource setup
* Updating queries to use druid DB
* Making metadata setup queries to be idempotent
* Restore intellij files
2023-01-04 10:40:05 +05:30
2f3b97194f
Fix harcoded version in pom file ( #13460 )
2022-12-01 10:10:04 +05:30
7cf761cee4
Prepare master branch for next release, 26.0.0 ( #13401 )
...
* Prepare master branch for next release, 26.0.0
* Use docker image for druid 24.0.1
* Fix version in druid-it-cases pom.xml
2022-11-22 15:31:01 +05:30
280a0f7158
Add sequential sketch merging to MSQ ( #13205 )
...
* Add sketch fetching framework
* Refactor code to support sequential merge
* Update worker sketch fetcher
* Refactor sketch fetcher
* Refactor sketch fetcher
* Add context parameter and threshold to trigger sequential merge
* Fix test
* Add integration test for non sequential merge
* Address review comments
* Address review comments
* Address review comments
* Resolve maxRetainedBytes
* Add new classes
* Renamed key statistics information class
* Rename fetchStatisticsSnapshotForTimeChunk function
* Address review comments
* Address review comments
* Update documentation and add comments
* Resolve build issues
* Resolve build issues
* Change worker APIs to async
* Address review comments
* Resolve build issues
* Add null time check
* Update integration tests
* Address review comments
* Add log messages and comments
* Resolve build issues
* Add unit tests
* Add unit tests
* Fix timing issue in tests
2022-11-22 09:56:32 +05:30
5172d76a67
Migrate current integration batch tests to equivalent MSQ tests ( #13374 )
...
* Migrate current integration batch tests to equivalent MSQ tests using new IT framework
* Fix build issues
* Trigger Build
* Adding more tests and addressing comments
* fixBuildIssues
* fix dependency issues
* Parameterized the test and addressed comments
* Addressing comments
* fixing checkstyle errors
* Adressing comments
2022-11-21 09:12:02 +05:30
81d005f267
Druid Catalog basics ( #13165 )
...
Druid catalog basics
Catalog object model for tables, columns
Druid metadata DB storage (as an extension)
REST API to update the catalog (as an extension)
Integration tests
Model only: no planner integration yet
2022-11-12 15:30:22 -08:00
728745a1d3
Add IT for MSQ task engine using the new IT framework ( #12992 )
...
* first test, serde causing problems
* serde working
* insert and select check
* Add cluster annotations for MSQ test cases
* Add cluster config for MSQ
* Add MSQ config to the pom.xml
* cleanup unnecessary changes
* Remove model classes
* Comments, checkstyle, check queries from file
* fixup test case name
* build failure fix
* review changes
* build failure fix
* Trigger Build
* Log the mismatch in QueryResultsVerifier
* Trigger Build
* Change the signature of the results verifier
* review changes
* LGTM fix
* build, change pom
* Trigger Build
* Trigger Build
* trigger build with minimal pom changes
* guice fix in tests
* travis.yml
2022-09-22 16:09:47 +05:30
7d332c6f6a
Suppress false CVEs ( #13026 )
...
* Suppress CVEs
* Add more suppressions
2022-09-06 11:46:56 +05:30
618757352b
Bump up the version to 25.0.0 ( #12975 )
...
* Bump up the version to 25.0.0
* Fix the version in console
2022-08-29 11:27:38 +05:30
21b73bde20
Update Curator to 5.3.0 ( #12939 )
...
* Update Curator to 5.3.0
* Update licenses.yaml
* Fix inspections + add tests.
* Fix checkstyle
* Another intellij inspection fix
* Update curator exclusions
* Cleanup new exhibitor references
* Remove unused dep and checkstyle fix
2022-08-26 18:23:40 -07:00
cfed036091
Add the new integration test framework ( #12368 )
...
This commit is a first draft of the revised integration test framework which provides:
- A new directory, integration-tests-ex that holds the new integration test structure. (For now, the existing integration-tests is left unchanged.)
- Maven module druid-it-tools to hold code placed into the Docker image.
- Maven module druid-it-image to build the Druid-only test image from the tarball produced in distribution. (Dependencies live in their "official" image.)
- Maven module druid-it-cases that holds the revised tests and the framework itself. The framework includes file-based test configuration, test-specific clients, test initialization and updated versions of some of the common test support classes.
The integration test setup is primarily a huge mass of details. This approach refactors many of those details: from how the image is built and configured to how the Docker Compose scripts are structured to test configuration. An extensive set of "readme" files explains those details. Rather than repeat that material here, please consult those files for explanations.
2022-08-24 17:03:23 +05:30
Vishesh Garg
Laksh Singla
Tejaswini Bandlamudi
dependabot[bot]
AmatyaAvadhanula
Jan Werner
Clint Wylie
abhagraw
Karan Kumar
Adarsh Sanjeev
Kashif Faraz
Paul Rogers
Abhishek Agarwal
Adam Peck