5ef689fc3f
Cloud deep storage tests in new IT framework (S3, GCS, Azure) ( #13535 )
...
* MSQ s3 deep storage tests
* Fix license check
* Getting config values from env variables
* Added s3TestUtils
* Merged AbstractITSQLBasedIngestionTest with AbstractITBatchIndexTest
* Fixing license issues
* Fixing checkstyle errors
* Fix spotbug errors
* Update s3util name in other files
* GCS and Azure deep storage tests
* Fix license and checkstyle errors
* Fix dependency error
* fix intellij check errors
* Copy credentials file in all containers
* Refactor and gcs file upload fix
* Fixing dependency check errors and codeQL warnings
* Fixing checkstyle errors
* Fixing intellij inspection errors
* Removing unrequired exceptions
* Addressing comments
2023-01-11 09:43:44 +05:30
56076d33fb
Worker retry for MSQ task ( #13353 )
...
* Initial commit.
* Fixing error message in retry exceeded exception
* Cleaning up some code
* Adding some test cases.
* Adding java docs.
* Finishing up state test cases.
* Adding some more java docs and fixing spot bugs, intellij inspections
* Fixing intellij inspections and added tests
* Documenting error codes
* Migrate current integration batch tests to equivalent MSQ tests (#13374 )
* Migrate current integration batch tests to equivalent MSQ tests using new IT framework
* Fix build issues
* Trigger Build
* Adding more tests and addressing comments
* fixBuildIssues
* fix dependency issues
* Parameterized the test and addressed comments
* Addressing comments
* fixing checkstyle errors
* Adressing comments
* Adding ITTest which kills the worker abruptly
* Review comments phase one
* Adding doc changes
* Adjusting for single threaded execution.
* Adding Sequential Merge PR state handling
* Merge things
* Fixing checkstyle.
* Adding new context param for fault tolerance.
Adding stale task handling in sketchFetcher.
Adding UT's.
* Merge things
* Merge things
* Adding parameterized tests
Created separate module for faultToleranceTests
* Adding missed files
* Review comments and fixing tests.
* Documentation things.
* Fixing IT
* Controller impl fix.
* Fixing racy WorkerSketchFetcherTest.java exception handling.
Co-authored-by: abhagraw <99210446+abhagraw@users.noreply.github.com>
Co-authored-by: Karan Kumar <cryptoe@karans-mbp.lan>
2023-01-11 07:38:29 +05:30
365474ff1d
New IT Framework - InputSource and InputFormat Tests ( #13597 )
...
* New IT Framework - InputSource and InputFormat Tests
* Fixing checkstyle errors
* Updating InputSource setup
* Updating queries to use druid DB
* Making metadata setup queries to be idempotent
* Restore intellij files
2023-01-04 10:40:05 +05:30
2f3b97194f
Fix harcoded version in pom file ( #13460 )
2022-12-01 10:10:04 +05:30
7cf761cee4
Prepare master branch for next release, 26.0.0 ( #13401 )
...
* Prepare master branch for next release, 26.0.0
* Use docker image for druid 24.0.1
* Fix version in druid-it-cases pom.xml
2022-11-22 15:31:01 +05:30
280a0f7158
Add sequential sketch merging to MSQ ( #13205 )
...
* Add sketch fetching framework
* Refactor code to support sequential merge
* Update worker sketch fetcher
* Refactor sketch fetcher
* Refactor sketch fetcher
* Add context parameter and threshold to trigger sequential merge
* Fix test
* Add integration test for non sequential merge
* Address review comments
* Address review comments
* Address review comments
* Resolve maxRetainedBytes
* Add new classes
* Renamed key statistics information class
* Rename fetchStatisticsSnapshotForTimeChunk function
* Address review comments
* Address review comments
* Update documentation and add comments
* Resolve build issues
* Resolve build issues
* Change worker APIs to async
* Address review comments
* Resolve build issues
* Add null time check
* Update integration tests
* Address review comments
* Add log messages and comments
* Resolve build issues
* Add unit tests
* Add unit tests
* Fix timing issue in tests
2022-11-22 09:56:32 +05:30
5172d76a67
Migrate current integration batch tests to equivalent MSQ tests ( #13374 )
...
* Migrate current integration batch tests to equivalent MSQ tests using new IT framework
* Fix build issues
* Trigger Build
* Adding more tests and addressing comments
* fixBuildIssues
* fix dependency issues
* Parameterized the test and addressed comments
* Addressing comments
* fixing checkstyle errors
* Adressing comments
2022-11-21 09:12:02 +05:30
81d005f267
Druid Catalog basics ( #13165 )
...
Druid catalog basics
Catalog object model for tables, columns
Druid metadata DB storage (as an extension)
REST API to update the catalog (as an extension)
Integration tests
Model only: no planner integration yet
2022-11-12 15:30:22 -08:00
728745a1d3
Add IT for MSQ task engine using the new IT framework ( #12992 )
...
* first test, serde causing problems
* serde working
* insert and select check
* Add cluster annotations for MSQ test cases
* Add cluster config for MSQ
* Add MSQ config to the pom.xml
* cleanup unnecessary changes
* Remove model classes
* Comments, checkstyle, check queries from file
* fixup test case name
* build failure fix
* review changes
* build failure fix
* Trigger Build
* Log the mismatch in QueryResultsVerifier
* Trigger Build
* Change the signature of the results verifier
* review changes
* LGTM fix
* build, change pom
* Trigger Build
* Trigger Build
* trigger build with minimal pom changes
* guice fix in tests
* travis.yml
2022-09-22 16:09:47 +05:30
7d332c6f6a
Suppress false CVEs ( #13026 )
...
* Suppress CVEs
* Add more suppressions
2022-09-06 11:46:56 +05:30
618757352b
Bump up the version to 25.0.0 ( #12975 )
...
* Bump up the version to 25.0.0
* Fix the version in console
2022-08-29 11:27:38 +05:30
21b73bde20
Update Curator to 5.3.0 ( #12939 )
...
* Update Curator to 5.3.0
* Update licenses.yaml
* Fix inspections + add tests.
* Fix checkstyle
* Another intellij inspection fix
* Update curator exclusions
* Cleanup new exhibitor references
* Remove unused dep and checkstyle fix
2022-08-26 18:23:40 -07:00
cfed036091
Add the new integration test framework ( #12368 )
...
This commit is a first draft of the revised integration test framework which provides:
- A new directory, integration-tests-ex that holds the new integration test structure. (For now, the existing integration-tests is left unchanged.)
- Maven module druid-it-tools to hold code placed into the Docker image.
- Maven module druid-it-image to build the Druid-only test image from the tarball produced in distribution. (Dependencies live in their "official" image.)
- Maven module druid-it-cases that holds the revised tests and the framework itself. The framework includes file-based test configuration, test-specific clients, test initialization and updated versions of some of the common test support classes.
The integration test setup is primarily a huge mass of details. This approach refactors many of those details: from how the image is built and configured to how the Docker Compose scripts are structured to test configuration. An extensive set of "readme" files explains those details. Rather than repeat that material here, please consult those files for explanations.
2022-08-24 17:03:23 +05:30
abhagraw
Karan Kumar
Adarsh Sanjeev
Kashif Faraz
Paul Rogers
Laksh Singla
Abhishek Agarwal
Adam Peck