Uses a custom continusou jfr profiler.
Modifies the github actions for tests to do profiling only in the case
of jdk17, as the profiler requires jdk17+ to use the JFR streaming API
plus a few other language features in the code.
Continuous Profiling service is provided to the Apache Druid project
free of charge by Imply and any committer can request free access to
the UI.
* Claim full support for Java 17.
No production code has changed, except the startup scripts.
Changes:
1) Allow Java 17 without DRUID_SKIP_JAVA_CHECK.
2) Include the full list of opens and exports on both Java 11 and 17.
3) Document that Java 17 is both supported and preferred.
4) Switch some tests from Java 11 to 17 to get better coverage on the
preferred version.
* Doc update.
* Update errorprone.
* Update docker_build_containers.sh.
* Update errorprone in licenses.yaml.
* Add some more run-javas.
* Additional run-javas.
* Update errorprone.
* Suppress new errorprone error.
* Add exports and opens in ForkingTaskRunner for Java 11+.
Test, doc changes.
* Additional errorprone updates.
* Update for errorprone.
* Restore old fomatting in LdapCredentialsValidator.
* Copy bin/ too.
* Fix Java 15, 17 build line in docker_build_containers.sh.
* Update busybox image.
* One more java command.
* Fix interpolation.
* IT commandline refinements.
* Switch to busybox 1.34.1-glibc.
* POM adjustments, build and test one IT on 17.
* Additional debugging.
* Fix silly thing.
* Adjust command line.
* Add exports and opens one more place.
* Additional harmonization of strong encapsulation parameters.
Currently, IntelliJ-inspections are run sequentially w.r.t static-checks, thereby increasing build time. Moving IntelliJ-inspections to a separate job to improve builds time and get a quick insight into such issues early on.
Our CI system has a lot of tests. And much of this testing is really unnecessary for most of the PRs. This PR adds some checks so we can skip these expensive tests when we know they are not necessary.
* Fix read timed out failures and remove containers before test
* remove containers before loading images
* add labels to IT docker containers, download stable minio docker image release instead of latest
This PR changes how we skip java UT and ITs with changes in the examples folder. After this change, any Markdown files within the examples folder and jupyter-notebooks directory will be excluded. The rationale behind these more specific exclusions is that some ITs use json files checked in examples, so we want to trigger the full workflow for all other changes.
Currently, there is no way to run ITs if unit-tests fail on coverage. This PR allows Revised, Standard ITs to run even when unit-tests fail on coverage errors, still failing the workflow. This PR also fixes existing GHA workflow naming.
Instead of downloading jdk everytime we run CI, we're using inbuilt temurin jdk distributions 8, 11, 17 by settiing JAVA_HOME variable. This is not working as expected since we were not setting this as global environment variable as a result all CI builds are running on jdk11. This PR fixes the issue.
Hadoop 2 often causes red security scans on Druid distribution because of the dependencies it brings. We want to move away from Hadoop 2 and provide Hadoop 3 distribution available. Switch druid to building with Hadoop 3 by default. Druid will still be compatible with Hadoop 2 and users can build hadoop-2 compatible distribution using hadoop2 profile.
*
1. Handling deletion/creation of container created during the previously run test in AzureTestUtil.java.
2. Adding/updating log messages and comments in Azure and GCS deep storage tests.
* Bump CycloneDX module to address POM errors
* Including web-console in the PR
---------
Co-authored-by: Elliott Freis <elliottfreis@Elliott-Freis.earth.dynamic.blacklight.net>
The FiniteFirehoseFactory and InputRowParser classes were deprecated in 0.17.0 (#8823) in favor of InputSource & InputFormat. This PR removes the FiniteFirehoseFactory and all its implementations along with classes solely used by them like Fetcher (Used by PrefetchableTextFilesFirehoseFactory). Refactors classes including tests using FiniteFirehoseFactory to use InputSource instead.
Removing InputRowParser may not be as trivial as many classes that aren't deprecated depends on it (with no alternatives), like EventReceiverFirehoseFactory. Hence FirehoseFactory, EventReceiverFirehoseFactory, and Firehose are marked deprecated.
Create the docker image in case of maven dependencies cache restore failure too as env.sh file is removed on maven rebuild.
Increase java heap size for security IT failing with error
* merge druid-core, extendedset, and druid-hll into druid-processing to simplify everything
* fix poms and license stuff
* mockito is evil
* allow reset of JvmUtils RuntimeInfo if tests used static injection to override
* Don't run UTs, ITs for docs, helm and console changes
* Test
* right place
* Revert test
* Remove ITs from ignore list
* Update unit-and-integration-tests-unified.yml
* Update unit-and-integration-tests-unified.yml
This adds an autolabeler GitHub action that will label any changes to the docs directories (docs and the Jupyter nb directory) as Area - Documentation.
The GHA will also remove the label if the PR changes and no longer touches files in those directories.
* rebuild maven project or docker image in case of cache hit miss
* rebuild maven project in case of docker cache hit miss too
* fix docker-restore cache hit fail issue
* migrate UTs form Travis to GHA
* update permissions
* rename file
* set fetch depth to 1
* debugs remote branches
* test with github.ref variable
* fetch github.base_ref for diff
* nit
* test git diff
* run tests
* test code coverage failure scenario
* nit
* nit
* revert code changes
* revert code changes
* Setup diff-test-coverage before tests
* build distribution module at end in packaging check
* nit
* remove redundant steps in static-checks workflow
* drop jdk8 unit tests from Travis
* update static-checks GHA to run sequentially
remove static-checks from travis.yml
move docs, web-console, packaging checks from travis to GHA
* nit
* nit
* groups all checks, runs on 8, 11, 17 jdks
* nit
* adds license info
* update permissions on scripts folder
* nit
* nit
* fix packaging check
* changes naming, cleans repo before license checks
* simulate failure
* bump up license checks
* test license checks failure
* test license checks failure
* test license checks failure
* verify gha script run exit code
* fail fast in case of shell script
* verified fail fast in case of shell script
* process: update PR template to include release notes
* Update .github/pull_request_template.md [ci skip]
Co-authored-by: Charles Smith <techdocsmith@gmail.com>
* Update .github/pull_request_template.md
Co-authored-by: Clint Wylie <cjwylie@gmail.com>
* incorporate feedback from paul
Co-authored-by: Charles Smith <techdocsmith@gmail.com>
Co-authored-by: Clint Wylie <cjwylie@gmail.com>
* Add more exempt tags for stalebot; Make stalebot to close PRs/issues in 21 days rather than in 7/14 days
* Exempt labels only for issues, not PRs; Add 'Uncategorized problem report' tag
* Add the pull-request template
* Rewording
* Replaced checklist link, added Rat exclusion
* Update the PR template. Add Concurrency Checklist to the repository
* Merge Description and Design sections. Softer language. Removed requirement to test in production environment. Added a committer's instruction to justify addition of meta tags.
* Rephrase item about comments
* Add license header
* Add item to concurrency checklist
* Stalebot config adjustments.
- Different wordings for PRs and issues.
- Extend stale timeout for issues.
- Exempt "Proposal" label.
- Exempt issues in projects and issues with assignees.
* Fix copy/paste error.
Sam Rash
Tejaswini Bandlamudi
Gian Merlino
Abhishek Agarwal
Clint Wylie
Abhishek Radhakrishnan
AmatyaAvadhanula
abhagraw
somu-imply
Elliott Freis
Kashif Faraz
317brian
Peter Stöckli
Rohan Garg
Suneet Saldanha
Dongjoon Hyun
zachjsh
Xavier Léauté
Jihoon Son
Maytas Monsereenusorn
Jonathan Wei
Benedict Jin
Roman Leventov
Gian Merlino
Fangjin Yang
Dylan Wylie