All JDK 8 based CI checks have been removed.
Images used in Dockerfile(s) have been updated to Java 17 based images.
Documentation has been updated accordingly.
* specify node type so that the log filename can get resolved
* Update distribution/docker/druid.sh
Co-authored-by: Benedict Jin <asdf2014@apache.org>
---------
Co-authored-by: Benedict Jin <asdf2014@apache.org>
This PR creates symlinks when there are duplicate jars present in the extension. Docker image includes contrib extensions, too, and the size of the image has bloated up quite a lot of late. This change also fixes "ITNestedQueryPushDownTest integration test"
This change is meant to fix a issue where passing too large of a task payload to the mm-less task runner will cause the peon to fail to startup because the payload is passed (compressed) as a environment variable (TASK_JSON). In linux systems the limit for a environment variable is commonly 128KB, for windows systems less than this. Setting a env variable longer than this results in a bunch of "Argument list too long" errors.
* Claim full support for Java 17.
No production code has changed, except the startup scripts.
Changes:
1) Allow Java 17 without DRUID_SKIP_JAVA_CHECK.
2) Include the full list of opens and exports on both Java 11 and 17.
3) Document that Java 17 is both supported and preferred.
4) Switch some tests from Java 11 to 17 to get better coverage on the
preferred version.
* Doc update.
* Update errorprone.
* Update docker_build_containers.sh.
* Update errorprone in licenses.yaml.
* Add some more run-javas.
* Additional run-javas.
* Update errorprone.
* Suppress new errorprone error.
* Add exports and opens in ForkingTaskRunner for Java 11+.
Test, doc changes.
* Additional errorprone updates.
* Update for errorprone.
* Restore old fomatting in LdapCredentialsValidator.
* Copy bin/ too.
* Fix Java 15, 17 build line in docker_build_containers.sh.
* Update busybox image.
* One more java command.
* Fix interpolation.
* IT commandline refinements.
* Switch to busybox 1.34.1-glibc.
* POM adjustments, build and test one IT on 17.
* Additional debugging.
* Fix silly thing.
* Adjust command line.
* Add exports and opens one more place.
* Additional harmonization of strong encapsulation parameters.
You can now add additional configuration files to be copied to the final conf directory on startup when running in a containerized environment. Useful for running on Kubernetes and needing to add more files with a config map. To specify the path where the configMap is mounted, utilize the DRUID_ADDITIONAL_CONF_DIR environment variable.
* Support for middle manager less druid, tasks launch as k8s jobs
* Fixing forking task runner test
* Test cleanup, dependency cleanup, intellij inspections cleanup
* Changes per PR review
Add configuration option to disable http/https proxy for the k8s client
Update the docs to provide more detail about sidecar support
* Removing un-needed log lines
* Small changes per PR review
* Upon task completion we callback to the overlord to update the status / locaiton, for slower k8s clusters, this reduces locking time significantly
* Merge conflict fix
* Fixing tests and docs
* update tiny-cluster.yaml
changed `enableTaskLevelLogPush` to `encapsulatedTask`
* Apply suggestions from code review
Co-authored-by: Abhishek Agarwal <1477457+abhishekagarwal87@users.noreply.github.com>
* Minor changes per PR request
* Cleanup, adding test to AbstractTask
* Add comment in peon.sh
* Bumping code coverage
* More tests to make code coverage happy
* Doh a duplicate dependnecy
* Integration test setup is weird for k8s, will do this in a different PR
* Reverting back all integration test changes, will do in anotbher PR
* use StringUtils.base64 instead of Base64
* Jdk is nasty, if i compress in jdk 11 in jdk 17 the decompressed result is different
Co-authored-by: Rahul Gidwani <r_gidwani@apple.com>
Co-authored-by: Abhishek Agarwal <1477457+abhishekagarwal87@users.noreply.github.com>
This commit contains the cleanup needed for the new integration test framework.
Changes:
- Fix log lines, misspellings, docs, etc.
- Allow the use of some of Druid's "JSON config" objects in tests
- Fix minor bug in `BaseNodeRoleWatcher`
There aren't any changes in this patch that improve Java 11
compatibility; these changes have already been done separately. This
patch merely updates documentation and explicit Java version checks.
The log message adjustments in DruidProcessingConfig are there to make
things a little nicer when running in Java 11, where we can't measure
direct memory _directly_, and so we may auto-size processing buffers
incorrectly.
* Change ordering of config file vs env vars in Docker
Currently if you provide a config file it negates any settings set via environment variables.
This change allows use of a config file as a base and allow environment variables to override.
Additionally this allows dynamic features such as DRUID_SET_HOST to function correctly when a config file has been provided.
* Custom JAVA_OPTS should override service jvm.config
* support using mariadb connector with mysql extensions
* cleanup and more tests
* fix test
* javadocs, more tests, etc
* style and more test
* more test more better
* missing pom
* more pom
* Create /opt/data to fix permission problem
* eliminate symlink to avoid compatibility problem on AWS Fargate
* Add a workaround section
* Update instruction for named volume
* Use named volume in docker-compose
* Revert some doc change
* Resolve review comments
* Consolidate the number of Dockerfiles
* add build-arguments to choose which Java base image to use at runtime
* default to building image with Java 11
* base k8s integration test image off of the default image: this ensures
our docker image now gets tested as part of integration tests.
* upgrade maven help plugin to 3.2.0
Docker volume directory was accidentally removed due to reordering of statements.
This causes ownership and permissions on the volume directory to be reset, preventing startup.
fixes#11166
Signed-off-by: frank chen <frank.chen021@outlook.com>
* do build and stop action in IT
* change base dir from druidHome to druidHome/integration-tests
* add env DRUID_HOME
* bug fix
* modify stop_sh
* ready to test
* bug fix
* modify dir
* tested on dev
* modify dir
* move DRUID_HOME env
* done
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
* add a travls job to do integrate test on K8s
* revert build_run_cluster.sh
* revert msic
* run IT test
* ready to test
* modify before/after script
* done
* change mod for script
* done
* add env DRUID_OPERATOR_VERSION=0.0.3
* change version
Co-authored-by: yuezhang <yuezhang@freewheel.tv>
Currently `getConfPath` function returns an empty string for `indexer` service. So the config directory path for this service is not set properly when installed using docker environment.
Previously, when the Docker entrypoint script generated the config
files, it would append the configuration without including a newline.
This could result in incorrect configuration. This has been fixed by
always appending a newline before any configuration.
Co-authored-by: Bryson Chen <brysonjackychen@gmail.com>
Co-authored-by: Bryson Chen <brysonjackychen@gmail.com>
Akshat Jain
Kashif Faraz
aho135
Sensor
Alberic Liu
Charles Smith
Adarsh Sanjeev
Zoltan Haindrich
Abhishek Agarwal
George Shiqi Wu
Karan Kumar
Laksh Singla
George Shiqi Wu
AmatyaAvadhanula
Gian Merlino
Rishabh Singh
Clint Wylie
Benedict Jin
EylonLevy
Varachit W
Dr. Sizzles
Frank Chen
Atul Mohan
wiquan
Rui Chen
Paul Rogers
Gian Merlino
Joseph Glanville
Xavier Léauté
kazuhirokomoda
zhangyue19921010
Jihoon Son
Harini Rajendran
Valdemar
David Palmer
Himanshu