This upgrade required a few significant changes. Firstly, the build
scan plugin has been renamed, and changed to be a Settings plugin rather
than a project plugin so the declaration of this has moved to our
settings.gradle file. Second, we were using a rather old version of the
Nebula ospackage plugin for building deb and rpm packages, the migration
to the latest version required some updates to get things working as
expected as we had some workarounds in place that are no longer
applicable with the latest bug fixes.
(cherry picked from commit 87f9c16e2f8870e3091062cde37b43042c3ae1c5)
* Move env vars to script, add build scan wrapper
Having the env vars being set up in the build script makes it easier to
reason about.
This PR also adds the gradle scan wrapper back as support for it was
added t o jjbb.
* Add explanatory comment
Backport of #48849. Update `.editorconfig` to make the Java settings the
default for all files, and then apply a 2-space indent to all `*.gradle`
files. Then reformat all the files.
* Move periodic job to ES repo
This change kickstarts the process of moving CI job definitions to this
repo.
* Added a minimal readme to provide pointers to the documentation
* Update .ci/README.md
Co-Authored-By: Rory Hunter <pugnascotia@users.noreply.github.com>
* Update .ci/README.md
Co-Authored-By: Rory Hunter <pugnascotia@users.noreply.github.com>
* point to main repo
* PR review
* Add link to JJBB
Backport of #46599 and #47640. Add packaging tests for Docker.
* Introduce packaging tests for Docker (#46599)
Closes#37617. Add packaging tests for our Docker images, similar to what
we have for RPMs or Debian packages. This works by running a container and
probing it e.g. via `docker exec`. Test can also be run in Vagrant, by
exporting the Docker images to disk and loading them again in VMs. Docker
is installed via `Vagrantfile` in a selection of boxes.
* Only define Docker pkg tests if Docker is available (#47640)
Closes#47639, and unmutes tests that were muted in b958467.
The Docker packaging tests were being defined irrespective of whether
Docker was actually available in the current environment. Instead,
implement exclude lists so that in environments where Docker is not
available, no Docker packaging tests are defined. For CI hosts, the build
checks `.ci/dockerOnLinuxExclusions`. The Vagrant VMs can defined the
extension property `shouldTestDocker` property to opt-in to packaging
tests.
As part of this, define a seperate utility class for checking Docker,
and call that instead of defining checks in-line in BuildPlugin.groovy
* Work around incorrect lintian version on debian 10
Relates to #48573 and intended as a work around to stop the CI failures
until we fix the image.
* Need to pass --allow-downgrades
This PR changes the PS1 script that starts os tests for the packaging
test matrix to match the bash script we use on Linux in terms of reading
the runtime and build java versions.
Relates to elastic/infra#11593
This PR makes the necesary adaptations to the tests and adds a power shell script to
invoke the OS tests on GCP instances connected as CI workers.
Also noticed that logs were not being produced by the tests and that theses were not using log4j so fixed that too.
One of the difficulties in working on theses tests was that the tests just stalled with no indication where the problem is.
To ease with the debugging, after process explorer suggested that the tests are running some commands, we now have multiple timeouts: one for the tests ( which will generate a thread dump ) and one for individual commands ( that bails with the command being ran and output and error so far ) to make it easier to see what went wrong.
The tests were blocking because apparently the pipes to the sub-process were not closing, thus the threads were blocking on them and we were blocking indefinitely on the join. I'm not sure why this doesn't happen in vagrant, but we now properly deal with it.
- Add a vault integration so that we don't need Jenkins to do that for
us
- This will make it easier to enable for windows too
- Move everything to a single file so we can read other secrets in the
same way
This PR adds a list of index compatible versions to the `.ci` directory
as well as a way to generate and verify it.
Unfortunetly there is no easy way in Jenkins to have the build generate then
consume this YML axis config.
I hate that this would need maintenance on versions bumps, but the
potential benefir here is reducing the bwc builds that can take more than
24 hours to less than 20 minutes.
This is possible because the CI setup would use a matrix job to run
something like:
```
./graldew 'v7.0.0#bwcTest'
```
For every index compatible version.
On top of that `--parallel` should be possible even without testclusters
due to the limited number of clusters being set up here.
The example command above runs in exactly 10 minutes on my laptop,
thus I'm proposing to accept this compromise while we work out the
infra to do this more dinamically.
This commit disables building on JDK 13 in CI. The reason for this is
because Gradle is not yet ready to run on JDK 13. We could re-introduce
infrastructure to enable Gralde to run on a different JDK than the build
JDK, but rather than introducing such complexity we will instead wait
for Gradle to be ready to run on JDK 13.
Before this change we would recurse to cache bwc versions.
This proved to be problematic due to the number of steps it was
generating taking too long.
Also this required tricky maintenance to break the recursion for old
branches we don't really care about.
With this change we now cache specific branches only.
We now have coretto on the ephemeral CI images so we can add it to our
regular java test matrix.
On backporting will also add corretto8 to branches that support it.
The script will be use in CI to configure our own Artifactory instance
so CI does not depend on external network resources nor do we generate
excessive load on these resources.
In case the repo is not accessible Gradle should fall back to using the
public ones.
* Make pullFixture a task dependency of resolveAllDependencies
With this change we will pull the docker test fixtures ( and thus cache
the images ) for older versions too as the `resolveAllDependencies` is
already being called on the bwc checkouts too.
The script is used to create a cache on ephemeral CI workers.
Changes:
- create and use a `pullFixture` task that always exists regardless
of docker support
- wire dependencies correctly so any pre fixture setup runs for pull
as well
- set up java env vars so bwc versions can build
Now that JDK 12 has entered rampdown phase one, it is time for us to add
JDK 12 to the CI rotation. This commit adds matrix parameters such that
we try to build with JDK 12, and we test running on JDK 12.
This commit adds Zulu to the CI rotation. This commit only means that we
are going to try running CI against Zulu, it does not carry any
implications of support in the past, now, or in the future.
Switch to parametric CI jobs. Given the changes in CI this also switches to running Gradle with the build java version only ( we used to also run it with run-time java version )
Mark Vieira
jaymode
Alpar Torok
Rory Hunter
Tom Callahan
jimczi
Colin Goodheart-Smithe
Paul Sanwald
Jason Tedor
Jake Landis
Lee Hinman