Commit Graph

375 Commits

Author SHA1 Message Date
jureaky 09d25ef818 [DOCS] Fix typos in build.gradle comments (#52883) 2020-02-27 11:08:37 -05:00
Mark Vieira f06d692706
[Backport] Consolidate docker availability logic (#52656) 2020-02-21 15:24:05 -08:00
Mark Vieira 42610c6d74
Only pull docker images for fixture projects (#52157) 2020-02-10 13:50:31 -08:00
James Rodewig 4ea7297e1e
[DOCS] Change http://elastic.co -> https (#48479) (#51812)
Co-authored-by: Jonathan Budzenski <jon@budzenski.me>
2020-02-03 09:50:11 -05:00
Rory Hunter 1009f92b03
Format projects under :distribution:tools (#51292)
Backport of #51226. Opt-in the sub-projects of :distribution:tools for
automatic formatting.
2020-01-22 11:19:17 +00:00
Jason Tedor 9ce4d2b901
Initial autoscaling commit (#51161)
This commit merely adds the skeleton for the autoscaling project, adding
the basics to include the autoscaling module in the default
distribution, opt-in to code formatting, and a placeholder for the docs.
2020-01-17 15:31:12 -05:00
Rory Hunter 80d925e225
Auto-format buildSrc (#51043)
Backport / reimplementation of #50786 on 7.x.

Opt-in `buildSrc` for automatic formatting. This required a config tweak
in order to pick up all the Java sources, and as a result more files are
now found in the Enrich plugin, that were previously missed.

I also moved the 2 Java files in `buildSrc/src/main/groovy` into the Java
directory, which required some follow-up changes.
2020-01-16 10:26:27 +00:00
Ryan Ernst a0746dbfec Add --data-dir option to run task (#50342)
This commit adds a special run.datadir system property that may be
passed to `./gradlew run` which sets the root data directory used by the
task. While normally overriding the data path is not allowed for test
clusters, it is useful when experimenting with the run task.

closes #50338
2019-12-19 12:48:24 -08:00
Mark Vieira d3cf89b563
Upgrade to Gradle 6.0 (#49211) (#49994)
This upgrade required a few significant changes. Firstly, the build
scan plugin has been renamed, and changed to be a Settings plugin rather
than a project plugin so the declaration of this has moved to our
settings.gradle file. Second, we were using a rather old version of the
Nebula ospackage plugin for building deb and rpm packages, the migration
to the latest version required some updates to get things working as
expected as we had some workarounds in place that are no longer
applicable with the latest bug fixes.

(cherry picked from commit 87f9c16e2f8870e3091062cde37b43042c3ae1c5)
2019-12-09 11:34:35 -08:00
Alpar Torok 6e775cfc97 Provision the correct JDK for test tasks (#48561)
This PR adds build configuration to use the `jdk-download` plugin with
unit tests when no runtime java is configured externally.

It's a first part in a longer chain of changes described in #40531.
2019-11-18 10:28:02 +02:00
Rory Hunter c46a0e8708
Apply 2-space indent to all gradle scripts (#49071)
Backport of #48849. Update `.editorconfig` to make the Java settings the
default for all files, and then apply a 2-space indent to all `*.gradle`
files. Then reformat all the files.
2019-11-14 11:01:23 +00:00
Martijn van Groningen 18d5d73305
Enable spotless for enrich gradle project in 7 dot x branch. (#48976)
Backport of #48908

The enrich project doesn't have much history as all the other gradle projects,
so it makes sense to enable spotless for this gradle project.
2019-11-12 13:22:34 +01:00
Rory Hunter 014e1b1090
Improve resiliency to auto-formatting in server (#48940)
Backport of #48450.

Make a number of changes so that code in the `server` directory is more
resilient to automatic formatting. This covers:

* Reformatting multiline JSON to embed whitespace in the strings
* Move some comments around to they aren't auto-formatted to a strange
  place. This also required moving some `&&` and `||` operators from the
  end-of-line to start-of-line`.
* Add helper method `reformatJson()`, to strip whitespace from a JSON
  document using XContent methods. This is sometimes necessary where
  a test is comparing some machine-generated JSON with an expected
  value.

Also, `HyperLogLogPlusPlus.java` is now excluded from formatting because it
contains large data tables that don't reformat well with the current settings,
and changing the settings would be worse for the rest of the codebase.
2019-11-11 14:33:04 +00:00
Mark Vieira e5c6440a4f
Simplify usage of Gradle Shadow plugin (#48478) (#48597)
This commit simplifies and standardizes our usage of the Gradle Shadow
plugin to conform more to plugin conventions. The custom "bundle" plugin
has been removed as it's not necessary and performs the same function
as the Shadow plugin's default behavior with existing configurations.

Additionally, this removes unnecessary creation of a "nodeps" artifact,
which is unnecessary because by default project dependencies will in
fact use the non-shadowed JAR unless explicitly depending on the
"shadow" configuration.

Finally, we've cleaned up the logic used for unit testing, so we are
now correctly testing against the shadow JAR when the plugin is applied.
This better represents a real-world scenario for consumers and provides
better test coverage for incorrectly declared dependencies.

(cherry picked from commit 3698131109c7e78bdd3a3340707e1c7b4740d310)
2019-10-28 12:11:55 -07:00
Mark Vieira 0e24e6fdd1
Upgrade to build scan plugin 2.4.2 (#48220) 2019-10-18 11:04:52 -07:00
Alpar Torok ca99014d8b Create an upload report once the build completes (#47642)
* Create an upload report once the build completes

We used to have this logic in Jenkins, but that forced us to make it
platform dependent and gave us less control on what to include here.

With this change we create a single archive to be uploaded after the
build completes, and we include a link in the build scan to where we
know this should get uploaded.

* Fix when there's nothign to upload
* Log the directory size
* Switch to ant to walk the project tree
* Collect journlas
* Filter for regular files
* only call journalctl on unix where we have bash
* Grab only logs fro this gradle version
* restrict demon log to relevant one
2019-10-18 08:15:49 +03:00
Alpar Torok 36d018c909 Convert RunTask to use testclusers, remove ClusterFormationTasks (#47572)
* Convert RunTask to use testclusers, remove ClusterFormationTasks

This PR adds a new RunTask and a way for it to start a
testclusters cluster out of band and block on it to replace
the old RunTask that used ClusterFormationTasks.

With this we can now remove ClusterFormationTasks.
2019-10-08 14:43:29 +03:00
Alpar Torok 0a14bb174f Remove eclipse conditionals (#44075)
* Remove eclipse conditionals

We used to have some meta projects with a `-test` prefix because
historically eclipse could not distinguish between test and main
source-sets and could only use a single classpath.
This is no longer the case for the past few Eclipse versions.

This PR adds the necessary configuration to correctly categorize source
folders and libraries.
With this change eclipse can import projects, and the visibility rules
are correct e.x. auto compete doesn't offer classes from test code or
`testCompile` dependencies when editing classes in `main`.

Unfortunately the cyclic dependency detection in Eclipse doesn't seem to
take the difference between test and non test source sets into account,
but since we are checking this in Gradle anyhow, it's safe to set to
`warning` in the settings. Unfortunately there is no setting to ignore
it.

This might cause problems when building since Eclipse will probably not
know the right order to build things in so more wirk might be necesarry.
2019-10-03 11:55:00 +03:00
Rory Hunter 1c7dc3a5bf Format Java source files automatically (#46745)
This commit adds a Java source formatter and checker into the build process.
This is not yet enabled for any sub-projects - to format and check a
sub-project, add its Gradle path into `build.gradle` and run:

    ./gradlew spotlessApply

to format, and:

    ./gradlew spotlessJavaCheck
    # or:
    ./gradlew precommit

to verify formatting.
2019-09-30 10:35:47 +03:00
Ioannis Kakavas 58d2bf7e09
Re-enable BWC tests (#47102)
Re-enable BWC tests now that #46534 has been backported to 7.x
2019-09-25 16:05:42 +03:00
Ioannis Kakavas 23bceaadf8
Handle RelayState in preparing a SAMLAuthN Request (#46534) (#47092)
This change allows for the caller of the `saml/prepare` API to pass
a `relay_state` parameter that will then be part of the redirect
URL in the response as the `RelayState` query parameter.

The SAML IdP is required to reflect back the value of that relay
state when sending a SAML Response. The caller of the APIs can
then, when receiving the SAML Response, read and consume the value
as it see fit.
2019-09-25 13:23:46 +03:00
Ioannis Kakavas 3513eaeed8 Re-enable bwc testing 2019-09-02 16:53:46 +03:00
Ryan Ernst d50d700f14 Don't use assemble task on root project (#45999)
The root project uses the base plugin to get a clean task, but does not
actually need the assemble task. This commit changes the root project to
use the lifecycle-base plugin, which while still creating the assemble
task, won't add any dependencies to it.
2019-08-26 16:35:11 -07:00
Ioannis Kakavas 2bee27dd54
Allow Transport Actions to indicate authN realm (#45946)
This commit allows the Transport Actions for the SSO realms to
indicate the realm that should be used to authenticate the
constructed AuthenticationToken. This is useful in the case that
many authentication realms of the same type have been configured
and where the caller of the API(Kibana or a custom web app) already
know which realm should be used so there is no need to iterate all
the realms of the same type.
The realm parameter is added in the relevant REST APIs as optional
so as not to introduce any breaking change.
2019-08-25 19:36:41 +03:00
Mark Vieira bba7017262
Restore check part 1 and 2 order mistakenly reversed in [#45098] (#45522) 2019-08-13 16:39:00 -07:00
Mark Vieira 250544c4eb
Upgrade to build scan plugin 2.4 (#45426) 2019-08-12 09:10:57 -07:00
Mark Vieira 0ae103c40f
Avoid unnecessary eager creation of Gradle tasks (#45098) (#45310) 2019-08-08 10:50:09 -07:00
Alpar Torok 3a199d044c Add a list of BWC versions for CI (#44418)
This PR adds a list of index compatible versions to the `.ci` directory
as well as a way to generate and verify it.

Unfortunetly there is no easy way in Jenkins to have the build generate then
consume this YML axis config.
I hate that this would need maintenance on versions bumps, but the
potential benefir here is reducing the bwc builds that can take more than
24 hours to less than 20 minutes.
This is possible because the CI setup would use a matrix job to run
something like:
```
./graldew 'v7.0.0#bwcTest'
```
For every index compatible version.
On top of that `--parallel` should be possible even without testclusters
due to the limited number of clusters being set up here.

The example command above runs in exactly 10 minutes on my laptop,
thus I'm proposing to accept this compromise while we work out the
infra to do this more dinamically.
2019-07-17 17:44:49 +03:00
Mark Vieira 54194021bf
Improve build scan metadata (#44247)
(cherry picked from commit 2797b2452b1d3696a4c3a367f21630306da5c818)
2019-07-16 09:37:23 -07:00
Hendrik Muhs 33627ef410 re-enable bwc tests 2019-07-13 08:53:44 +02:00
Hendrik Muhs 684b562381
[7.x][ML-DataFrame] Rewrite continuous logic to prevent terms count limit (#44287)
Rewrites how continuous data frame transforms calculates and handles buckets that require an update. Instead of storing the whole set in memory, it pages through the updates using a 2nd cursor. This lowers memory consumption and prevents problems with limits at query time (max_terms_count). The list of updates can be re-retrieved in a failure case (#43662)
2019-07-13 06:58:04 +02:00
Mark Vieira e44b8b1e2e
[Backport] Remove dependency substitutions 7.x (#42866)
* Remove unnecessary usage of Gradle dependency substitution rules (#42773)

(cherry picked from commit 12d583dbf6f7d44f00aa365e34fc7e937c3c61f7)
2019-06-04 13:50:23 -07:00
Mark Vieira c1816354ed
[Backport] Improve build configuration time (#42674) 2019-05-30 10:29:42 -07:00
Mark Vieira ac8a9515a3
Remove usage of deprecated compare gradle builds plugin (#42687) 2019-05-30 08:42:24 -07:00
Alpar Torok cef4b9ba76 Move the FIPS configuration back to the build plugin (#41989)
* Move the FIPS configuration back to the build plugin

This is necesary for external users of build-tools.
Closes #41721
2019-05-21 16:46:54 +03:00
Jason Tedor d7fd51a84e
Provide names for all artifact repositories (#41857)
This commit adds a name for each Maven and Ivy repository used in the
build.
2019-05-07 06:35:28 -04:00
Ryan Ernst bcd0939b61 Add a rule for task dependencies (#41322)
This commit adds a task rule to print the task dependencies of any task.
It only prints the direct dependencies, but makes debugging missing
dependencies a lot easier.
2019-04-19 10:02:27 -07:00
Alpar Torok e1e2568fa3 Add FIPS specific testclusters configuration (#41199)
ClusterFormationTasks auto configured these properties for clusters.
This PR adds FIPS specific configuration across all test clusters from
the main build script to prevent coupling betwwen testclusters and the
build plugin.

Closes #40904
2019-04-19 10:36:54 +03:00
Mark Vieira 2569fb60de Avoid sharing source directories as it breaks intellij (#40877)
* Avoid sharing source directories as it breaks intellij
* Subprojects share main project output classes directory
* Fix jar hell
* Fix sql security with ssl integ tests
* Relax dependency ordering rule so we don't explode on cycles
2019-04-08 17:26:46 +03:00
Mark Vieira 47a3c42bf2
Upgrade to latest build scan plugin (#40702) 2019-04-03 10:54:07 -07:00
Luca Cavanna 1ec9fb3597
Re-enable bwc tests (#40217)
Relates to #40177 which is now merged and backported to all branches.
2019-03-19 23:27:49 +01:00
Luca Cavanna 4c9d7df887 Disable bwc tests
Relates to #40177
2019-03-19 14:43:39 +01:00
Jason Tedor 2df0405d97
Reenable BWC tests after removing cluster state size (#40127)
This commit reenables the BWC tests after removing cluster state size
and backporting that work.
2019-03-16 18:47:15 -04:00
Jason Tedor 86d1d03c37
Remove cluster state size (#40109)
This commit removes the cluster state size field from the cluster state
response, and drops the backwards compatibility layer added in 6.7.0 to
continue to support this field. As calculation of this field was
expensive and had dubious value, we have elected to remove this field.
2019-03-15 17:16:25 -04:00
Alpar Torok 6c75a2f2b0 Testclusters: start using it for testing some plugins (#39693)
* enable testclusters for some plugins
2019-03-07 17:52:50 +02:00
Alpar Torok 7dcc191aa8 Fix verify versions (#39624)
closes #38708
2019-03-07 17:40:25 +02:00
Armin Braun a9d74711fa
Reenable BwC Tests 7.x (#39650)
* #39550 is merged so we can reenable these
2019-03-04 16:50:50 +01:00
Armin Braun 68bc178017
Disable Bwc Tests (#39551)
* Disable Bwc Tests
* For #39550
2019-03-04 10:41:52 +01:00
Alpar Torok 813351fe26 Un-mute and fix BuildExamplePluginsIT (#38899)
* Un-mute and fix BuildExamplePluginsIT

There doesn't seem to be anything wrong with the test iteself.
I think the failure were CI performance related, but while it was muted,
some failures managed to sneak in.

Closes #38784

* PR review
2019-03-04 08:50:55 +02:00
Alan Woodward 54ced2949b
Re-enable BWC (#39460)
Follow up to #39444
2019-02-27 16:51:15 +00:00