Commit Graph

918 Commits

Author SHA1 Message Date
Yannick Welsch 2738c783e5
Fix Gradle >=4.2 compatibility (#27591)
Gradle 4.2 introduced a feature for safer handling of stale output files. Unfortunately, due to the way some of our tasks are written, this broke execution of our REST tests tasks. The reason for this is that the extract task (which extracts the ES distribution) would clean its output directory, and thereby also remove the empty cwd subdirectory which was created by the clean task. The reason why Gradle would remove the directory is that the earlier running clean task would programmatically create the empty cwd directory, but not make Gradle aware of this fact, which would result in Gradle believing that it could just safely clean the directory.

This commit explicitly defines a task to create the cwd subdirectory, and marks the directory as output of the task, so that the subsequent extract task won't eagerly clean it. It thereby restores full compatibility of the build with Gradle 4.2 and above.

This commit also removes the @Input annotation on the waitCondition closure of AntFixture, which conflicted with the extended input/output validation of Gradle 4.3.
2017-11-30 10:03:16 +01:00
Adrien Grand 996990ad1f
Upgrade to lucene-7.2.0-snapshot-8c94404. (#27496)
The main highlight of this new snapshot is that it introduces the opportunity
for queries to opt out of caching. In case a query opts out of caching, not only
will it never be cached, but also no compound query that wraps it will be
cached.
2017-11-28 14:52:42 +01:00
David Turner e8c2e5d6dd
CURRENT should not be a -SNAPSHOT version if build.snapshot is false (#27512) 2017-11-24 11:19:41 +00:00
David Turner 8916935639
Remove unused method (#27508) 2017-11-24 08:09:31 +00:00
David Turner 89ba8996c6 Consolidate version numbering semantics (#27397)
Fixes to the build system, particularly around BWC testing, and to make future
version bumps less painful.
2017-11-23 20:21:53 +00:00
Martijn van Groningen 9fbbc46ba4
Added 6.2 version 2017-11-22 17:37:19 +01:00
Dimitrios Liappis d3e3bc8656
Tests: Add Fedora-27 to packaging tests
Replace Fedora-25 with Fedora-27 and fix old references in
`TESTING.asciidoc`.

Relates #27434
2017-11-20 16:59:00 +02:00
Yannick Welsch 7b71cdd34e
Fix Gradle 4.3.1 compatibility for logging (#27382)
The build currently does not work with Gradle 4.3.1 as the Gradle team stopped publishing the gradle-logging dependency to jcenter, starting with 4.3.1 (not sure why). There are two options:

- Add the repository managed by Gradle team (https://repo.gradle.org/gradle/libs-releases-local) to our build
- Use an older version (4.3) of the dependency when running with Gradle 4.3.1.

Not to depend on another external repo, I've chosen solution 2. Note that this solution could break on future versions, but as this is a compileOnly dependency, and the interface we use has been super stable since forever, I don't envision this to be an issue (and easily detected by a breaking build).
2017-11-15 14:50:13 +01:00
Yannick Welsch a45f2bd225
Set minimum_master_nodes to all nodes for REST tests (#27344)
PR #26911 set minimum_master_nodes from number_of_nodes to (number_of_nodes / 2) + 1 in our REST tests. This has led to test failures (see #27233) as the REST tests only configure the first node in its unicast.hosts pinging list (see explanation here: #27233 (comment)). Until we have a proper fix for this, I'm reverting the change in #26911.
2017-11-10 15:07:49 +01:00
Jason Tedor 58a28dacbd
Remove colons from task and configuration names
Gradle 5.0 will remove support for colons in configuration and task
names. This commit fixes this for our build by removing all current uses
of colons in configuration and task names.

Relates #27305
2017-11-08 15:22:31 -05:00
Tanguy Leroux 43e7a4a349
Upgrade to Jackson 2.8.10 (#27230)
While it's not possible to upgrade the Jackson dependencies 
to their latest versions yet (see #27032 (comment) for more) 
it's still possible to upgrade to the latest 2.8.x version.
2017-11-06 10:20:05 +01:00
David Pilato 0635778c90
Add support for Gradle 4.3 (#27249)
Closes #26840
Related to #27087
2017-11-03 14:36:59 +01:00
Colin Goodheart-Smithe c1b8140c83
Upgrade to Lucene 7.1 (#27225) 2017-11-02 13:25:33 +00:00
Jason Tedor d5efc30968 Blacklist Gradle 4.2 and above
An upstream Gradle change has broken us starting on version 4.2. This
commit blacklists these versions until we can either find a workaround,
or the upstream issue is addressed.

Relates #27087
2017-10-24 09:11:30 -04:00
Tanguy Leroux 463e7e6fa3 Revert "Upgrade to Jackson 2.9.2 (#27032)"
This reverts commit 0b9acc5ace.
2017-10-20 08:25:41 +02:00
Tanguy Leroux 0b9acc5ace Upgrade to Jackson 2.9.2 (#27032)
Upgrade to Jackson 2.9.2 and also use a boolean `closed` flag to
indicate that a FastStringReader instance is closed, so that length
is still correctly reported after the reader is closed.
2017-10-19 15:15:02 +02:00
Yannick Welsch a4436195f8 Set minimum_master_nodes on rolling-upgrade test (#26911)
The rolling-upgrade test was only writing the "minimum_master_nodes" setting to the configuration file of the old nodes, but not the upgraded ones.

Also changes the value of "minimum_master_nodes" from "number_of_nodes" to "(number_of_nodes / 2) + 1".
2017-10-09 10:45:03 +02:00
Martijn van Groningen dca787ed8a
upgrade to Lucene 7.1.0 snapshot version 2017-10-05 09:06:56 +02:00
Uwe Schindler dd520cf0b6 Upgrade to forbiddenapis 2.4.1 (#26874)
Update provides:
 * Adds support for Gradle 4.0 (without deprecation warning)
 * Full Java 9 support through ASM 6.0

Version 2.4 was buggy (it broke Gradle dependencies on SourceSets), but this version fixes it.
2017-10-04 01:20:23 -06:00
Dimitrios Liappis 5079afd0bb Tests: Add Fedora-26 to packaging tests
Add Fedora-26 to packaging tests. Don't retire Fedora-25 yet, as it
will be EOL, one month after Fedora-27 is out.

Relates #26726
2017-10-03 12:01:20 +03:00
Jason Tedor 2e63a13c0a Upgrade to Log4j 2.9.1
This commit upgrades the Log4j dependency, picking up a fix for an issue
with handling stack traces on JDK 9.

Relates #26750
2017-09-22 11:57:06 -04:00
Jason Tedor e0db89bc35 Upgrade to Lucene 7.0.0
This commit upgrades to the GA release of Luence 7!

Relates #26744
2017-09-21 19:19:33 -04:00
David Pilato b6c6effa2a Move all repository-azure classes under one single package (#26624)
As we did for S3, we can collapse all packages within one single `org.elasticsearch.repositories.azure` package name.

Follow up for https://github.com/elastic/elasticsearch/pull/23518#issuecomment-328903585
2017-09-14 11:56:02 +02:00
Adrien Grand 93da7720ff Move non-core mappers to a module. (#26549)
Today we have all non-plugin mappers in core. I'd like to start moving those
that neither map to json datatypes nor are very frequently used like `date` or
`ip` to a module.

This commit creates a new module called `mappers-extra` and moves the
`scaled_float` and `token_count` mappers to it. I'd like to eventually move
`range` fields there but it's more complicated due to their intimate
relationship with range queries.

Relates #10368
2017-09-13 17:58:53 +02:00
Jason Tedor 670849894f Ensure module is bundled before installing in tests
This commit adds a dependency to the install module task on the task
that builds the module. This is needed for standalone integration
tests that require other modules to be installed. Without this, we do
not have a guarantee that the module is bundled.
2017-09-13 08:17:16 -04:00
Ryan Ernst 95d40758c2 Build: Remove norelease from forbidden patterns (#26592)
closes #26547
2017-09-11 15:55:02 -07:00
Ryan Ernst 8ba4ff3be0 Build: Move javadoc linking to root build.gradle (#26529)
Javadoc linking between projects currently relies on
projectSubstitutions. However, that is an extension variable that is not
part of BuildPlugin. This commit moves the javadoc linking into the root
build.gradle, alongside where projectSubstitutions are defined.
2017-09-11 15:43:34 -07:00
Jason Tedor e16cb0e4db Allow multiple digits in Vagrant 2.x minor versions
This commit allows the minor version in Vagrant 2.x versions to have two
digits when we perform the Vagrant version check.
2017-09-09 15:28:26 -04:00
Kevin Risden f476c10825 Support Vagrant 2.x
This commit adds support for Vagrant 2.x by allowing these versions to
pass the Vagrant version check.

Relates #26563
2017-09-09 15:27:21 -04:00
Jason Tedor 19a2156d18 Skip some logging tests on JDK 9
There is a bug in Log4j on JDK 9 for walking the stack to find where a
log line is coming from. This bug is impacting some of our testing, so
this commit marks these tests as skippable only on JDK 9 until the bug
is fixed upstream.

Relates #26467
2017-09-01 12:38:22 -04:00
Alexander Reelsen 563ff3cbce Tests: Use script.max_compilations_rate for BWC tests
As the setting is now backported to 6.0, the cluster
formation task can use only the new setting, no need
to use the old one anymore.
2017-09-01 14:43:00 +02:00
Alexander Reelsen 80d0a32f8e ScriptService: Replace max compilation per minute setting with max compilation rate (#26399)
The current script service has a script compilation limit for a one
minute window. This is set to a small default value of 15. Instead of
increasing that default value, this commit introduces a new setting 
that allows to configure a rate per time unit, so that the script service can deal with bursts better.

The new setting is named `script.max_compilations_rate`,
requires a nonnegative number and a positive time value.

The default is `75/5m`, which is equivalent to the existing 15 per minute.
2017-09-01 10:15:27 +02:00
Jason Tedor 697bc266ce Upgrade to Log4j 2.9.0
This commit upgrades the Log4j dependency from version 2.8.2 to version
2.9.0.

Relates #26450
2017-08-31 09:54:35 -04:00
Adrien Grand 78681bc9e5 Upgrade to lucene-7.0.0-snapshot-d94a5f0. (#26441) 2017-08-31 09:06:40 +02:00
Jason Tedor e8a7b1b8e5 Remove dead path conf BWC code in build
When changing how the config path is configured (from a command-line
flag to an environment variable) we had to add BWC code in the build so
that we could form clusters with 5.x nodes in them. Now that this branch
has moved to 7.x, we no longer need to be BWC with 5.x for starting
nodes. This commit removes this dead BWC code.

Relates #26446
2017-08-30 13:54:29 -04:00
Jason Tedor 53b7ad31e7 Use short name for invoking wrapper script
When starting a node in standalone tests, we sometimes use a wrapper
script as opposed to starting Elasticsearch directly (this is used for
backgrounding). On Windows, the path to this wrapper script can be
exceptionally long, exceeding the length of a batch script that cmd.exe
will invoke without whining. This commit replaces using the full path
name for this wrapper script by the short name for the wrapper script.

Additionally, the data, configuration, and jvm.options paths can also
end up being too long so we shorten those too. Care has to be taken with
the data directory because we usually rely on the node creating it on
startup but doing that will not be compatible with getting the short
name as that requires the directory already existing. Therefore, we
create that directory on-demand immediately before actually resolving
the short name.

Relates #26444
2017-08-30 13:13:03 -04:00
Tanguy Leroux f95dec797d [Docs] Convert more doc snippets (#26359)
This commit converts some remaining doc snippets so that they are now
testable.
2017-08-28 11:23:09 +02:00
Michael Basnight cfd14cd2b8 Revert shading for the low level rest client (#26367)
At current, we do not feel there is enough of a reason to shade the low
level rest client. It caused problems with commons logging and IDE's
during the brief time it was used. We did not know exactly how many
users will need this, and decided that leaving shading out until we
gather more information is best. Users can still shade the jar
themselves. For information and feeback, see issue #26366.

Closes #26328

This reverts commit 3a20922046.
This reverts commit 2c271f0f22.
This reverts commit 9d10dbea39.
This reverts commit e816ef89a2.
2017-08-25 14:13:12 -05:00
Ryan Ernst 661648b3aa Build: Allow build to configure which license/notice to embed in jars (#26373)
We currently add the apache license/notice for elasticsearch to any
plugin that uses our ES plugin gradle plugin. However, each plugin
should be able to use their own license. This commit adds a licenseFile
and noticeFile property to the root of project's using BuildPlugin,
which is added to jar files for that project.
2017-08-24 22:46:30 -07:00
Jason Tedor 911e1f6203 Handle long paths on Windows for standalone tests
In some cases our Windows builds fail due to long path names that arise
from a combination of long build job names plus long sub-project
names. While newer versions of Windows can handle long paths, invoking
batch scripts longer than 260 characters via cmd.exe is still
problematic. This leads to failing integration tests because we can not
run the commands to install plugins, create the keystore, and start the
node. This commit handles this by converting all paths on Windows used
to start an Elasticsearch node to short path names.

Relates #26365
2017-08-24 18:46:49 -04:00
Ryan Ernst 5202e7e93b Settings: Move keystore creation to plugin installation (#26329)
This commit removes the keystore creation on elasticsearch startup, and
instead adds a plugin property which indicates the plugin needs the
keystore to exist. It does still make sure the keystore.seed exists on
ES startup, but through an "upgrade" method that loading the keystore in
Bootstrap calls.

closes #26309
2017-08-24 12:12:47 -07:00
Yannick Welsch cfff71aa64 Use Javadoc tool from JAVA_HOME (#26330)
We currently run the javadoc executable from the JDK that runs Gradle, not the JDK from JAVA_HOME.

Relates to #24733
2017-08-23 10:18:27 +09:30
Yannick Welsch 3d8feff66e Use Java 9 FilePermission model (#26302)
This commit makes the security code aware of the Java 9 FilePermission changes (see #21534) and allows us to remove the `jdk.io.permissionsUseCanonicalPath` system property.
2017-08-22 11:22:00 +09:30
Yannick Welsch 41f81e2279 Use explicit test classes output dir for configuring test task
This allows the build to run with Gradle 4.1 as well, where oldTestTask.testClassesDir will return null
at configuration time.
2017-08-21 14:55:23 +09:30
Luca Cavanna 1309dfd44d Add links to external classes in clients javadoc (#25998)
The client sniffer depends on the low-level REST client, while the Java high-level REST client and the transport client depend on Elasticsearch itself. Javadoc are not that useful unless they have links to the Elasticsearch classes in the latter case, and to the low-level REST client in the sniffer javadoc. This commit adds those links.
2017-08-17 21:03:47 +02:00
Nik Everett 6d2c40e546 Enforce that responses in docs are valid json (#26249)
All of the snippets in our docs marked with `// TESTRESPONSE` are
checked against the response from Elasticsearch but, due to the
way they are implemented they are actually parsed as YAML instead
of JSON. Luckilly, all valid JSON is valid YAML! Unfurtunately
that means that invalid JSON has snuck into the exmples!

This adds a step during the build to parse them as JSON and fail
the build if they don't parse.

But no! It isn't quite that simple. The displayed text of some of
these responses looks like:
```
{
    ...
    "aggregations": {
        "range": {
            "buckets": [
                {
                    "to": 1.4436576E12,
                    "to_as_string": "10-2015",
                    "doc_count": 7,
                    "key": "*-10-2015"
                },
                {
                    "from": 1.4436576E12,
                    "from_as_string": "10-2015",
                    "doc_count": 0,
                    "key": "10-2015-*"
                }
            ]
        }
    }
}
```

Note the `...` which isn't valid json but we like it anyway and want
it in the output. We use substitution rules to convert the `...`
into the response we expect. That yields a response that looks like:
```
{
    "took": $body.took,"timed_out": false,"_shards": $body._shards,"hits": $body.hits,
    "aggregations": {
        "range": {
            "buckets": [
                {
                    "to": 1.4436576E12,
                    "to_as_string": "10-2015",
                    "doc_count": 7,
                    "key": "*-10-2015"
                },
                {
                    "from": 1.4436576E12,
                    "from_as_string": "10-2015",
                    "doc_count": 0,
                    "key": "10-2015-*"
                }
            ]
        }
    }
}
```

That is what the tests consume but it isn't valid JSON! Oh no! We don't
want to go update all the substitution rules because that'd be huge and,
ultimately, wouldn't buy much. So we quote the `$body.took` bits before
parsing the JSON.

Note the responses that we use for the `_cat` APIs are all converted into
regexes and there is no expectation that they are valid JSON.

Closes #26233
2017-08-17 09:02:10 -04:00
Simon Willnauer d26c8b5c88 Evaluate settings values before passing it to task closure (#26243)
The secure settings tool reads from stdIn and we use a closure to
provide a value for this. Yet, we evaluate they value too late and end up
with the last provided value for all keys.
2017-08-16 22:31:49 +02:00
Jason Tedor e9687622bd Rename CONF_DIR to ES_PATH_CONF
The environment variable CONF_DIR was previously inconsistently used in
our packaging to customize the location of Elasticsearch configuration
files. The importance of this environment variable has increased
starting in 6.0.0 as it's now used consistently to ensure Elasticsearch
and all secondary scripts (e.g., elasticsearch-keystore) all use the
same configuration. The name CONF_DIR is there for legacy reasons yet
it's too generic. This commit renames CONF_DIR to ES_PATH_CONF.

Relates #26197
2017-08-15 06:19:06 +09:00
Nik Everett 99ac7beb8e Teach the build about betas and rcs (#26066)
The build was ignoring suffixes like "beta1" and "rc1" on the version numbers which was causing the backwards compatibility packaging tests to fail because they expected to be upgrading from 6.0.0 even though they were actually upgrading from 6.0.0-beta1. This adds the suffixes to the information that the build scrapes from Version.java. It then uses those suffixes when it resolves artifacts build from the bwc branch and for testing.

Closes #26017
2017-08-10 14:30:00 -04:00
Adrien Grand f0c1e30544 Upgrade to lucene-7.0.0-snapshot-a128fcb. (#26090) 2017-08-08 13:03:19 +02:00
Ryan Ernst 10cf5a8d3e Build: Fork javac to avoid GC overhead (#26047)
Compiling all of elasticsearch classes in one jvm, which is shared with
all of the loaded classes of gradle, can trip gc overhead limits. This
commit re-enables forking javac.
2017-08-03 15:47:29 -04:00
Ryan Ernst 072281d5aa Update version to 7.0.0-alpha1 (#25876)
This commit updates the version for master to 7.0.0-alpha1. It also adds
the 6.1 version constant, and fixes many tests, as well as marking some
as awaits fix.

Closes #25893
Closes #25870
2017-08-01 15:47:48 -04:00
Jason Tedor 6d68e8a10c Increase max compilation limit in standalone tests
Some REST tests can rapid-fire script compilations that exceed the
default script compilations per minute. Rather than subjecting ourselves
to spurious failures because of the limit being too low, we opt for a
larger limit here.
2017-08-01 14:49:47 +09:00
Yannick Welsch 1a01514081 Move tribe to a module (#25778)
This commit moves tribe to a module, stripping core from the tribe functionality.
2017-07-28 11:23:50 +02:00
Yannick Welsch efd79882a2 Allow build to directly run under JDK 9 (#25859)
With Gradle 4.1 and newer JDK versions, we can finally invoke Gradle directly using a JDK9 JAVA_HOME without requiring a JDK8 to "bootstrap" the build. As the thirdPartyAudit task runs within the JVM that Gradle runs in, it needs to be adapted now to be JDK9 aware.

This commit also changes the `JavaCompile` tasks to only fork if necessary (i.e. when Gradle's JVM and JAVA_HOME's JVM differ).
2017-07-27 16:14:04 +02:00
Simon Willnauer b72c71083c Cleanup IndexFieldData visibility (#25900)
Today we expose `IndexFieldDataService` outside of IndexService to do maintenance
or lookup field data in different ways. Yet, we have a streamlined way to access IndexFieldData
via `QueryShardContext` that should encapsulate all access to it. This also ensures that we control all other functionality like cache clearing etc.

This change also removes the `recycler` option from `ClearIndicesCacheRequest` this option is a no-op and should have been removed long ago.
2017-07-26 20:03:42 +02:00
Martijn van Groningen a9ae52e78b
inner hits: Only access stored fields when needed
Stored fields were still being accessed for nested inner hits even if the _source was not requested.
This was done to figure out the id of the root document. However this is already known higher up the stack.
So instead this change adds the id to the nested search context, so that it is no longer required to be fetched via the stored fields.

In case the _source is large and no source is requested then hot threads like these ones would still appear:

```
100.3% (501.3ms out of 500ms) cpu usage by thread 'elasticsearch[AfXKKfq][search][T#6]'
     2/10 snapshots sharing following 22 elements
       org.apache.lucene.store.DataInput.skipBytes(DataInput.java:352)
       org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.skipField(CompressingStoredFieldsReader.java:246)
       org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:601)
       org.apache.lucene.index.CodecReader.document(CodecReader.java:88)
       org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:411)
       org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:347)
       org.elasticsearch.search.fetch.FetchPhase.createNestedSearchHit(FetchPhase.java:219)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:150)
       org.elasticsearch.search.fetch.subphase.InnerHitsFetchSubPhase.hitsExecute(InnerHitsFetchSubPhase.java:73)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:166)
       org.elasticsearch.search.fetch.subphase.InnerHitsFetchSubPhase.hitsExecute(InnerHitsFetchSubPhase.java:73)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:166)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:422)
```

and:

```
8/10 snapshots sharing following 27 elements
       org.apache.lucene.codecs.compressing.LZ4.decompress(LZ4.java:135)
       org.apache.lucene.codecs.compressing.CompressionMode$4.decompress(CompressionMode.java:138)
       org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$BlockState$1.fillBuffer(CompressingStoredFieldsReader.java:531)
       org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader$BlockState$1.readBytes(CompressingStoredFieldsReader.java:550)
       org.apache.lucene.store.DataInput.readBytes(DataInput.java:87)
       org.apache.lucene.store.DataInput.skipBytes(DataInput.java:350)
       org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.skipField(CompressingStoredFieldsReader.java:246)
       org.apache.lucene.codecs.compressing.CompressingStoredFieldsReader.visitDocument(CompressingStoredFieldsReader.java:601)
       org.apache.lucene.index.CodecReader.document(CodecReader.java:88)
       org.apache.lucene.index.FilterLeafReader.document(FilterLeafReader.java:411)
       org.elasticsearch.search.fetch.FetchPhase.loadStoredFields(FetchPhase.java:347)
       org.elasticsearch.search.fetch.FetchPhase.createNestedSearchHit(FetchPhase.java:219)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:150)
       org.elasticsearch.search.fetch.subphase.InnerHitsFetchSubPhase.hitsExecute(InnerHitsFetchSubPhase.java:73)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:166)
       org.elasticsearch.search.fetch.subphase.InnerHitsFetchSubPhase.hitsExecute(InnerHitsFetchSubPhase.java:73)
       org.elasticsearch.search.fetch.FetchPhase.execute(FetchPhase.java:166)
       org.elasticsearch.search.SearchService.executeFetchPhase(SearchService.java:422)
```
2017-07-25 12:10:59 +02:00
Michael Basnight e816ef89a2 Shade external dependencies in the rest client jar
This commit removes all external dependencies from the rest client jar
and shades them in an 'org.elasticsearch.client' package within the jar
using shadowJar gradle plugin. All projects that depended on the
existing jar have been converted to using the 'org.elasticsearch.client'
package prefixes to interact with the rest client.

Closes #25208
2017-07-24 12:55:43 -05:00
Adrien Grand 91fe8d5366 Enforce that bash is used when running `gradle run`.
Using `sh` means we used whatever default the system has, which is `dash` on
Ubuntu, even though our startup script is written for bash (see the shebang).
2017-07-21 10:46:06 +02:00
Jason Tedor 0310a6a947 Introduce elasticsearch-env
This commit introduces the elasticsearch-env script. The purpose of this
script is threefold:
 - vastly simplify the various scripts used in Elasticsearch
 - provide a script that can be included in other scripts in the
   Elasticsearch ecosystem (e.g., plugins)
 - correctly establish the environment for all scripts (e.g., so that
   users can run `elasticsearch-keystore` from a package distribution
   without having to worry about setting `CONF_DIR` first, otherwise the
   keystore would be created in the wrong location)

Relates #25815
2017-07-21 09:38:49 +09:00
Jason Tedor 5a416b9922 Use config directory to find jvm.options
This commit removes the environment variable ES_JVM_OPTIONS that allows
the jvm.options file to sit separately from the rest of the config
directory. Instead, we use the CONF_DIR environment variable for custom
configuration location just as we do for the other configuration files.

Relates #25679
2017-07-12 15:29:13 -04:00
Jason Tedor e165c405ac Add an underscore to flood stage setting
This is a minor nitty bikeshedding change that renames the suffix of the
disk flood stage setting to "flood_stage" from "floodstage".

Relates #25659
2017-07-11 22:02:00 -04:00
Adrien Grand 481d5d09b2 Upgrade to lucene-7.0.0-snapshot-00142c9. (#25641)
Lucene 7.0 is feature-frozen now, so there should not be many changes until GA.
2017-07-11 13:58:55 +02:00
Jason Tedor c084542731 Bump version to 6.0.0-beta1
This commit does two things:
 - bumps the version from 6.0.0-alpha3 to 6.0.0-beta1
 - renames the 6.0.0-alpha3 version constant to 6.0.0-beta1

Relates #25621
2017-07-09 18:12:50 -04:00
Jason Tedor bc22c1c286 Add disk threshold settings validation
This commit adds cross-settings validation for the low/high/flood stage
disk watermark settings. This validation was enabled by the introduction
of multiple settings validation.

Relates #25600
2017-07-07 19:54:36 -04:00
Martijn van Groningen 6db708ef75
Move more token filters to analysis-common module
The following token filters were moved: common grams, limit token, pattern capture and pattern raplace.

Relates to #23658
2017-07-07 10:02:52 +02:00
Simon Willnauer 1c4ef0d214 Upgrade randomizedrunner to 2.5.2 (#25533)
An issue causing confusing error messages during test execution
has been fixed randomizedtesting/randomizedtesting#250
2017-07-04 16:48:11 +02:00
Dimitrios Liappis 39d42292dd Tests: Add Debian 9 (Stretch) to the packaging tests
Debian 9 aka Stretch is the current stable[1].

Add Debian-9 to the packaging tests.

[1] https://wiki.debian.org/DebianStretch

Relates #25494
2017-07-03 11:02:17 +03:00
Drew Raines 6deb18c0de Preliminary support for ARM
This commit adds preliminary support for 64-bit ARM architectures.

Relates #25318
2017-06-30 14:22:20 -04:00
Tim Brooks 5f8be0e090 Introduce NioTransport into framework for testing (#24262)
This commit introduces a nio based tcp transport into framework for
testing.

Currently Elasticsearch uses a simple blocking tcp transport for
testing purposes (MockTcpTransport). This diverges from production
where our current transport (netty) is non-blocking.

The point of this commit is to introduce a testing variant that more
closely matches the behavior of production instances.
2017-06-28 10:51:20 -05:00
Jason Tedor cca18a2c35 Make plugin loading stricter
Today we load plugins reflectively, looking for constructors that
conform to specific signatures. This commit tightens the reflective
operations here, not allowing plugins to have ambiguous constructors.

Relates #25405
2017-06-26 21:42:53 -04:00
Jason Tedor 22beb8d03c Remove hacky node version check in NodeInfo
This commit removes a hacky way of checking that a node is running
Elasticsearch 5.x when starting standalone nodes in tests.

Relates #25406
2017-06-26 20:32:13 -04:00
Jason Tedor 5a9fc8aa2a Remove path.conf setting
This commit removes path.conf as a valid setting and replaces it with a
command-line flag for specifying a non-default path for configuration.

Relates #25392
2017-06-26 15:18:29 -04:00
Nik Everett 4306315ff6 Throw useful error on bad docs snippets (#25389)
You can continue a test started in a previous snippet by marking the
next snippet with `// TEST[continued]`. The trouble is, if you mark the
first snippet in a file with `// TEST[continued]` you'd get difficult
to predict behavior because you'd continue the test started in another
file. This will usually create a test that fails the build but it isn't
easy to track down what you did wrong. This commit catches this
scenario up front and fails the build with a useful error message.

Similarly, if you put `// TEST[continued]` directly after a
`// TESTSETUP` section then the docs tests will fail to run but the
error message did not point you to the `// TEST[continued]` snippet.
This commit catches this scenario up front as well and fails the build
with a useful error message.
2017-06-26 11:52:16 -04:00
Martijn van Groningen a34f5fa812
Move more token filters to analysis-common module
The following token filters were moved: stemmer, stemmer_override, kstem, dictionary_decompounder, hyphenation_decompounder, reverse, elision and truncate.

Relates to #23658
2017-06-26 09:02:16 +02:00
Adrien Grand 44e9c0b947 Upgrade to lucene-7.0.0-snapshot-ad2cb77. (#25349)
Most notable changes:
 - better update concurrency: LUCENE-7868
 - TopDocs.totalHits is now a long: LUCENE-7872
 - QueryBuilder does not remove the boolean query around multi-term synonyms:
   LUCENE-7878
 - removal of Fields: LUCENE-7500

For the `TopDocs.totalHits` change, this PR relies on the fact that the encoding
of vInts and vLongs are compatible: you can write and read with any of them as
long as the value can be represented by a positive int.
2017-06-22 12:35:33 +02:00
Nik Everett 21b1db2965 Remove assemble from build task when assemble removed
Removes the `assemble` task from the `build` task when we have
removed `assemble` from the project. We removed `assemble` from
projects that aren't published so our releases will be faster. But
That broke CI because CI builds with `gradle precommit build` and,
it turns out, that `build` includes `check` and `assemble`. With
this change CI will only run `check` for projects without an
`assemble`.
2017-06-16 17:19:14 -04:00
Nik Everett 7b358190d6 Remove assemble task when not used for publishing (#25228)
Removes the `assemble` task from projects that are not published.
This should speed up `gradle assemble` by skipping projects that
don't need to be built. Which is useful because `gradle assemble`
is how we cut releases.
2017-06-16 11:46:34 -04:00
Christoph Büscher 39d9c8aa67 Remove some redundant 140 character checkstyle suppressions 2017-06-16 13:58:22 +02:00
Martijn van Groningen 428e70758a
Moved more token filters to analysis-common module.
The following token filters were moved: `edge_ngram`, `ngram`, `uppercase`, `lowercase`, `length`, `flatten_graph` and `unique`.

Relates to #23658
2017-06-15 18:28:31 +02:00
Adrien Grand 0c117145f6 Upgrade to lucene-7.0.0-snapshot-92b1783. (#25222)
This snapshot has faster range queries on range fields (LUCENE-7828), more
accurate norms (LUCENE-7730) and the ability to use fake term frequencies
(LUCENE-7854).
2017-06-15 09:52:07 +02:00
James Baiera 2e29b69f6a Revert "Revert "Sense for VirtualBox and $HOME when deciding to turn on vagrant testing. (#24636)""
This reverts commit b9e2a1f989.
2017-06-12 09:41:35 -04:00
Jim Ferenczi 8250aa4267 Remove the postings highlighter and make unified the default highlighter choice (#25028)
This change removes the `postings` highlighter. This highlighter has been removed from Lucene master (7.x) because it behaves
exactly like the `unified` highlighter when index_options is set to `offsets`:
https://issues.apache.org/jira/browse/LUCENE-7815

It also makes the `unified` highlighter the default choice for highlighting a field (if `type` is not provided).
The strategy used internally by this highlighter remain the same as before, it checks `term_vectors` first, then `postings` and ultimately it re-analyzes the text.
Ultimately it rewrites the docs so that the options that the `unified` highlighter cannot handle are clearly marked as such.
There are few features that the `unified` highlighter is not able to handle which is why the other highlighters (`plain` and `fvh`) are still available.
I'll open separate issues for these features and we'll deprecate the `fvh` and `plain` highlighters when full support for these features have been added to the `unified`.
2017-06-09 14:09:57 +02:00
Jason Tedor 4034cd429d Fix formatting of random testing seed
The whitespace is not aligned correctly and it might be crazy-making to
some developers.
2017-06-07 15:16:46 -04:00
Nik Everett d6cb73b5ef Build: Pin the random seed at startup (#24990)
Pins the random testing seed at build start rather than letting
it vary with every randomized testing invocation. This is useful
for projects where random decisions in one randomized testing run
can effect the outcome of a second randomized testing run such as
the full cluster restart tests.

The goal isn't for tests to be able to assume that random decision
will be the same in both tests. It is more to make sure that the
seed printed when a test fails reproduces the appropriate random
decisions. And pinning the seed at startup should do just that.

This works by taking the key passed as a system property if one
is passed, otherwise picking a random long and getting it into
appropriate key format. The build just calls
`new Random().nextLong()` to get the seed while randomized testing
uses a Murmur3 hash of `System.nanoTime`.
2017-06-07 09:18:43 -04:00
Tim Brooks feca0a9f33 Bumping version to v6.0.0-alpha3 (#25077) 2017-06-06 15:47:23 -05:00
Nik Everett 66007078d4 Build: Allow preserving shared dir (#24962)
This adds an option to `ClusterConfiguration` to preserve the
`shared` directory when starting up a new cluster and switches
the `qa:full-cluster-restart` tests to use it rather than
disable the clean shared task.

Relates to #24846
2017-06-04 20:40:25 -04:00
Lee Hinman b9e2a1f989 Revert "Sense for VirtualBox and $HOME when deciding to turn on vagrant testing. (#24636)"
This reverts commit 4ed0abe72d.
2017-06-02 14:42:52 -06:00
James Baiera 4ed0abe72d Sense for VirtualBox and $HOME when deciding to turn on vagrant testing. (#24636)
We're using Vagrant in more places now than before. This commit includes a plugin that verifies
the Vagrant and Virtualbox installations for projects that depend on them. This shared code
should fix up the errors we've seen from CI builds relating to the new Kerberos fixture.
2017-06-02 16:26:11 -04:00
Nik Everett a301bbbe8b Build: fix version sorting
Prevents us from incorrectly thinking the current version is
the bwc version.
2017-06-01 14:53:08 -04:00
Ryan Ernst 160a049930 Build: Move verifyVersions to new branchConsistency task (#25009)
This commit adds a new `branchConsistency` task which will run in CI
once a day, instead of on every commit. This allows `verifyVersions` to
not break immediately once a new version is released in maven.
2017-06-01 10:29:51 -07:00
Dimitrios Liappis 8f8fc74b2e Tests: Switch to openSUSE 42 (Leap) for packaging tests
openSUSE-13 has reached [EOL](https://en.opensuse.org/Lifetime).

Replace openSUSE-13 with openSUSE-42 (Leap) for packaging tests and
update docs.

Relates #25001
2017-06-01 15:40:47 +03:00
Nik Everett 5da8ce8318 Remove the need for _UNRELEASED suffix in versions (#24798)
Removes the need for the `_UNRELEASED` suffix on versions by detecting if a version should be unreleased or not based on the versions around it. This should make it simpler to automate the task of adding a new version label.
2017-05-26 18:36:32 -04:00
Nik Everett e072cc7770 Begin replacing static index tests with full restart tests (#24846)
These tests spin up two nodes of an older version of Elasticsearch,
create some stuff, shut down the nodes, start the current version,
and verify that the created stuff works.

You can run `gradle qa:full-cluster-restart:check` to run these
tests against the head of the previous branch of Elasticsearch
(5.x for master, 5.4 for 5.x, etc) or you can run
`gradle qa:full-cluster-restart:bwcTest` to run this test against
all "index compatible" versions, one after the other. For master
this is every released version in the 5.x.y version *and* the tip
of the 5.x branch.

I'd love to add more to these tests in the future but these
currently just cover the functionality of the `create_bwc_index.py`
script and start to cover the assertions in the
`OldIndexBackwardsCompatibilityIT` test.
2017-05-26 14:07:48 -04:00
Ryan Ernst 8dbf8a230e Build: Add back explicit exclusions and remove gradle exclusions (#24879)
When transitive dependencies are disable for a dependency, gradle adds a
wildcard exclusion to the generated pom. However, some external tools
like ivy have bugs with wildcards. This commit adds back the explicit
generation of transitive excludes, and removes the gradle generated
exclusions element from the pom.

closes #24490
2017-05-25 12:22:08 -07:00
Nik Everett 13a86fec99 Add magic $_path stash key to docs tests (#24724)
Adds a "magic" key to the yaml testing stash mostly for use with
documentation tests. When unstashing an object, `$_path` is the
path into the current position in the object you are unstashing.
This means that in docs tests you can use
`// TESTRESPONSEs/somevalue/$body.${_path}/` to mean "replace
`somevalue` with whatever is the response in the same position."

Compare how you must carefully mock out all the numbers in the profile
response without this change:
```
// TESTRESPONSE[s/"id": "\[2aE02wS1R8q_QFnYu6vDVQ\]\[twitter\]\[1\]"/"id": $body.profile.shards.0.id/]
// TESTRESPONSE[s/"rewrite_time": 51443/"rewrite_time": $body.profile.shards.0.searches.0.rewrite_time/]
// TESTRESPONSE[s/"score": 51306/"score": $body.profile.shards.0.searches.0.query.0.breakdown.score/]
// TESTRESPONSE[s/"time_in_nanos": "1873811"/"time_in_nanos": $body.profile.shards.0.searches.0.query.0.time_in_nanos/]
// TESTRESPONSE[s/"build_scorer": 2935582/"build_scorer": $body.profile.shards.0.searches.0.query.0.breakdown.build_scorer/]
// TESTRESPONSE[s/"create_weight": 919297/"create_weight": $body.profile.shards.0.searches.0.query.0.breakdown.create_weight/]
// TESTRESPONSE[s/"next_doc": 53876/"next_doc": $body.profile.shards.0.searches.0.query.0.breakdown.next_doc/]
// TESTRESPONSE[s/"time_in_nanos": "391943"/"time_in_nanos": $body.profile.shards.0.searches.0.query.0.children.0.time_in_nanos/]
// TESTRESPONSE[s/"score": 28776/"score": $body.profile.shards.0.searches.0.query.0.children.0.breakdown.score/]
// TESTRESPONSE[s/"build_scorer": 784451/"build_scorer": $body.profile.shards.0.searches.0.query.0.children.0.breakdown.build_scorer/]
// TESTRESPONSE[s/"create_weight": 1669564/"create_weight": $body.profile.shards.0.searches.0.query.0.children.0.breakdown.create_weight/]
// TESTRESPONSE[s/"next_doc": 10111/"next_doc": $body.profile.shards.0.searches.0.query.0.children.0.breakdown.next_doc/]
// TESTRESPONSE[s/"time_in_nanos": "210682"/"time_in_nanos": $body.profile.shards.0.searches.0.query.0.children.1.time_in_nanos/]
// TESTRESPONSE[s/"score": 4552/"score": $body.profile.shards.0.searches.0.query.0.children.1.breakdown.score/]
// TESTRESPONSE[s/"build_scorer": 42602/"build_scorer": $body.profile.shards.0.searches.0.query.0.children.1.breakdown.build_scorer/]
// TESTRESPONSE[s/"create_weight": 89323/"create_weight": $body.profile.shards.0.searches.0.query.0.children.1.breakdown.create_weight/]
// TESTRESPONSE[s/"next_doc": 2852/"next_doc": $body.profile.shards.0.searches.0.query.0.children.1.breakdown.next_doc/]
// TESTRESPONSE[s/"time_in_nanos": "304311"/"time_in_nanos": $body.profile.shards.0.searches.0.collector.0.time_in_nanos/]
// TESTRESPONSE[s/"time_in_nanos": "32273"/"time_in_nanos": $body.profile.shards.0.searches.0.collector.0.children.0.time_in_nanos/]
```

To how you can cavalierly mock all the numbers at once with this change:
```
// TESTRESPONSE[s/(?<=[" ])\d+(\.\d+)?/$body.$_path/]
```
2017-05-23 15:33:48 -04:00
javanna 7a3e38eb8e Merge branch 'master' into feature/client_aggs_parsing 2017-05-22 12:25:14 +02:00
Jason Tedor 8e2ab43009 Fix handling of dirty PID file
When Elasticsearch dies during a standalone REST test we might leave a
dirty PID file laying around. We tried to log about this, but the log
messages contained references to undefined variables so we simply died
instead of providing a helpful message to run clean. This commit
addresses this issue.
2017-05-19 13:29:14 -04:00
javanna db0490343e Merge branch 'master' into feature/client_aggs_parsing 2017-05-19 18:17:06 +02:00
Nicholas Knize deb7caf4d3 Upgrade to lucene-7.0.0-snapshot-a0aef2f
This commit upgrades master to a current lucene snapshot with commit id a0aef2f.
2017-05-19 10:20:55 -05:00