Commit Graph

28582 Commits

Author SHA1 Message Date
Luca Cavanna b9ce222a6e [DOCS] add links to javadocs to clients docs (#25745)
We publish javadocs to artifacts.elastic.co (and snapshots.elastic.co) for a while. This commit adds the link to them to the transport client, low level REST client, sniffer and high level REST client pages.

Closes #23761
2017-07-31 15:41:21 +02:00
Martijn van Groningen ff3b909a83
Moved HtmlStripCharFilterFactory to analyis.common package like the other factories. 2017-07-31 15:34:54 +02:00
Martijn van Groningen 0b776a1de0
Move more token filters to analysis-common module
The following token filters were moved: delimited_payload_filter, keep, keep_types, classic, apostrophe, decimal_digit, fingerprint, min_hash and scandinavian_folding.

Relates to #23658
2017-07-31 15:15:04 +02:00
Jason Tedor fd18e3239a Remove mention of http_address in nodes info docs
This commit removes an outdated reference to http_address in the nodes
info docs. This information is available in the http object for each
node in the nodes info API response.

Relates #25980
2017-07-31 22:04:16 +09:00
Jason Tedor c0511882b2 Add test for limits on systemd
We set some limits in the service file for Elasticsearch when installed
as a service on systemd-based systems. This commit adds a packaging test
that these limits are indeed set correctly.

Relates #25976
2017-07-31 21:53:16 +09:00
Jason Tedor 65a75250bc Set address space limit in systemd service file
We have a bootstrap check for the maximum size of the virtual memory
address space for the Elasticsearch process. We can set this in the
service file for Elasticsearch when installed as a service on
systemd-based systems for a better user experience than them fumbling
through thinking they should set this via /etc/security/limits.d (as a
lot of pages on the Internet would tell them) not realizing that systemd
completely ignores these for services and then trying to figure out how
to add a unit file for the Elasticsearch service.

Relates #25975
2017-07-31 21:51:00 +09:00
Jason Tedor 14d46754e6 Remove memlock suggestion from systemd service
The systemd service file that ships with Elasticsearch installs on
systemd-based systems contains a suggestion for setting LimitMEMLOCK if
the user wants to enable bootstrap.memory_lock. However, this setting
this in the installed service file goes against best practices for
working with systemd, and goes against our existing documentation for
how to set this. Therefore, we should not have this suggestion in the
service file otherwise users might be led to think they should edit it
there.

Relates #25979
2017-07-31 21:38:38 +09:00
Jason Tedor 540413b24a Also skip JAVA_TOOL_OPTIONS on Windows
On non-Windows platforms, we ignore the environment variable
JAVA_TOOL_OPTIONS (this is an environment variable that the JVM respects
by default for picking up extra JVM options). The primary reason that we
ignore this because of the Jayatana agent on Ubuntu; a secondary reason
is that it produces an annoying "Picked up JAVA_TOOL_OPTIONS: ..."
output message. When the elasticsearch-env batch script was introduced
for Windows, ignoring this environment variable was deliberately not
carried over as the primary reason does not apply on Windows. However,
after additional thinking, it seems that we should simply be consistent
to the extent possible here (and also avoid that annoying "Picked up
JAVA_TOOL_OPTIONS: ..." on Windows too). This commit causes the Windows
version of elasticsearch-env to also ignore JAVA_TOOL_OPTIONS.

Relates #25968
2017-07-31 21:27:42 +09:00
Jason Tedor d9e0c5e2fe Remove useless checks from evil JNA tests
This commit removes some useless empty lines checks from the evil JNA
tests. These empty lines checks are useless because if the lines are
actually empty, the for loop will never be entered and we will hit the
fail condition at the bottom as intended anyway.
2017-07-31 21:04:48 +09:00
Jason Tedor 2ef0f8af38 Add max file size bootstrap check
This commit adds a bootstrap check for the maximum file size, and
ensures the limit is set correctly when Elasticsearch is installed as a
service on systemd-based systems.

Relates #25974
2017-07-31 21:01:47 +09:00
Jason Tedor b7fe8b4722 Exit Windows scripts promptly on failure
When invoking the elasticsearch-env.bat batch script on Windows, if the
script exits due to an error (e.g., Java can not be found, or the wrong
version of Java is found), then the script exits. Sadly, on Windows,
this does not also terminate the caller, instead returning control. This
means we have to explicitly exit so that is what we do in this commit.

Relates #25959
2017-07-31 20:40:17 +09:00
Adrien Grand 57e8b9ee29 Remove another reference to 64-bit systems. 2017-07-31 09:56:06 +02:00
Adrien Grand 1b34f691e5 Remove reference to 32-bit systems. (#25971)
They are not supported anymore as of #25435.
2017-07-31 09:55:09 +02:00
propulkit a2fd1b9f6d No more using 'hybrid mmapfs / niofs' (#25944)
It looks a bit ambiguous here.

ElasticSearch no more using 'hybrid mmapfs / niofs' which chooses filesystem based on the file. It is any one of the mmapfs, niofs or simplefs depending on the operating system.
As quoted here https://www.elastic.co/guide/en/elasticsearch/reference/5.5/index-modules-store.html

Thanks,
Pulkit Agrawal
2017-07-31 09:53:55 +02:00
Jason Tedor 1afc9afcac Version option should display if snapshot
We have a command-line flag -V or --version that can be used to display
the version of Elasticsearch. However, the version that we display does
not contain whether or not the version is a snapshot build. This commit
changes the behavior here so that if the build is a snapshot, that is
included in the version string.

Relates #25970
2017-07-31 11:45:06 +09:00
Jason Tedor bd538aa72c Ignore JVM options before checking Java version
Today we strip some ignored JVM options before starting the main Java
process (e.g., we unset JAVA_TOOL_OPTIONS, and we ignore
JAVA_OPTS). However, there is another Java process that we start before
starting the main process: the Java version checker. We are currently
starting this before ignoring the undesired JVM options so the Java
version checker will pick up JAVA_TOOL_OPTIONS and it will silently
ignore JAVA_OPTS. Instead, we should ignore JAVA_TOOL_OPTIONS here too,
and not silently ignore JAVA_OPTS but instead warn before doing so (as
we already do for the main Java process). This commit rearranges the
execution of these steps so that we do the right thing here.

Relates #25969
2017-07-31 11:41:27 +09:00
Jason Tedor 9267048878 Remove dead code for checking exclusive options
Previously we manually checked if mutually exclusive options are passed
on the command line. Yet, after an upgrade to our option parser
dependency, we were able to use built-in functionality to establish
these mutually exclusive options and the parser would take care of
checking if such options are passed on the command line. However, the
previous manually checking code is now dead and was left behind. This
commit removes that dead code.

Relates #19278
2017-07-31 10:00:31 +09:00
Jason Tedor b54886d502 Fix typo in Elasticsearch help
This commit fixes a small typo in the help output displayed by
Elasticsearch when the --help flag is passed.
2017-07-31 09:56:47 +09:00
Jason Tedor 80620bd086 Add version command to issue template
This commit adds a hint to use the Elasticsearch --version flag to
produce the requested version information.
2017-07-31 08:55:31 +09:00
Jason Tedor 40a7777925 Fix typo in Windows elasticsearch-env script
This commit fixes a simple typo in the Windows version of the
elasticsearch-env script; this typo is harmless as it appears in a
comment.
2017-07-31 06:37:36 +09:00
Jason Tedor 691f2940da Remove running outside of distribution check
This commit removes a legacy check against running bin/elasticsearch
that is not produced from a distribution. This check exists for legacy
reasons, namely when bin/elasticsearch previously sat in the root of the
Elasticsearch repository. In this old scenario, someone might clone the
repository, see the bin folder and try to run bin/elasticsearch without
first production a distribution. Today, this is unlikely since
bin/elasticsearch now sits in
distribution/src/main/resources/bin/elasticsearch so first, bin is no
longer in the root of the repository, and second, the src indicates this
is source and not already for production. Moreover, our README in the
root of the repository provides clear instructions for getting started:
either download a distribution or build one from source. In the name of
simplicity, we therefore remove this legacy check.

Relates #25960
2017-07-30 16:24:57 +09:00
Jason Tedor 8eb4a3f6fa Remove busted rolling upgrade script test
This commit removes a rolling upgrade test for scripting that is totally
busted yet is preventing builds from succeeding. We elect to remove this
test as opposed to skipping the test as:
 - it has beeen being skipped for months with no apparent loss
 - it appears to need significant work to get to an unbusted state
2017-07-30 12:01:03 +09:00
Jason Tedor 4c37335f1d Format CLI error message when es.path.conf not set
This commit adds some formatting to the message displayed when
es.path.conf is not set.
2017-07-30 09:49:55 +09:00
Jason Tedor 8229a57c53 Cleanup script packaging
This commit cleans up a few items with the script packaging:
 - remove the now dead elasticsearch.in.sh script
 - add assertions for the existence elasticsearch-env and
   elasticsearch-keystore
2017-07-30 09:32:17 +09:00
Jason Tedor 7afe96b2ff Introduce elasticsearch-env for Windows
This commit introduces the elasticsearch-env.bat script for Windows to
simplify development and maintenance of scripts on Windows.

Relates #25958
2017-07-30 09:31:41 +09:00
Jim Ferenczi 636748e270 [Test] Make sure the same exception is thrown for every test run. Fixes #25952 2017-07-28 19:02:58 +02:00
Igor Motov fe46ef393b Snapshot/Restore: Ensure that shard failure reasons are correctly stored in CS (#25941)
The failure reason for snapshot shard failures might not be propagated properly if the master node changes after the errors were reported by other data nodes. This commits ensures that the snapshot shard failure reason is preserved properly and adds workaround for reading old snapshot files where this information might not have been preserved.

Closes #25878
2017-07-28 12:28:02 -04:00
Jason Tedor e508c277e1 Fix path.conf usage on Windows
This commit addresses a change in core Elasticsearch where the
command-line flag --path.conf is no longer respected. Instead, the
configuration path must be passed through the system property
es.path.conf. We adapt the Windows batch file and the service for this
change.
2017-07-29 00:19:18 +09:00
Martijn van Groningen ec7ac32772
docs: document work around for the percolator if query time text analysis is expensive. 2017-07-28 15:04:15 +02:00
Martijn van Groningen 7c3735bdc4
percolator: Store the QueryBuilder's Writable representation instead of its XContent representation.
The Writeble representation is less heavy to parse and that will benefit percolate performance and throughput.

The query builder's binary format has now the same bwc guarentees as the xcontent format.

Added a qa test that verifies that percolator queries written in older versions are still readable by the current version.
2017-07-28 12:24:10 +02:00
Daniel Mitterdorfer db90455afd Update plugin-related output in reference docs (#25897)
The example output for node info and cluster stats was outdated w.r.t.
to the information that is shown for plugins. With this commit we
updated the example output and update the explanation of the respective
fields.
2017-07-28 11:27:54 +02:00
Yannick Welsch 1a01514081 Move tribe to a module (#25778)
This commit moves tribe to a module, stripping core from the tribe functionality.
2017-07-28 11:23:50 +02:00
Martijn van Groningen 5cf56a846a
docs: Remove incorrect warning
Closes #25935
2017-07-28 10:53:47 +02:00
Jim Ferenczi 562c3744ca Merge FunctionScoreQuery and FiltersFunctionScoreQuery (#25889)
This change merges the functionality of the FiltersFunctionScoreQuery in the FunctionScoreQuery.
It also ensures that an exception is thrown when the computed score is equals to Float.NaN or Float.NEGATIVE_INFINITY.
These scores are invalid for TopDocsCollectors that relies on score comparison.

Fixes #15709
Fixes #23628
2017-07-28 09:22:20 +02:00
Jason Tedor 9f12f3c338 Fix checkstyle violation in remove plugin tests
This commit fixes a redundant modifier checkstyle violation in
RemovePluginCommandTests.java.
2017-07-28 14:52:11 +09:00
Jason Tedor 1492ccd7ae Fix environment-aware command tests
This commit fixes tests for environment-aware commands. A previous
change added a check that es.path.conf is not null. The problem is that
this system property is not being set in tests so this check trips every
single time. To fix this, we move the check into a method that can be
overridden, and then override this method in relevant places in tests to
avoid having to set the property in tests. We also add a test that this
check works as expected.
2017-07-28 14:37:04 +09:00
Tim Brooks 7d2d6bd752 Make calls to CloudBlobContainer#exists privileged (#25937)
This is related to #25931. In CloudBlobContainer#exists it is possible
that a socket connection will be opened. This commit ensures that those
calls have the proper socket privileges.
2017-07-27 22:29:24 -05:00
Tim Brooks 71f58e6f26 Ensure that gcs client creation is privileged (#25938)
This is related to #25932. Currently when we create the
`GoogleCloudStorageService` client we do not wrap that call in a
doPrivileged block. The call might open a connection. This commit
ensures that the creation is wrapped in a doPrivileged block.
2017-07-27 22:28:47 -05:00
Jason Tedor c1ee65f990 Remove unused imports from EnvironmentAwareCommand
This commit removes two unused imports from EnvironmentAwareCommand that
were left behind after a previous change.
2017-07-28 12:20:18 +09:00
Jason Tedor 8639bf4a1a Pass config path as a system property
A previous change enabled it so that users could configure the
configuration path via a command-line option --path.conf. However, a
subsequent change has made it so that we expect users to set the
configuration path via the environment variable CONF_DIR. To enable
this, we now pass the value of CONF_DIR as the value for the
command-line option --path.conf. This has two problems:
 - the presence of --path.conf always being on the command line breaks
   other flags like --help for multi-commands
 - the scripts for which --help is not broken say that you can pass
   --path.conf but this is a lie since passing it will make it appear
   twice in the command-line arguments breaking the script

Since --path.conf is no longer the way that we want users to set the
configuration path, we should remove the --path.conf option. However, we
still need a way to get the configuration path from the scripts to the
running Java process. To do this, we now pass the configuration path as
a system property. This keeps it off the script command line fixing the
above problems.

The only remaining question (that I can see) is whether or not to
respect -Des.path.conf=<some path> if the user sets this in their
jvm.options or via ES_JAVA_OPTS. I think that we should not do this (as
has been our tradition), es.path.home and es.path.conf are special,
should be set by our scripts only so users should not be setting them at
all so we should not take any effort to respect these flags if the user
tries to otherwise use them.

Relates #25943
2017-07-28 12:15:22 +09:00
Jason Tedor 6c650874c9 Fix scripts when launching from bin
When running a script that depends on elasticsearch-env, the
elasticsearch-env script seeks backwards from the directory containing
the script to find Elasticsearch home. This is done by seeking backwards
in the path to find bin, and then going one directory above
that. Unfortunately, if the script is started relatively from the bin
directory, then bin will appear in the path since it is a relative
path. This commit fixes this by making the starting path absolute before
attempting to seek backwards.
2017-07-28 10:21:34 +09:00
Michael Basnight 3a20922046 Fix eclipse issues related to rest client shading (#25874)
* A cycle was detected in eclipse, and was fixed in the same fashion as
  core and core-tests.
* The rest client deps jar was not properly exported in the generated
  eclipse classpath file for rest client.

Relates #25208
2017-07-27 10:20:53 -05:00
Michael Basnight 2c271f0f22 Remove the shadow plugin apply in the rest client (#25921)
The low level rest client does not need the shadow plugin applied, it
only needs the plugin jar in the classpath, in order to create a
ShadowJar task.

Relates #25208
2017-07-27 09:22:18 -05:00
Yannick Welsch efd79882a2 Allow build to directly run under JDK 9 (#25859)
With Gradle 4.1 and newer JDK versions, we can finally invoke Gradle directly using a JDK9 JAVA_HOME without requiring a JDK8 to "bootstrap" the build. As the thirdPartyAudit task runs within the JVM that Gradle runs in, it needs to be adapted now to be JDK9 aware.

This commit also changes the `JavaCompile` tasks to only fork if necessary (i.e. when Gradle's JVM and JAVA_HOME's JVM differ).
2017-07-27 16:14:04 +02:00
Clinton Gormley 3e568f52c1 Fixed asciidoc formatting 2017-07-27 15:55:52 +02:00
Yannick Welsch 020ba41c5d Close translog view after primary-replica resync (#25862)
The translog view was being closed too early, possibly causing a failed resync. Note: The bug only affects unreleased code.

Relates to #24841
2017-07-27 14:36:51 +02:00
Yannick Welsch 620536f850 Release operation permit on thread-pool rejection (#25930)
At the shard level we use an operation permit to coordinate between regular shard operations and special operations that need exclusive access. In ES versions < 6, the operation requiring exclusive access was invoked during primary relocation, but ES versions >= 6 this exclusive access is also used when a replica learns about a new primary or when a replica is promoted to primary.

These special operations requiring exclusive access delay regular operations from running, by adding them to a queue, and after finishing the exclusive access, release these operations which then need to be put back on the original thread-pool they were running on. In the presence of thread pool rejections, the current implementation had two issues:

- it would not properly release the operation permit when hitting a rejection (i.e. when calling ThreadedActionListener.onResponse from IndexShardOperationPermits.acquire).
- it would not invoke the onFailure method of the action listener when the shard was closed, and just log a warning instead (see ThreadedActionListener.onFailure), which would ultimately lead to the replication task never being cleaned up (see #25863).

This commit fixes both issues by introducing a custom threaded action listener that is permit-aware and properly deals with rejections.

Closes #25863
2017-07-27 14:15:00 +02:00
javanna 19843b50e9 [DOCS] update low level client artifact name 2017-07-27 11:19:40 +02:00
Adrien Grand 1cd5e3413d Caching a MinDocQuery can lead to wrong results. (#25909)
Queries are supposed to be cacheable per segment, yet matches of this query
also depend on how many documents exist on previous segments.
2017-07-27 11:19:20 +02:00
Adrien Grand 876c7e0400 Fix random score generation when no seed is provided. (#25908)
It fixes random score generation to ensure that you will not always get the
same scores on a read-only index by integrating the seed into the score
computation when using doc ids. It also removes `ctx.docBase` from the formula
since it might change over time if deletes are compacted while scores are
supposed to be cacheable per segment.
2017-07-27 11:17:56 +02:00