Commit Graph

6458 Commits

Author SHA1 Message Date
Simon Willnauer d402ca0dd7 Remove poor-mans compression in InternalSearchHit and friends (#20472)
We still use some crazy poor mans compression in InternalSearchHit that
uses a thread local and an unordered map as a lookup table if requested.
Stuff like this should be handled by compression on the transport layer
rather than in-line in the serialization code. This code is complex enough.
2016-09-14 15:25:25 +02:00
Simon Willnauer c1e84618a6 Only try to read new segments info if we really flushed the index (#20474)
There is no reason to read the current segments info unless we flushed
/ committed the lucene index.
2016-09-14 15:23:17 +02:00
Boaz Leskes 74fc074e5e fix styling 2016-09-14 10:52:10 +02:00
Simon Willnauer a1cd6be777 Don't register SearchTransportService handlers more than once (#20468)
This utility class is used in 3 places while we only need to register
the handlers once per node. Otherwise we will see nasty `WARN` logs like:
`registered two transport handlers for action indices:data/read/search[phase/fetch/id/scroll]...`

This change will only register handlers inside the main TransportSearchAction.
2016-09-14 10:34:40 +02:00
Simon Willnauer 89640965d2 Unguice SearchModule (#20456)
After this change SearchModule doesn't subclass AbstractModule anymore and all wiring
happens in `Node.java`. As a side-effect several tests don't need a guice injector anymore.
2016-09-14 10:07:53 +02:00
Jason Tedor 7560101ec7 Complete Elasticsearch logger names
This commit modifies the logger names within Elasticsearch to be the
fully-qualified class name as opposed removing the org.elasticsearch
prefix and dropping the class name. This change separates the root
logger from the Elasticsearch loggers (they were equated from the
removal of the org.elasticsearch prefix) and enables log levels to be
set at the class level (instead of the package level).

Relates #20457
2016-09-13 22:46:54 -04:00
Jason Tedor 0eff7daf5b Fix logging hierarchy configs
Today when setting the logging level via the command-line or an API
call, the expectation is that the logging level should trickle down the
hiearchy to descendant loggers. However, this is not necessarily the
case. For example, if loggers x and x.y are already configured then
setting the logging level on x will not descend to x.y. This is because
the logging config for x.y has already been forked from the logging
config for x. Therefore, we must explicitly descend the hierarchy when
setting the logging level and that is what this commit does.

Relates #20463
2016-09-13 22:46:14 -04:00
Ali Beyad 4431720c3d File-based discovery plugin (#20394)
This commit introduces a new plugin for file-based unicast hosts
discovery. This allows specifying the unicast hosts participating
in discovery through a `unicast_hosts.txt` file located in the
`config/discovery-file` directory. The plugin will use the hosts 
specified in this file as the set of hosts to ping during discovery.

The format of the `unicast_hosts.txt` file is to have one host/port
entry per line. The hosts file is read and parsed every time
discovery makes ping requests, thus a new version of the file that
is published to the config directory will automatically be picked
up.

Closes #20323
2016-09-13 20:52:39 -04:00
Nicholas Knize 87b06c75b0 [TEST] Fix geo_point backcompat tests
This commit fixes the following geo_point bwc tests:

* GeoDistanceIT to test deprecated GeoDistanceRangeQuery on legacy indexes only.
* ExternalFieldMapperTests to correctly handle LatLonPoint type
* GeoPointFieldMapperTests to correctly test stored geo_point fields
2016-09-13 16:27:55 -05:00
Nicholas Knize 821004d5cd [TEST] Refactor LegacyGeohashMappingGeoPointTests to 2.x indices only
These tests should only exist to ensure backcompat with 2.x indices.
2016-09-13 15:27:39 -05:00
Jim Ferenczi 1764ec56b3 Fixed naming inconsistency for fields/stored_fields in the APIs (#20166)
This change replaces the fields parameter with stored_fields when it makes sense.
This is dictated by the renaming we made in #18943 for the search API.

The following list of endpoint has been changed to use `stored_fields` instead of `fields`:
* get
* mget
* explain

The documentation and the rest API spec has been updated to cope with the changes for the following APIs:
* delete_by_query
* get
* mget
* explain

The `fields` parameter has been deprecated for the following APIs (it is replaced by _source filtering):
* update: the fields are extracted from the _source directly.
* bulk: the fields parameter is used but fields are extracted from the source directly so it is allowed to have non-stored fields.

Some APIs still have the `fields` parameter for various reasons:
* cat.fielddata: the fields paramaters relates to the fielddata fields that should be printed.
* indices.clear_cache: used to indicate which fielddata fields should be cleared.
* indices.get_field_mapping: used to filter fields in the mapping.
* indices.stats: get stats on fields (stored or not stored).
* termvectors: fields are retrieved from the stored fields if possible and extracted from the _source otherwise.
* mtermvectors:
* nodes.stats: the fields parameter is used to concatenate completion_fields and fielddata_fields so it's not related to stored_fields at all.

Fixes #20155
2016-09-13 20:54:41 +02:00
Jason Tedor fbe27664a6 Fix prefix logging
Today we add a prefix when logging within Elasticsearch. This prefix
contains the node name, and index and shard-level components if
appropriate.

Due to some implementation details with Log4j 2 , this does not work for
integration tests; instead what we see is the node name for the last
node to startup. The implementation detail here is that Log4j 2 there is
only one logger for a name, message factory pair, and the key derived
from the message factory is the class name of the message factory. So,
when the last node starts up and starts setting prefixes on its message
factories, it will impact the loggers for the other nodes.

Additionally, the prefixes are lost when logging an exception. This is
due to another implementation detail in Log4j 2. Namely, since we log
exceptions using a parameterized message, Log4j 2 decides that that
means that we do not want to use the message factory that we have
provided (the prefix message factory) and so logs the exception without
the prefix.

This commit fixes both of these issues.

Relates #20429
2016-09-13 14:46:34 -04:00
Nicholas Knize 1a60e1c3d2 Update docs for LatLonPoint cut over
This commit removes documentation for:

* geohash cell query
* lat_lon parameter
* geohash parameter
* geohash_precision parameter
* geohash_prefix parameter

It also updates failing tests that reference these parameters for backcompat.
2016-09-13 12:18:21 -05:00
Nicholas Knize ef926894f4 Cut over geo_point field and queries to new LatLonPoint type
This commit cuts over geo_point fields to use Lucene's new point-based LatLonPoint type for indexes created in 5.0. Indexes created prior to 5.0 continue to use their respective encoding type. Below is a description of the changes made to support the new encoding type:

* New indexes use a new LatLonPointFieldMapper which provides a parse method for the new type
* The new LatLonPoint parse method removes support for lat_lon and geohash parameters
* Backcompat testing for deprecated lat_lon and geohash parameters is added to all unit and integration tests
* LatLonPointFieldMapper provides DocValues support (enabled by default) which uses Lucene's new LatLonDocValuesField type
* New LatLonPoint field data classes are added for aggregation support (wraps LatLonPoint's Numeric Doc Values)
* MultiFields use the geohash as the string value instead of the lat,lon string making it easier to perform geo string queries on the geohash instead of a lat,lon comma delimited string.

Removed Features:

* With the removal of geohash indexing, GeoHashCellQuery support is removed for all new indexes (still supported on existing indexes)
* LatLonPoint does not support a Distance Range query because it is super inefficient. Instead, the geo_distance_range query should be accomplished using either the geo_distance aggregation, sorting by descending distance on a geo_distance query, or a boolean must not of the excluded distance (which is what the distance_range query did anyway).

TODO:

* fix/finish yaml changes for plugin and rest integration tests
* update documentation
2016-09-13 12:17:36 -05:00
javanna e0074ee9d4 [TEST] fix MultiMatchQueryIT random docs generation so that they don't interfere in score tests
When generating random bogus documents, it could happen that they contain both the terms "the" and "ultimate", which would match the query "the ultimate" better than all the other non bogus documents, which would cause testCrossFieldMode to fail. "the" is a term that's relatively likely to be randomly generated given its length; we can simply increase the minimum length of randomly generated terms to 5, so that there are no collisions, as "the" cannot be generated anymore (nor can "ultimate" as the lenght doesn't go up to 8).

Also made some assertions more accurate to check how many hits match a query rather than checking only that the first or second hits are there.

Closes #18873
2016-09-13 18:25:53 +02:00
Nik Everett afbd7cbeb8 Rework the basic IT for GETing running tasks
This integ test relied on the false assumption that
`MockTaskManagerListener#onTaskUnregistered` was called *before* the
task was unregistered. It is in fact called after the task is unregistered.
This mistake led the test to *rarely* miss the task it was looking
for and fail.

Found by https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+5.x+multijob-unix-compatibility/os=ubuntu/4/consoleText
2016-09-13 11:59:50 -04:00
Simon Willnauer c84bc25500 Cleanup version constant for unsupported version in QuerySearchResult 2016-09-13 17:21:04 +02:00
Nik Everett 7888dbfb31 Add second test case for two fields in range query
In this test one field is a number and the other is a date.

Closes #20447
2016-09-13 09:26:29 -04:00
Britta Weber 444c4f1af8 remove workaround for highlighter bug with geo queries (#20418)
This has been fixed in Lucene
https://issues.apache.org/jira/browse/LUCENE-7293
This commit also adds the tests from #20412
2016-09-13 14:59:56 +02:00
Tanguy Leroux 6090c51fc5 Add quiet option to disable console logging (#20422)
This commit adds a -q/--quiet option to Elasticsearch so that it does not log anything in the console and closes stdout & stderr streams. This is useful for SystemD to avoid duplicate logs in both journalctl and /var/log/elasticsearch/elasticsearch.log while still allows the JVM to print error messages in stdout/stderr if needed.

closes #17220
2016-09-13 14:08:24 +02:00
Jason Tedor c7bfbe3e69 Add health status parameter to cat indices API
This commit adds a health status parameter to the cat indices API for
filtering on indices that match the specified status (green|yellow|red).

Relates #20393
2016-09-13 07:57:18 -04:00
Michael Nitschinger 9ee6624fd1 Network: Allow to listen on virtual interfaces.
Previously when trying to listen on virtual interfaces during
bootstrap the application would stop working - the interface
couldn't be found by the NetworkUtils class.

The NetworkUtils utilize the underlying JDK NetworkInterface
class which, when asked to lookup by name only takes physical
interfaces into account, failing at virtual (or subinterfaces)
ones (returning null).

Note that when interating over all interfaces, both physical and
virtual ones are taken into account.

This changeset asks for all known interfaces, iterates over them
and matches on the given name as part of the loop, allowing it
to catch both physical and virtual interfaces.

As a result, elasticsearch can now also serve on virtual
interfaces.

A test case has been added which  makes sure that all
iterable interfaces can be found by their respective name.

Note that this PR is a second iteration over the previously
merged but later reverted #19537 because it causes tests
to fail when interfaces are down. The test has been modified
to take this into account now.

Closes #17473
Closes #19568
Relates #19537
2016-09-13 13:40:09 +02:00
javanna 7894eba2b3 [TEST] add test for match query parsing error when providing an array of terms
Match query throws parsing errors when an array of terms is provided, we should test that to make sure this behaviour doesn't change.

Relates to #15741
2016-09-13 12:46:35 +02:00
Boaz Leskes 10dcfa3304 Fix concurrency issues between cancelling a relocation and marking shard as relocated (#20443)
Once a primary is marked as relocated, we can not safely move it back to started (as we have no way of waiting on inflight operations that are performed on the target primary). If the master cancels the relocation in that state, we fail the primary. Sadly, there is a racing condition between the `updateRoutingEntry` method (which is called when the relocation is cancelled by the master) and the `relocated` method. That racing condition can leave the shard as marked "relocated" but have the routing entry not reflect the target relocation. This in turns causes NPEs in TransportReplicationAction:

```
java.util.Objects requireNonNull Objects.java 203
org.elasticsearch.action.support.replication.TransportReplicationAction$ConcreteShardRequest <init> TransportReplicationAction.java 982
```

Sadly, once we end up in this state, we will never recover.

This commit fixes that race condition by making sure `updateRoutingEntry` acquires the mutex when checking for the relocated status.  While at it, I also tightened up the code and added lots of assertions/hard checks.
2016-09-13 12:44:40 +02:00
makeyang 1ae8d6123f Add node name to decider trace logging (#20437)
Adds the entire DiscoveryNode object to the trace log in AllocationDeciders.
The allocation decider logging at TRACE level can sometimes be helpful to determine why a shard is not getting allocated on specific nodes. Currently, we only log the node id for these messages. It will be helpful to also include the node name (esp. when dealing with a lot of nodes in the cluster).
2016-09-13 11:17:39 +02:00
Lee Hinman 3439796df3 Merge branch 'pr/18683' 2016-09-12 16:24:09 -06:00
Lee Hinman 44278db1bc Merge pull request #20433 from dakrone/remove-cluster-name-folder-fallback
No longer allow cluster name in data path
2016-09-12 17:01:49 -05:00
Lee Hinman 94625d74e4 No longer allow cluster name in data path
In 5.x we allowed this with a deprecation warning. This removes the code
added for that deprecation, requiring the cluster name to not be in the
data path.

Resolves #20391
2016-09-12 15:47:01 -06:00
Simon Willnauer 686994ae2d Deguice SearchService and friends (#20423)
This change removes the guice dependency handling for SearchService and
several related classes like SearchTransportController and SearchPhaseController.
The latter two now have package private constructors and dependencies like FetchPhase
are now created by calling their constructors explicitly. This also cleans up several users
of the DefaultSearchContext and centralized it's creation inside SearchService.
2016-09-12 22:42:55 +02:00
Boaz Leskes 7f92971f26 remove assumeX methods from IndexShardTests
The cause early termination of tests, which means we don't clean up and close shards, but also don't cause a failure. This in turns makes TestRuleTemporaryFilesCleanup fail on windows (because it does try to clean up, but the files are referenced). Getting stuff like:

 ```
 >    C:\jenkins\workspace\es_core_master_windows-2012-r2\core\build\testrun\test\J3\temp\org.elasticsearch.index.shard.IndexShardTests_68B5E1103D78A58B-001\tempDir-006\indices\_na_\0\translog\translog-1.tlog: java.nio.file.AccessDeniedException: C:\jenkins\workspace\es_core_master_windows-2012-r2\core\build\testrun\test\J3\temp\org.elasticsearch.index.shard.IndexShardTests_68B5E1103D78A58B-001\tempDir-006\indices\_na_\0\translog\translog-1.tlog
 ```
2016-09-12 22:29:42 +02:00
Ali Beyad b1e87aa13c Split allocator decision making from decision application (#20347)
Splits the PrimaryShardAllocator and ReplicaShardAllocator's decision
making for a shard from the implementation of that decision on the
routing table. This is a step toward making it easier to use the same
logic for the cluster allocation explain APIs.
2016-09-12 16:21:39 -04:00
Luca Cavanna 119d198cc5 Merge pull request #20424 from javanna/enhancement/error_fetch_source_disabled
Throw error when trying to fetch fields from source and source is disabled
2016-09-12 18:36:43 +02:00
Boaz Leskes b08352047d Introduce IndexShardTestCase (#20411)
Introduce a base class for unit tests that are based on real `IndexShard`s. The base class takes care of all the little details needed to create and recover shards. 

This commit also moves `IndexShardTests` and `ESIndexLevelReplicationTestCase` to use the new base class. All tests in `IndexShardTests` that required a full node environment were moved to a new `IndexShardIT` suite.
2016-09-12 18:20:25 +02:00
Ali Beyad f39f9b9760 Update discovery nodes after cluster state is published (#20409)
Before, when there was a new cluster state to publish,
zen discovery would first update the set of nodes to
ping based on the new cluster state, then publish the new
cluster state. This is problematic because if the cluster
state failed to publish, then the set of nodes to ping
should not have been updated.

This commit fixes the issue by updating the set of
nodes to ping for fault detection only *after* the new
cluster state has been published.
2016-09-12 12:07:51 -04:00
javanna 2a1ed80262 With #20093 we fixed a NPE thrown when using _source include/exclude and source is disabled in the mappings. Fixing meant ignoring the _source parameter in the request as no fields can be extracted from it.
We should rather throw a clear exception to clearly point out that we cannot extract fields from _source. Note that this happens only when explicitly trying to extract fields from source. When source is disabled and no _source parameter is specified, no errors will be thrown and no source will be returned.

Closes #20408
Relates to #20093
2016-09-12 17:36:48 +02:00
Jim Ferenczi 82fd95fd24 Merge pull request #20400 from jimferenczi/function_score_highlight
Fix highlighting of MultiTermQuery within a FunctionScoreQuery
2016-09-12 15:56:06 +02:00
Luca Cavanna b1a2768d7d Merge pull request #20188 from qwerty4030/fix/3839_multi_index_add_remove
Fix IndexNotFoundException in multi index search request.
2016-09-12 14:42:56 +02:00
Jun Ohtani 770abd7af8 Merge pull request #20396 from johtani/fix/fail_loading_non_prebuilt_tokenfilter_in_analyze_api
Can load non-PreBuiltTokenFilter in Analyze API
2016-09-10 09:35:23 +09:00
Luca Cavanna 4b00cc37a1 Merge pull request #20382 from javanna/enhancement/cleanup_parse_elements
Cleanup sub fetch phase extension point
2016-09-09 22:47:15 +02:00
javanna 9a84cb99f4 remove writeBoolean from searchExtBuilders serialization in SearchSourceBuilder
The list is not optional anymore, default is empty list
2016-09-09 21:24:18 +02:00
Tal Levy dda32545bb add ignore_missing option to relevant processors (#20194) 2016-09-09 12:20:18 -07:00
javanna 17d48c1ff6 adjust SearchExtBuilder javadocs 2016-09-09 21:17:16 +02:00
javanna 90ab460fcc move parsing of search ext sections to the coordinating node 2016-09-09 19:10:42 +02:00
Nicholas Knize 297fc8373d [TEST] Fix offsets in BaseXContentTestCase.testBinaryValueWithOffsetLength
The max value for randomIntBetween is inclusive, so we should use byte array length minus one to avoid an AIOB exception.
2016-09-09 11:47:37 -05:00
javanna 65c7f61ad9 decouple registration of SearchExtParsers from sub fetch phases
Search section supports an ext section that is used to provide additional config needed from plugins. It is now tied to sub fetch phases because it is the only section that may need additional config, but there is no reason for the two to be tightly coupled.

It is now possible to register a searchExtParser independently from a sub fetch phase. All a search ext parser does is parsing some ext section of a search request, whose parsed resulting object is stored in the search context for later retrieval.
2016-09-09 18:05:49 +02:00
javanna 455a2143f1 move SearchExtParser back to o.e.search package
The parser is now needed only for sub fetch phases, but doesn't have to be strictly connected to them, it could be used for something else as well potentially
2016-09-09 18:05:49 +02:00
javanna 12eaeb3945 FetchSubPhase to support a single parser that extends SearchExtParser 2016-09-09 18:05:49 +02:00
javanna f9530dfe8f remove FetchSubPhaseContext in favour of generic fetch sub phase builder of type object
The context was an object where the parsed info are stored. That is more of what we call the builder since after the search refactoring. No need for generics in FetchSubPhaseParser then. Also the previous setHitsExecutionNeeded wasn't useful, it can be removed as well, given that once there is a parsed ext section, it will become a builder that can be retrieved by the sub fetch phase. The sub fetch phase is responsible for doing nothing in case the builder is not set, meaning that the fetch sub phase is plugged in but the request didn't have the corresponding section.
2016-09-09 18:05:49 +02:00
javanna dc2ba90f48 clarify that SearchParseElement is only used for custom fetch sub phases and clean up extension point
SearchParseElement is renamed to FetchSubPhaseParser and moved to the search.fetch package. Its parse method doesn't get the SearchContext as argument anymore, only the XContentParser, and the return type is what gets parsed (the fetch sub phase context which we may as well rename later).

It is the parser that initializes the FetchSubPhaseContext then. SearchService retrieves the parser by name, calls parse against it and stores the result of parsing by name. No need for FetchSubPhase.ContextFactory anymore, which can be removed.
2016-09-09 18:05:49 +02:00
javanna 4b57a0fd97 resolve some line length problems and remove some entry from checkstyle suppressions (deleted classes) 2016-09-09 18:05:49 +02:00
javanna a33ca70ff5 make docValueFields similar to other standard sub fetch phases
Given that doc value fields is our own fetch sub phase, it doesn't need to be implemented like if it was plugged in from the outside. It doesn't need its own fetch sub phase context, but it can just be an instance member in SearchContext
2016-09-09 18:05:49 +02:00
javanna 060e732f50 remove unused topLevelSearcher method from FetchSubPhase.HitContext 2016-09-09 18:05:49 +02:00
javanna 319280bde3 add java docs to all of the SearchPhase implementations 2016-09-09 18:05:49 +02:00
javanna fe6b9d62a5 Remove parseElements method from SearchPhase interface
Parse elements are always empty for all of our search phases. They can be non empty only for sub fetch phases as they are pluggable and search parse element is left to be used only for plugins that plug in their own sub fetch phase. Only FetchPhase needs now the parseElements method then.
2016-09-09 18:05:49 +02:00
Jason Tedor 55a2f26b21 Logging shutdown hack
Log4j has a bug where on shutdown it ignores that JMX might be disabled;
since it does not respect this on shutdown, it proceeds to attempt to
access JMX leading to a security exception that should have otherwise
not occurred had it respected that JMX is disabled. This commit
intentionally introduces jar hell with the Server class to work around
this bug until a fix is released.

Relates #20389
2016-09-09 11:59:15 -04:00
Jason Tedor d8475488b8 Disable console logging
Previously we would disable console logging in certain circumstances
(for example, if Elasticsearch is not in the foreground, or if
Elasticsearch is in the foreground but an exception was thrown during
bootstrap). This commit makes this handling work with Log4j 2. This will
prevent users from seeing double bootstrap check failure messages.

Relates #20387
2016-09-09 09:15:35 -04:00
Tanguy Leroux a148cda476 [TEST] make BaseXContentTestCase platform independent (bis) 2016-09-09 15:07:07 +02:00
Jim Ferenczi 4a1a09cf43 Fix highlighting of MultiTermQuery within a FunctionScoreQuery
Since the sub query of a function score query is checked on CustomQueryScorer#extractUnknwonQuery we try to extract the terms from the rewritten form of the sub query.
MultiTermQuery rewrites query within a constant score query/weight which returns an empty array when extractTerms is called.
The extraction of the inner terms of a constant score query/weight changed in Lucene somewhere between ES version 2.3 and 2.4 (https://issues.apache.org/jira/browse/LUCENE-6425) which is why this problem occurs on ES > 2.3.
This change moves the extraction of the sub query from CustomQueryScorer#extractUnknownQuery to CustomQueryScorer#extract in order to do the extraction of the terms on the original form of the sub query.
This fixes highlighting of sub queries that extend MultiTermQuery since there is a special path for this kind of query in the QueryScorer (which extract the terms to highlight).
2016-09-09 13:00:41 +02:00
Simon Willnauer ac959847ff [TEST] make BaseXContentTestCase platform independent 2016-09-09 09:19:11 +02:00
Jun Ohtani 4c8452f31e Can load non-PreBuiltTokenFilter in Analyze API
Fix the error when using default min_hash in analyze API
2016-09-09 16:10:17 +09:00
Lee Hinman 49695af2ac Remove FORCE version_type
This was an error-prone version type that allowed overriding previous
version semantics. It could cause primaries and replicas to be out of
sync however, so it has been removed.

Resolves #19769
2016-09-08 15:29:47 -06:00
Chris Earle 07ccabbebc Add "version" field to Templates
This adds a version field to Templates, which is itself is unused by Elasticsearch, but exists for users to better manage their own templates. Like description, it's optional.
2016-09-08 16:01:42 -04:00
Simon Willnauer 51de39f21b Remove unreleased version, these versons should be added once they are released 2016-09-08 20:42:03 +02:00
Jason Tedor 27ff4f327c Remove allow unquoted JSON
Previous versions of Elasticsearch permitted unquoted JSON field names even though this is against the JSON spec. This leniency was disabled by default in the 5.x series of Elasticsearch but a backwards compatibility layer was added via a system property with the intention of removing this layer in 6.0.0. This commit removes this backwards compatibility layer.

Relates #20388
2016-09-08 13:36:31 -04:00
Jason Tedor ef2ff1aeab Remove assertion for cluster name in data path
This commit removes an assertion regarding removing the support for
cluster name being part of the data path in favor of a tracking issue.

Relates #20391
2016-09-08 13:09:05 -04:00
Lee Hinman 6666892038 Default `include_in_all` for numeric-like types to false
This includes:

- All regular numeric types such as int, long, scaled-float, double, etc
- IP addresses
- Dates
- Geopoints and Geoshapes

Relates to #19784
2016-09-08 09:09:48 -06:00
Jason Tedor 8fc38f623b Remove logging configuration warning
The 5.x series of Elasticsearch emits a warning if any of the old
logging configuration formats are present. This commit removes that
warning.

Relates #20386
2016-09-08 10:57:48 -04:00
Jason Tedor de43565abc Do not log full bootstrap checks exception
By default, when an exception causes the JVM to terminate, the stack
trace is printed. In the case of failing bootstrap checks, this stack
trace is useless to the user, and might even distract them from seeing
that the bootstrap checks failed for reasons under their control. With
this commit, we cause the stack trace for a failing bootstrap check to
be truncated.

We also modify some methods to not declare that they throw the top level
checked exception type Exception, but instead explicitly declare the
exceptions that they throw. These exceptions are caught and wrapped in a
BootstrapException so that we can percolate only two exception types out
of Bootstrap#init as checked exception, BootstrapException and
NodeValidationException.

Relates #19989
2016-09-08 10:56:11 -04:00
Simon Willnauer 9e3d399ad2 Skip unreleased versions in RestoreBackwardsCompatIT 2016-09-08 16:16:46 +02:00
Lee Hinman b418889665 Revert "Remove FORCE version_type"
This reverts commit b4cc3cd35d.
2016-09-08 07:43:55 -06:00
Jason Tedor 8e7dfae7d1 Remove collect payloads parameter
The collect_payloads parameter of the span_near query was previously
deprecated with the intention to be removed. This commit removes this
parameter.

Relates #20385
2016-09-08 09:37:36 -04:00
Tanguy Leroux 4fb7ac8254 Clean up XContentBuilder
This commit cleans most of the methods of XContentBuilder so that:
- Jackson's convenience methods are used instead of our custom ones (ie field(String,long) now uses Jackson's writeNumberField(String, long) instead of calling writeField(String) then writeNumber(long))
- null checks are added for all field names and values
- methods are grouped by type in the class source
- methods have the same parameters names
- duplicated methods like field(String, String...) and array(String, String...) are removed
- varargs methods now have the "array" name to reflect that it builds arrays
- unused methods like field(String,BigDecimal) are removed
- all methods now follow the execution path: field(String,?) -> field(String) then value(?), and value(?) -> writeSomething() method. Methods to build arrays also follow the same execution path.
2016-09-08 15:09:09 +02:00
Jim Ferenczi f761038dfd Fix UpdateSettingsIT#testUpdateMergeMaxThreadCount.
Log the setting(s) that changed in the compound updater.
2016-09-08 13:38:18 +02:00
Simon Willnauer f319545814 Prepare master branch to be 6.0.0-alpha1 2016-09-08 12:55:30 +02:00
Simon Willnauer e41ae25b1e Add Version [5.0.0.beta1] 2016-09-08 12:50:25 +02:00
Jim Ferenczi 3ad8459e7b Validate max thread/merge settings
This change checks that `index.merge.scheduler.max_thread_count` < `index.merge.scheduler.max_merge_count` and fails index creation
and settings update if the condition is not met.

Fixes #20380
2016-09-08 10:47:43 +02:00
Jason Tedor b8396cd2d6 Shutdown logging in logging configuration tests
The logging configuration tests write to log files which are deleted at
the end of the test. If these files are not closed, some operating
systems will complain when these deletes are performed. This commit
ensures that the logging system is properly shutdown so that these files
can be properly deleted.
2016-09-07 21:27:39 -04:00
Lee Hinman 409fae7607 Merge remote-tracking branch 'dakrone/remove-force-version-type' 2016-09-07 15:48:58 -06:00
Lee Hinman b4cc3cd35d Remove FORCE version_type
This was an error-prone version type that allowed overriding previous
version semantics. It could cause primaries and replicas to be out of
sync however, so it has been removed.

Resolves #19769
2016-09-07 13:05:18 -06:00
Simon Willnauer a96f3d46b7 Add field with dot in name to 2.4+ static bwc indexes (#20360)
This change adds a `field.with.dots` to all 2.4 bwc indicse and above.
It also adds verification code to OldIndexBackwardsCompatibilityIT to
ensure we upgrade the indices cleanly and the field is present.

Closes #19956
2016-09-07 17:21:28 +02:00
Simon Willnauer 8502d2761f Ensure all static BWC indices have some translog entries (#20362)
Due to the way the nodes where shut down etc. we always flushed
away the translog. This means we never tested upgrades of transaction
logs from older version. This change regenerates all valid bwc indices
and repositories with transaction logs and adds correspondent changes
to the OldIndexBackwardsCompatibilityIT.java
2016-09-07 16:53:24 +02:00
Chris Earle 6a7309c09a Add "version" field to Pipelines
This adds a version field to Pipelines, which is itself is unused by Elasticsearch, but exists for users to better manage their own pipelines.
2016-09-07 10:27:40 -04:00
Igor Motov d34fdaac5e Parse script on storage instead of on retrieval
Parsing a script on retrieval causes it to be re-parsed on every single script call, which can be very expensive for large frequently called scripts. This change switches to parsing scripts only once during store operation.
2016-09-07 08:21:33 -04:00
javanna a96c76dcbd Remove FetchSubPhaseParseElement
With the search refactoring we don't use SearchParseElement anymore to define our own parsing code but only for plugins. There was an abstract subclass called FetchSubPhaseParseElement in our production code, only used in one of our tests. We can remove that abstract class as it is not needed and not that useful for the test that depends on it.
2016-09-07 10:05:21 +02:00
Luca Cavanna 0d21d9ff6e Merge pull request #20327 from javanna/enhancement/stats_writeable
NodeStats classes to implement Writeable rather then Streamable
2016-09-07 10:04:52 +02:00
Jun Ohtani bc3d1a8362 Merge pull request #20331 from johtani/cleanup_analyze_api_test
Clean up Analyze API test case
2016-09-07 17:04:28 +09:00
javanna dd9b29b6c6 remove stale comment from CircuitBreakerStats 2016-09-07 10:03:41 +02:00
javanna e3f2af8384 make memoryPools in JvmStats not nullable, default to empty list 2016-09-07 10:03:41 +02:00
javanna a33e4b1d76 use Collections.addAll rather manually copying array 2016-09-07 10:03:41 +02:00
javanna 1ff22fe32a remove bw comp layer that's not needed in CommonStatsFlags 2016-09-07 10:03:41 +02:00
javanna 1a2c7e0d25 [TEST] introduce more intermediate variables in NodeStatsTests to prevent too much line wrapping 2016-09-07 10:03:41 +02:00
javanna a035ca102f Use a list for JvmStats memoryPools rather than an array 2016-09-07 10:03:41 +02:00
javanna 42f88406ee add NodeStatsTests to test NodeStats serialization 2016-09-07 10:03:41 +02:00
javanna dae0580a67 add missing getters to FsInfo.IoStats class
Without the getters there is no way to retrieve the values for its instance members from the java api, they only get printed out on the REST layer
2016-09-07 10:03:41 +02:00
javanna af633a293c Eagerly compute FsInfo#total so that the member instance can become final
FsInfo#total is removed in favour of getTotal, which allows to retrieve the total value

[TEST] fix FsProbeTests: null is not accepted as path constructor argument
2016-09-07 10:03:41 +02:00
javanna f1b1d1cae0 CommonStats and CommonStatsFlags to implement Writeable rather than Streamable 2016-09-07 10:03:41 +02:00
javanna b36bad6cc2 AllCircuitBreakerStats and CircuitBreakerStats to implement Writeable rather than Streamable 2016-09-07 10:03:41 +02:00
javanna 38a7427c51 DiscoveryStats and PendingClusterStateStats to implement Writeable rather than Streamable 2016-09-07 10:03:41 +02:00
javanna d7ad748be7 ScriptStats to implement Writeable rather than Streamable
Also removed ScriptStats#add method which was unused
2016-09-07 10:03:41 +02:00
javanna 3521e2e1a9 HttpStats to implement Writeable rather than Streamable 2016-09-07 10:03:41 +02:00
javanna e263c64072 TransportStats to implement Writeable rather than Streamable 2016-09-07 10:03:41 +02:00
javanna 9c62a12fee ThreadPoolStats to implement Writeable rather than Streamable 2016-09-07 10:03:41 +02:00
javanna 102dac2cd9 JvmStats to implement Writeable rather than Streamable
also removed null checks in toXContent for subobjects that cannot be null and added @Nullable annotation for memory pools
2016-09-07 10:03:41 +02:00
javanna 931a164b1f ProcessStats to implement Writeable rather than Streamable 2016-09-07 10:03:41 +02:00
Colin Goodheart-Smithe 55d9e99f51 Fix filter cache setting to allow percentages
During adding the new settings infrastructure the option to specify the
size of the filter cache as a percentage of the heap size which accidentally
removed. This change adds that ability back.

In addition the `Setting` class had multiple `.byteSizeSetting` methods
which all except one used `ByteSizeValue.parseBytesSizeValue` to parse
the value. One method used `MemorySizeValue.parseBytesSizeValueOrHeapRatio`.
This was confusing as the way the value was parsed depended on how many
arguments were provided.

This change makes all `Setting.byteSizeSetting` methods parse the value
the same way using `ByteSizeValue.parseBytesSizeValue` and adds
`Setting.memorySizeSetting` methods to parse settings that express memory
sizes (i.e. can be absolute bytes values or percentages). Relevant settings
have been moved to use these new methods.

Closes #20330
2016-09-07 08:53:41 +01:00
Alexander Lin f825e8f4cb Exposing lucene 6.x minhash filter. (#20206)
Exposing lucene 6.x minhash tokenfilter

Generate min hash tokens from an incoming stream of tokens that can
be used to estimate document similarity.

Closes #20149
2016-09-07 09:38:12 +02:00
Lee Hinman 7da8be9874 Merge remote-tracking branch 'dakrone/disk-decider-relocation-switcharoo' 2016-09-06 14:46:15 -06:00
Lee Hinman 28d3c4488e Change DiskThresholdDecider's behavior when factoring in leaving shards
This changes DiskThresholdDecider to only factor in leaving shards when
checking if a shard can remain. Previously, leaving shards were factored
in for both the `canAllocate` and `canRemain` checks, however, this
makes only the leaving shard sizes subtracted in the `canRemain` check.

It was possible that multiple shards relocating away from the node would
have their entire size subtracted, and the node had a chance to go over
the disk threshold (or hit the disk full) because it subtracted space
that was still being used for other in-progress relocations.
2016-09-06 14:26:18 -06:00
Nik Everett eb9d2b6659 Make ConcreteShardRequest public and static
Request interceptors need to be able to work with it.
2016-09-06 15:41:14 -04:00
Martijn van Groningen 245882cde3 * Removed `script.default_lang` setting and made `painless` the hardcoded default script language.
** The default script language is now maintained in `Script` class.
* Added `script.legacy.default_lang` setting that controls the default language for scripts that are stored inside documents (for example percolator queries).  This defaults to groovy.
** Added `QueryParseContext#getDefaultScriptLanguage()` that manages the default scripting language. Returns always `painless`, unless loading query/search request in legacy mode then the returns what is configured in `script.legacy.default_lang` setting.
** In the aggregation parsing code added `ParserContext` that also holds the default scripting language like `QueryParseContext`. Most parser don't have access to `QueryParseContext`. This is for scripts in aggregations.
* The `lang` script field is always serialized (toXContent).

Closes #20122
2016-09-06 18:44:48 +02:00
Jason Tedor 0d7dfcd798 Merge pull request #20338 from jasontedor/remove-plugin
Print message when removing plugin with config
2016-09-06 11:43:51 -04:00
Jason Tedor 6df70444a3 Remove Log4j 1 jar hell exemption
When Elasticsearch depended on Log4j 1, there was jar hell from the
log4j and the apache-log4j-extras jar. As these dependencies are gone,
the jar hell exemption for Log4j 1 can be removed.

Relates #20336
2016-09-06 10:25:22 -04:00
Jason Tedor f427d7fe74 More verbose message on preserving plugin config
This commit expands on the message printed when config files are
preserved when removing a plugin to give the user an indication of the
reason the config files are preserved.
2016-09-06 08:51:12 -04:00
Boaz Leskes c56cd46162 Verify AllocationIDs in replication actions (#20320)
Replicated operation consist of a routing action (the original), which is in charge of sending the operation to the primary shard, a primary action which executes the operation on the resolved primary and replica actions which performs the operation on a specific replica. This commit adds the targeted shard's allocation id to the primary and replica actions and makes sure that those match the shard the actions end up executing on.

This helps preventing extremely rare failure mode where a shard moves off a node and back to it, all between an action is sent and the time it's processed. 

For example:
1) Primary action is sent to a relocating primary on node A.
2) The primary finishes relocation to node B and start relocating back.
3) The relocation back gets to the phase and opens up the target engine, on the original node, node A.
4) The primary action is executed on the target engine before the relocation finishes, at which the shard copy on node B is still the official primary - i.e., it is executed on the wrong primary.
2016-09-06 14:32:48 +02:00
Jason Tedor 75956604eb Print message when removing plugin with config
When removing a plugin with a config directory, we preserve the config
directory. This is because the workflow for upgrading a plugin involves
removing and then installing the plugin again and losing the plugin
config in this case would be terrible. This commit causes a message
regarding this to be printed in case the user wants to manually delete
these files.
2016-09-06 08:01:43 -04:00
Jason Tedor ab86660c65 Add finals to RemovePluginCommand
This commit marks the RemovePluginCommand class as final, and marks
some local variables as final too.
2016-09-06 07:39:23 -04:00
Jason Tedor e081b2b2e8 Remove length violation in RemovePluginCommand
This commit removes a line-length violation in RemovePluginCommand.java
and removes this file from the list of files for which the line-length
check is suppressed.
2016-09-06 07:28:05 -04:00
Jason Tedor 7b43d9b0ec Add test for Log4j throwable proxy leniency
We have intentionally introduced leniency for ThrowableProxy from Log4j
to work around a bug there. Yet, a test for this introduced leniency was
not addded. This commit introduces such a test.

Relates #20329
2016-09-06 05:55:06 -04:00
Jason Tedor 0003196749 Remove Joda-Time jar hell exemption
Previously we had an exemption for Joda-Time BaseDateTime because we
forked this class to remove the usage of a volatile field. This hack is
no longer in place, so the exemption is no longer necessary. This commit
removes that exemption.

Relates #20328
2016-09-06 04:47:42 -04:00
Jun Ohtani f0be657699 Clean up Analyze API test case
Using expectThrows instead of using try-catch
2016-09-06 15:46:18 +09:00
Simon Willnauer 5c2d9fa158 Improve error reporting for tests with BackgroundIndexer (#20324)
The BackgroundIndexer now uses auto-generated IDs randomly. This causes some problems
for tests that still rely on the fact that the IDs are increasing integers. This change
exposes all IDs via a Set<String> to iterate over for tests.
2016-09-05 16:28:49 +02:00
Jason Tedor 433cae47ed Mark CSIT#testLoggerLevelUpdate as awaits fix
This commit marks ClusterSettingsIT#testLoggerLevelUpdate as awaiting a
fix due to a test bug.
2016-09-04 11:09:08 -04:00
Jason Tedor 41637a1294 Only warn on old log configs if resolving configs
A warning was introduced if old log config files are present (e.g.,
logging.yml). However, this check is executed unconditionally. This can
lead to no such file exceptions when logging configs are not being
resolved, for example when installing a plugin. This commit moves this
check to only execute when logging configs are being resolved.
2016-09-03 09:48:09 -04:00
Jason Tedor e297fd419b Workaround possible JVM bug on Windows
Some assertions in MaxMapCountCheckTests assert that certain messages
are logged. These assertions pass everywhere except Windows where the
JVM seems confused. The issue is not the javac compiler as the bytecode
produced on OS X and Windows is identical for the relevant classes so
this leaves a possible JVM bug. It is not worth investigating the
ultimate cause of this bug so instead this commit introduces a
workaround.
2016-09-03 09:26:03 -04:00
Jason Tedor b9966fed36 Hack around Log4j bug rendering exceptions
Log4j has a bug where it does not handle a security exception that can
be thrown when it is rendering a stack trace. This commit intentionally
introduces jar hell with the ThrowableProxy class to work around this
bug until a fix is a released.

Relates #20306
2016-09-02 20:26:32 -04:00
Jason Tedor 40f889b825 Warn if unsupported logging configuration present
This commit adds a warning that an unsupported logging configuration is
present and points users to the new logging configuration file.

Relates #20309
2016-09-02 18:36:57 -04:00
Simon Willnauer c992a007c8 Pass on maxUnsafeAutoIdTimestamp on recovery / relocation (#20300)
To ensure we don't add documents more than once even if it's mostly paranoia
except of one case where we relocated a shards away and back to the same node
while an initial request is in flight but has not yet finished AND is retried.

Yet, this is a possible case and for that reason we ensure we pass on the
maxUnsafeAutoIdTimestamp on when we prepare for translog recovery.

Relates to #20211
2016-09-02 21:07:55 +02:00
Ali Beyad d2ab42eabe [TESTS] added higher level logging to the testShadowReplicaNaturalRelocation
test
2016-09-02 14:57:22 -04:00
Jun Ohtani c4759bcc02 Merge pull request #20285 from johtani/fix/remove_token_filter_param_in_analyze_api
Remove `token_filter` in _analyze API
2016-09-03 02:03:51 +09:00
Masaru Hasegawa af959c0c91 Merge pull request #20299 from masaruh/query_string_fuzzy
query_string_query should take term length into consideration when fuzziness is auto
2016-09-02 23:33:49 +09:00
Nik Everett 549ca3178b Rename method in OldIndexUtils
loadIndexList -> loadDataFilesList. The new method name is more accurate.
2016-09-02 10:16:30 -04:00
Masaru Hasegawa 3a13f54755 query_string_query should take term length into consideration when fuzziness is auto
Fixes #15972
2016-09-02 22:17:02 +09:00
javanna 52581d2df6 [TEST] fix bad merge 2016-09-02 10:27:59 +02:00
javanna 51620f755b [TEST] expand NodeInfoStreamingTests to also test serialization of nullable values 2016-09-02 10:23:49 +02:00
javanna 746632fcf9 remove redundant serialization test for JvmInfo and OsInfo and expand existing NodeInfoStreamingTests 2016-09-02 10:23:49 +02:00
javanna e5a741ab67 fix line length in some touched classes 2016-09-02 10:23:49 +02:00
javanna c0a0100308 [TEST] use single line ternary over more verbose ifs 2016-09-02 10:23:05 +02:00
javanna 6873454f33 use read/writeList and readMap where possible 2016-09-02 10:23:05 +02:00
javanna 68eb58f9e3 [TEST] use randomPositiveLong where possible 2016-09-02 10:23:05 +02:00
javanna 774244a61f ThreadPool.Info and SizeValue to implement Writeable rather than Streamable 2016-09-02 10:23:05 +02:00
javanna 84b8c9de19 PluginInfo to implement Writeable rather than Streamable 2016-09-02 10:23:05 +02:00
javanna 555db744f1 use read/writeOptionalWriteable in NodeInfo serialization code 2016-09-02 10:23:05 +02:00
javanna e98e37295a PluginsAndModules to implement Writeable rather than Streamable 2016-09-02 10:23:05 +02:00
javanna 2b2fb8daed TransportInfo to implement Writeable rather than Streamable 2016-09-02 10:23:05 +02:00
javanna 536d13ff11 ProcessInfo to implement Writeable rather than Streamable 2016-09-02 10:23:05 +02:00
javanna 2370c25fa4 ThreadPoolInfo to implement Writeable rather than Streamable 2016-09-02 10:23:05 +02:00
javanna 27e7fc734c HttpInfo to implement Writeable rather than Streamable 2016-09-02 10:23:05 +02:00
javanna 279f8b27e3 JvmInfo to implement Writeable rather than Streamable 2016-09-02 10:23:05 +02:00
javanna bea863c660 OsInfo to implement Writeable rather than Streamable
This allows to make all instance members final. Also added serialization tests and sorted out inizialization that was scattered in two places.
2016-09-02 10:23:05 +02:00
javanna f6ab4e1078 ByteSizeValue to implement Writeable rather than Streamable
With this we can make ByteSizeValue immutable for real.
2016-09-02 10:23:05 +02:00
Luca Cavanna faa03ad9fa Merge pull request #20255 from javanna/enhancement/cluster_stats_available_memory
Add mem section back to cluster stats
2016-09-02 10:19:51 +02:00
Simon Willnauer 724e8ec39c [TEST] Fix settings keys to be the actual keys rather than the toString() of the Setting 2016-09-02 10:00:31 +02:00
Adrien Grand 5bfab76c96 Source filtering should keep working when the source contains numbers greater than `Long.MAX_VALUE`. #20278
Currently it does not because our parsers do not support big integers/decimals
(on purpose) but we do not have to ask our parser for the number type, we can
just ask the jackson parser for a number representation of the value with the
right type.

Note that I did not add similar tests for big decimals because Jackson seems to
never return big decimals, even for decimal values that are out of the range of
values that can be represented by doubles.

Closes #11508
2016-09-02 08:56:04 +02:00
Jun Ohtani aef2e5d90e Remove `token_filter` in _analyze API
Fix wording in docs
Refactoring RestAnalyzeActionTests using expectThrows()

Closes #20283
2016-09-02 15:08:28 +09:00
Jason Tedor 1e80adbfbe Configure test logging with Log4j 2
This commit configures test logging for Log4j 2. The default logger
configuration uses the console appender but at the error level, so most
tests are missing logging. Instead, this commit provides a configuration
for tests which is picked up from the classpath by Log4j 2 when it
initializes. However, this now means that we can no longer initialize
Log4j with a bare-bones configuration when tests run as doing so will
prevent Log4j 2 from attempting to configure logging via the
classpath. Consequently, we move this needed initialization (as
commented, to avoid a message about a status logger not being configured
when we are preparing to configure Log4j from properties files in the
config directory) to only run when we are explicitly configuring Log4j
from properties files.

Relates #20284
2016-09-01 14:00:47 -04:00
javanna 186a5d74b8 [TEST] improve ClusterStatsIT to better check mem values returned
Rather than checking that those values are greater than 0, we can sum up the values gotten from all nodes and check that what is returned is that same value.
2016-09-01 19:22:13 +02:00
Jun Ohtani 3d9f8ed764 Remove `token_filter` in _analyze API
Remove the param and change docs

Closes #20283
2016-09-02 01:36:45 +09:00
Clinton Gormley 0e8a43e826 Elasticsearch 2.4.0 uses Lucene 5.5.2 2016-09-01 12:52:01 +02:00
Martijn van Groningen a110498ad8 settings: Make `action.auto_create_index` setting a dynamic cluster setting.
Closes #7513
2016-09-01 12:33:30 +02:00
Clinton Gormley e5ff3da802 Added version 2.4.0 with bwc indices 2016-09-01 11:36:49 +02:00
javanna 042675432e make sure that mem, cpu and swap are never null in OsStats 2016-09-01 11:26:03 +02:00
javanna 5f299ff46f add mem section back to cluster stats
The mem section was buggy in cluster stats and removed. It is now added back with the same structure as in node stats, containing total memory, available memory, used memory and percentages. All the values are the sum of all the nodes across the cluster (or at least the ones that we were able to get the values from).
2016-09-01 11:26:03 +02:00
javanna 5211b6b4bc OsStats.Cpu, OsStats.Mem & OsStats.Swap to implement ToXContent 2016-09-01 11:24:56 +02:00
javanna 0a7a52a31e OsStats and subobjects to implement Writeable rather than Streamable
We can now have final instance members, also drop some optional values and related null checks that weren't needed.
2016-09-01 11:24:56 +02:00
Adrien Grand 34aaea641d Fix NPE when running a range query on a `scaled_float` with no upper bound. #20253
The null check was there, but on the wrong variable.
2016-09-01 11:23:32 +02:00
Simon Willnauer a0becd26b1 Optimize indexing for the autogenerated ID append-only case (#20211)
If elasticsearch controls the ID values as well as the documents
version we can optimize the code that adds / appends the documents
to the index. Essentially we an skip the version lookup for all
documents unless the same document is delivered more than once.

On the lucene level we can simply call IndexWriter#addDocument instead
of #updateDocument but on the Engine level we need to ensure that we deoptimize
the case once we see the same document more than once.

This is done as follows:

1. Mark every request with a timestamp. This is done once on the first node that
receives a request and is fixed for this request. This can be even the
machine local time (see why later). The important part is that retry
requests will have the same value as the original one.

2. In the engine we make sure we keep the highest seen time stamp of "retry" requests.
This is updated while the retry request has its doc id lock. Call this `maxUnsafeAutoIdTimestamp`

3. When the engine runs an "optimized" request comes, it compares it's timestamp with the
current `maxUnsafeAutoIdTimestamp` (but doesn't update it). If the the request
timestamp is higher it is safe to execute it as optimized (no retry request with the same
timestamp has been run before). If not we fall back to "non-optimzed" mode and run the request as a retry one
and update the `maxUnsafeAutoIdTimestamp` unless it's been updated already to a higher value

Relates to #19813
2016-09-01 10:39:40 +02:00
Simon Willnauer 419627c460 Ensure ESTestCase is initialized before we run tests 2016-09-01 09:39:44 +02:00
Jason Tedor d9064f454e Fix additional exception logging calls
This commit modifies a pair of exception logging calls to use
parameterized messages from Log4j.
2016-08-31 23:14:13 -04:00
Jason Tedor 76ab02e002 Merge branch 'master' into log4j2
* master:
  Avoid NPE in LoggingListener
  Randomly use Netty 3 plugin in some tests
  Skip smoke test client on JDK 9
  Revert "Don't allow XContentBuilder#writeValue(TimeValue)"
  [docs] Remove coming in 2.0.0
  Don't allow XContentBuilder#writeValue(TimeValue)
  [doc] Remove leftover from CONSOLE conversion
  Parameter improvements to Cluster Health API wait for shards (#20223)
  Add 2.4.0 to packaging tests list
  Docs: clarify scale is applied at origin+offest (#20242)
2016-08-31 16:37:55 -04:00
Jason Tedor 487ffe8375 Remove code references to logging.yml
This commit removes code references to logging.yml in TranslogToolCli
and PluginCli.
2016-08-31 15:50:45 -04:00
Nik Everett bd93c7054c Revert "Don't allow XContentBuilder#writeValue(TimeValue)"
This reverts commit 7f70c00dad.
2016-08-31 14:45:03 -04:00
Nik Everett 7f70c00dad Don't allow XContentBuilder#writeValue(TimeValue)
We have specific support for writing `TimeValue`s in the form of
`XContentBuilder#timeValueField`. Writing a `TimeValue` using
`XContentBuilder#writeValue` is a bug waiting to happen.
2016-08-31 13:23:38 -04:00
Ali Beyad 4641254ea6 Parameter improvements to Cluster Health API wait for shards (#20223)
* Params improvements to Cluster Health API wait for shards

Previously, the cluster health API used a strictly numeric value
for `wait_for_active_shards`. However, with the introduction of
ActiveShardCount and the removal of write consistency level for
replication operations, `wait_for_active_shards` is used for
write operations to represent values for ActiveShardCount. This
commit moves the cluster health API's usage of `wait_for_active_shards`
to be consistent with its usage in the write operation APIs.

This commit also changes `wait_for_relocating_shards` from a
numeric value to a simple boolean value `wait_for_no_relocating_shards`
to set whether the cluster health operation should wait for
all relocating shards to complete relocation.

* Addresses code review comments

* Don't be lenient if `wait_for_relocating_shards` is set
2016-08-31 11:58:19 -04:00
Jason Tedor e166459bbe Merge branch 'master' into log4j2
* master:
  Increase visibility of deprecation logger
  Skip transport client plugin installed on JDK 9
  Explicitly disable Netty key set replacement
  percolator: Fail indexing percolator queries containing either a has_child or has_parent query.
  Make it possible for Ingest Processors to access AnalysisRegistry
  Allow RestClient to send array-based headers
  Silence rest util tests until the bogusness can be simplified
  Remove unknown HttpContext-based test as it fails unpredictably on different JVMs
  Tests: Improve rest suite names and generated test names for docs tests
  Add support for a RestClient base path
2016-08-31 10:59:27 -04:00
Jason Tedor 1a805bb675 Increase visibility of deprecation logger
The deprecation logger is an important way to make visible features of
Elasticsearch that are deprecated. Yet, the default logging makes the
log messages for the deprecation logger invisible. We want these log
messages to be visible, so the default logging for the deprecation
logger should enable these log messages. This commit changes the log
level of deprecation log message to warn, and configures the deprecation
logger so that these log messages are visible out of the box.

Relates #20254
2016-08-31 10:51:17 -04:00
Jason Tedor ac8c2e98ab Enable console logging for CLI tools
This commit enables CLI tools to have console logging. For the CLI
tools, we skip configuring the logging infrastructure via the config
file, and instead set the level only via a system property.
2016-08-31 09:05:26 -04:00
Jason Tedor 0fdc5ca587 Remove logger getter from DeprecationLogger
This commit removes an unused getter for the logger field from the
DeprecationLogger.
2016-08-30 21:19:16 -04:00
Igor Motov a68083f5cb Make it possible for Ingest Processors to access AnalysisRegistry
The analysis registry will be used in PMML plugin ingest processor.
2016-08-30 21:09:41 -04:00
Jason Tedor abe3efdfa9 Fix failing max map count check test
This commit fixes failing max map count check test due to the use of a
logging message supplier.
2016-08-30 18:49:39 -04:00
Jason Tedor abf8a1a3f0 Avoid allocating log parameterized messages
This commit modifies the call sites that allocate a parameterized
message to use a supplier so that allocations are avoided unless the log
level is fine enough to emit the corresponding log message.
2016-08-30 18:17:09 -04:00
Jason Tedor 7da0cdec42 Introduce Log4j 2
This commit introduces Log4j 2 to the stack.
2016-08-30 13:31:24 -04:00
Nik Everett df73292256 Add an alias action to delete an index
While removing an index isn't actually an alias action, if we add
an alias action that deletes an index then we can delete and index
and add an alias with the same name as the index atomically, in
the same cluster state update.

Closes #20064
2016-08-30 10:15:21 -04:00
Simon Willnauer 497e7d1054 User lambda instead of annoymous class in SearchPhaseController 2016-08-30 12:58:54 +02:00
Tanguy Leroux b4245c7ad9 Add exclusion filters support to filter_path
This commit adds the support for exclusion filter to the response filtering (filter_path) feature. It changes the XContentBuilder APIs so that it now accepts two types of filters: inclusive and exclusive. Filters are no more String arrays but sets of String instead.
2016-08-30 09:08:30 +02:00
Martijn van Groningen 1925813e09 ingest: Fix rename processor change rename leaf fields into branch fields
Instead of get, set and remove we do get, remove and then set to avoid type conflicts in IngestDocument.
If the set still fails we try to restore the original field in ingest document.

Closes #19892
2016-08-30 07:38:01 +02:00
Ali Beyad a132405642 Ensures that during the restore process, if a file in the snapshot (#20220)
already has a file of the same name in the Store, but is different
in content (different checksum/length), then those files are first
deleted before restoring the files in question.
2016-08-29 17:51:35 -04:00
Ali Beyad 55b91cdc17 Removes unused test helper method to write old blob store format 2016-08-29 12:44:58 -04:00
Areek Zillur 99734ec576 Merge pull request #20034 from areek/cleanup/index_operation
Set created flag in index operation
2016-08-29 12:34:24 -04:00
Nik Everett 9c3f6d58ac Support downgrading keyword/text into string
This changes Elasticsearch to automatically downgrade `text` and
`keyword` fields into appropriate `string` fields when changing the
mapping of indexes imported from 2.x. This allows users to use the
modern, documented syntax against 2.x indexes. It also makes it clear
that reindexing in order to recreate the index in 5.0 is required for
any long lived indexes. This change is useful for the times when you
can't (cluster is just starting, not stable enough for reindex) or
shouldn't (index will only live 90 days or something).
2016-08-29 11:27:37 -04:00
javanna d7ec2db9b0 [TEST] enable cacheKey check in ShardSearchTransportRequestTests
Now that #20081 is merged we can check that cacheKey is consistent across equal search requests, something that wasn't true before due to ordering of map keys when using index boost.

Relates to #19986
2016-08-29 17:20:26 +02:00
Tanguy Leroux 9727f123b9 Rename Netty TCP transports thread factories from http_* to transport_*
Netty3/4 TcpTransport implementations are creating thread factories with a "http_server" thread prefix whereas it should start with "transport_server" and let the "http_server" prefix for the HttpServerTransport implementations.
2016-08-29 13:49:52 +02:00
Yannick Welsch f070c8727b [TEST] Add additional logging to testStaleMasterNotHijackingMajority
This test is periodically failing. As I suspect that the GCDisruption scheme is somehow making the wrong node block on
its cluster state update thread, I've added some more logging and a thread dump once the given assertion triggers
again.
2016-08-29 13:42:13 +02:00
Martijn van Groningen 2d82bea040 fix test bug 2016-08-29 13:28:23 +02:00
Jun Ohtani 2a00c9dc46 Merge pull request #19860 from johtani/fix/validate_empty_field_name
Validate blank field name
2016-08-29 11:52:18 +09:00
Simon Willnauer 62b821ccf4 [TEST] Ensure test never hangs but fails if it doesn't finish after 10 seconds waiting for threads 2016-08-27 23:20:55 +02:00
Simon Willnauer 162ad1251c Fsync documents in an async fashion (#20145)
today we fsync in a blocking fashion where all threads block while another
syncs. Yet, we can improve this and make use of the async infrastrucutre added
for `wait_for_refresh` and make fsyncing single threaded while all other threads
can continue indexing. The syncing thread then notifies a listener once the requests
location is synced. This also allows to send docs to replicas before its actually fsynced
allowing for cocurrent replica processing.

This patch has a significant impact on performance on slower discs. An initial single node benchmark
shows that on very fast SSDs there is no noticable impact but on slow spinning disk this
patch shows a ~32% performance improvement.

```
NVME SSD:

336ec0ac9a (master):

 Total docs/sec: 47200.9
 Total docs/sec: 46440.4

23543a97e3e7f72a31e26b50e00931919784426c (async wait for translog):

 Total docs/sec: 47461.6
 Total docs/sec: 46188.3
-------------------------------------------------------------------
Spinning disk:

336ec0ac9a (master):

 Total docs/sec: 22733.0
 Total docs/sec: 24129.8

23543a97e3e7f72a31e26b50e00931919784426c (async wait for translog):

 Total docs/sec: 32724.1
 Total docs/sec: 32845.4
--------------------------------------------------------------------
```
2016-08-27 21:42:38 +02:00
Igor Motov 3d6270b5cd Don't rebuild pipeline on every cluster state update
Currently, after at least one pipeline is registered it is getting rebuilt on every single cluster state update, even when this update is not related to ingest metadata. This change adds a check that the ingest metadata changed before trying to rebuild all pipelines.
2016-08-27 10:11:51 -04:00
Yannick Welsch 1b75cb63a2 Add recovery source to ShardRouting (#19516)
Adds an explicit recoverySource field to ShardRouting that characterizes the type of recovery to perform:

- fresh empty shard copy
- existing local shard copy
- recover from peer (primary)
- recover from snapshot
- recover from other local shards on same node (shrink index action)
2016-08-27 16:11:10 +02:00
qwerty4030 9172653211 Fix NPE during search with source filtering if the source is disabled. (#20093)
* Fix NPE during search with source filtering if the source is disabled.
Instead of throwing an NPE, a search response with source filtering will not contain the source if it is disabled in the mapping.

Closes #7758

* Created unit tests for FetchSourceSubPhase. Tests similar to SourceFetchingIT.
Removed SourceFetchingIT#testSourceDisabled (now covered via unit test FetchSourceSubPhaseTests#testSourceDisabled).

* Updated FetchSouceSubPhase unit tests per comments.
Renamed main unit test method.
Use assertEquals and assertNull instead of assertThat (less code).
2016-08-27 07:24:45 -04:00
Ali Beyad 230f0b514f Fixes test to use admin client to check the cluster state
instead of a random node's cluster service.
2016-08-27 01:29:29 -04:00