Commit Graph

25836 Commits

Author SHA1 Message Date
David Pilato 2511442a92 Merge pull request #22300 from dadoonet/doc/ingest-attachment
Adds more information about ingest attachment properties extraction
2016-12-23 11:30:19 +01:00
David Pilato e1b6166f21 Makes more obvious that we expect an array 2016-12-23 11:28:12 +01:00
David Pilato 08f556c133 Merge pull request #22310 from gameldar/patch-3
Add ingest-attachment-with-arrays section to ingest attachments doc
2016-12-23 10:33:23 +01:00
Adrien Grand 70594a66c7 Only run the unmapped+missing tests on 5.2+. 2016-12-23 09:38:20 +01:00
Boaz Leskes eb7450bcdc UnicastZenPing add trace logging on connection opening 2016-12-23 09:15:11 +01:00
Boaz Leskes 215874aff3 process TestLogging annotation value in prefix-first order
We have to sort the logger names so they wouldn't override each other. Processing org.elasticsearch:DEBUG after org.elasticsearch.transport:TRACE resets the setting of the later
2016-12-23 09:03:43 +01:00
gameldar b100f18505 Fix the ingest attachment array examples
Fix up the ingest attachment array handling example so they are full
examples and validated by the build system correctly.
2016-12-23 13:51:14 +08:00
Jason Tedor faaa671fb6 Enable assertions in integration tests
When starting a standalone cluster, we do not able assertions. This is
problematic because it means that we miss opportunities to catch
bugs. This commit enables assertions for standalone integration tests,
and fixes a couple bugs that were uncovered by enabling these.

Relates #22334
2016-12-22 20:08:02 -05:00
Tim B 9cd9576b84 Merge pull request #22332 from tbrooks8/add_clean_idea_build_task
Add cleanIdeaBuildDir gradle task that cleans the idea build directory
2016-12-22 13:25:09 -06:00
Nik Everett 55099df1cb Support negative numbers in writeVLong (#22314)
We don't *want* to use negative numbers with `writeVLong`
so throw an exception when we try. On the other
hand unforeseen bugs might cause us to write negative numbers (some versions of Elasticsearch don't have the exception, only an assertion)
so this fixes `readVLong` so that instead of reading a wrong
value and corrupting the stream it reads the negative value.
2016-12-22 13:02:39 -05:00
Boaz Leskes 13c5881f3e UnicastZenPing's PingingRound should prevent opening connections after being closed
This may cause them to leak. Provisioning for it was made in #22277 but sadly a crucial ensureOpen call was forgotten
2016-12-22 18:45:44 +01:00
Tim Brooks 864e3128e7 Add task to clean idea build directory. Make cleanIdea task invoke it. 2016-12-22 11:24:13 -06:00
Boaz Leskes 7d0dbd2082 add trace logging to UnicastZenPingTests.testResolveReuseExistingNodeConnections 2016-12-22 18:10:15 +01:00
Tal Levy 6d7261c4d3 Adds ingest processor headers to exception for unknown processor. (#22315)
Optimistically check for `tag` of an unknown processor for better tracking of which
processor declaration is to blame in an invalid configuration.

Closes #21429.
2016-12-22 08:24:00 -08:00
Nik Everett f5f2149ff2 Remove much ceremony from parsing client yaml test suites (#22311)
* Remove a checked exception, replacing it with `ParsingException`.
* Remove all Parser classes for the yaml sections, replacing them with static methods.
* Remove `ClientYamlTestFragmentParser`. Isn't used any more.
* Remove `ClientYamlTestSuiteParseContext`, replacing it with some static utility methods.

I did not rewrite the parsers using `ObjectParser` because I don't think it is worth it right now.
2016-12-22 11:00:34 -05:00
Stéphane Campinas e1b8528ab8 Support numeric bounds with decimal parts for long/integer/short/byte datatypes (#21972)
Close #21600
2016-12-22 15:20:15 +01:00
Martijn van Groningen b9a90eca20 inner hits: Don't inline inner hits if the query the inner hits is inlined into can't resolve mappings and ignore_unmapped has been set to true
Closes #21620
2016-12-22 14:51:33 +01:00
Colin Goodheart-Smithe 9a73a2efb3 Fix stackoverflow error on InternalNumericMetricAggregation 2016-12-22 13:50:54 +00:00
Adrien Grand 9b3b693d15 Date detection should not rely on a hardcoded set of characters. (#22171)
Currently we only apply date detection on strings that contain either `:`, `-`
or `/`. This commit inverses the heuristic in order to only apply date detection
on strings that are not parseable as a number, so that more date formats can be
used as dynamic dates formats.

Closes #1694
2016-12-22 14:35:59 +01:00
Adrien Grand e39942fc02 `value_type` is useful regardless of scripting. (#22160)
Today we only expose `value_type` in scriptable aggregations, however it is
also useful with unmapped fields. I suspect we never noticed because
`value_type` was not documented (fixed) and most aggregations are scriptable.

Closes #20163
2016-12-22 14:35:12 +01:00
Adrien Grand fd6e1a30de Improve concurrency of ShardCoreKeyMap. (#22316)
`ShardCoreKeyMap.add` is called on each segment for all search requests, which
means it might become a bottleneck under a cocurrent load of cheap search
requests since this method acquires a mutex. This change proposes to use a
`ConcurrentHashMap` which allows to only take the mutex in the case that the
`LeafReader` has never been seen before.
2016-12-22 14:34:08 +01:00
Martijn van Groningen acd64c6ee1 fixed jdocs and removed already fixed norelease 2016-12-22 14:18:30 +01:00
Gameldar e3eb363882 Link directly to the attachments in arrays section
The link should be made to the relevant section of the ingest attachments documentation, rather than the top of the page.
2016-12-22 20:52:08 +08:00
Colin Goodheart-Smithe 06576ed13b Adds abstract test classes for serialisation (#22281)
This adds test classes that can be used to test the wire serialisation and (optionally) the XContent serialisation of objects that implement Streamable/Writeable and ToXContent.

These test classes will enable classes sich as InternalAggregation (or at least its implementations) to be tested in a consistent way when is comes to testing serialisation.
2016-12-22 10:49:18 +00:00
Jason Tedor 7946396fe6 Introduce translog no-op
As the translog evolves towards a full operations log as part of the
sequence numbers push, there is a need for the translog to be able to
represent operations for which a sequence number was assigned, but the
operation did not mutate the index. Examples of how this can arise are
operations that fail after the sequence number is assigned, and gaps in
this history that arise when an operation is assigned a sequence number
but the operation never completed (e.g., a node crash). It is important
that these operations appear in the history so that they can be
replicated and replayed during recovery as otherwise the history will be
incomplete and local checkpoints will not be able to advance. This
commit introduces a no-op to the translog to set the stage for these
efforts.

Relates #22291
2016-12-21 23:08:16 -05:00
Jason Tedor 91cb563247 Provide helpful error message if a plugin exists
Today if an older version of a plugin exists, we fail to notify the user
with a helpful error message. This happens because during plugin
verification, we attempt to read the plugin descriptors for all existing
plugins. When an older version of a plugin is sitting on disk, we will
attempt to read this old plugin descriptor and fail due to a version
mismatch. This leads to an unhelpful error message. Instead, we should
check for existence of the plugin as part of the verification phase, but
before attempting to read plugin descriptors for existing plugins. This
enables us to provide a helpful error message to the user.

Relates #22305
2016-12-21 22:37:07 -05:00
Nik Everett 8aca504c86 Clear static variable after suite
This was causing test failures:
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+java9-periodic/1101/console
https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+dockeralpine-periodic/513/consoleFull
2016-12-21 13:39:28 -05:00
Francesc Gil dec6fc2d40 Repeated language analyzers (#22240)
* Repeated language analyzers

The `catalan` analyzer was repeated on the supported list :)

* Reordered the languages to have alphabetic order

* Added space for format

* Reordered the languages and removed repeated
2016-12-21 17:32:02 +01:00
gameldar d404ee3533 Add ingest-attachment-with-arrays section to ingest attachments doc
Added a new section detailing how to use the attachment processor
within an array.

This reverts commit #22296 and instead links to the foreach processor.
2016-12-22 00:18:33 +08:00
David Pilato 2adb310508 Merge pull request #22308 from nicpalmer/master
Support for eu-west-2 (London) cloud-aws plugin

See:

* http://docs.aws.amazon.com/general/latest/gr/rande.html#ec2_region
* http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
2016-12-21 16:57:42 +01:00
Luca Cavanna 5c8232c03b Restore deprecation warning for invalid match_mapping_type values (#22304)
The deprecation warning gives now the same message as 5.x. The deprecation warning was previously removed, but given that we are still lenient with old indices we should still output the warning.
2016-12-21 16:56:55 +01:00
Adrien Grand 84edf36f11 Make `-0` compare less than `+0` consistently. (#22173)
Our `float`/`double` fields generally assume that `-0` compares less than `+0`,
except when bounds are exclusive: an exclusive lower bound on `-0` excludes
`+0` and an exclusive upper bound on `+0` excludes `-0`.

Closes #22167
2016-12-21 16:51:45 +01:00
Jason Tedor 31bf279ec7 Tell Netty not to be unsafe in transport client
Today we ship with default jvm.options for server Elasticsearch that
prevents Netty from using some unsafe optimizations. Yet, the settings
do nothing for the transport client since it is embedded in other
applications that will not read and use those settings. This commit adds
these settings for the transport client, and is done so in a way that
still enables users to go unsafe if they want to go unsafe (they
shouldn't, but the option is there).

Relates #22284
2016-12-21 10:49:33 -05:00
Adrien Grand 3b3f9216db Allow terms aggregations on pure boolean scripts. (#22201)
The way aggregations on scripts work is by hiding scripts behind the same API
that we use for regular fields. However, there is no native support for boolean
fields, those need to be exposed as integers, with `0` standing for `false` and
`1` for true.

Relates #20941
2016-12-21 16:48:53 +01:00
Nic Palmer 3894ec9bae Fixed eu-west-2 entries for discovery-ec2 and repository-s3 also updated the asciidocs 2016-12-21 15:48:07 +00:00
Boaz Leskes 0e9186e137 Simplify Unicast Zen Ping (#22277)
The `UnicastZenPing` shows it's age and is the result of many small changes. The current state of affairs is confusing and is hard to reason about. This PR cleans it up (while following the same original intentions). Highlights of the changes are:

1) Clear 3 round flow - no interleaving of scheduling.
2) The previous implementation did a best effort attempt to wait for ongoing pings to be sent and completed. The pings were guaranteed to complete because each used the total ping duration as a timeout. This did make it hard to reason about the total ping duration and the flow of the code. All of this is removed now and ping should just complete within the given duration or not be counted (note that it was very handy for testing, but I move the needed sync logic to the test).
3) Because of (2) the pinging scheduling changed a bit, to give a chance for the last round to complete. We now ping at the beginning, 1/3 and 2/3 of the duration.
4) To offset for (3) a bit, incoming ping requests are now added to on going ping collections.
5) UnicastZenPing never establishes full blown connections (but does reuse them if there). Relates to #22120
6) Discovery host providers are only used once per pinging round. Closes #21739
7) Usage of the ability to open a connection without connecting to a node ( #22194 ) and shorter connection timeouts helps with connections piling up. Closes #19370
8) Beefed up testing and sped them up.
9) removed light profile from production code
2016-12-21 15:09:58 +01:00
Nik Everett 567c65b0d5 Replace IndicesQueriesRegistry (#22289)
* Switch query parsing to namedObject
* Remove IndicesQueriesRegistry
2016-12-21 09:05:14 -05:00
Rui Hu 80f8dfe852 Fixed document mistake and fit for 5.1.1 API 2016-12-21 08:18:16 -05:00
Nic Palmer 2653208147 Merge remote-tracking branch 'elastic/master' 2016-12-21 13:10:50 +00:00
Nic Palmer 8847c34093 Push for eu-west-2 issue 2016-12-21 13:10:33 +00:00
javanna 7141f6b554 [TEST] improve error message in ESTestCase#assertWarnings 2016-12-21 13:31:02 +01:00
javanna 55feae1e9a [TEST] remove deleted test classes from checkstyle suppressions
Relates to #21915
2016-12-21 13:28:43 +01:00
David Pilato 6d96cdb87c Add // CONSOLE to the snippet 2016-12-21 12:46:01 +01:00
David Pilato 80843afb19 Adds more information about ingest attachment properties extraction
This is coming from thsi thread on discuss: https://discuss.elastic.co/t/ingest-attachment-plugin-exception/69167/10
2016-12-21 12:13:16 +01:00
Luca Cavanna ae01a51b44 [TEST] make ESSingleNodeTestCase tests repeatable (#22283)
If we conditionally do random things, e.g. initialize a node only after the first test, we have to make sure that we unconditionally create a new seed calling random.nextLong(), then initialize the node under a private randomness context. This makes sure that any random usage through Randomness.get() will retrieve the proper random instance through RandomizedContext.current().getRandom(). When running under private randomness, the context will return the Random instance that was created with the provided seed (forked from the main random instance) rather than the main Random that's exposed to tests as well. Otherwise tests become non repeatable because that initialization part happens only before the first executed test.
2016-12-21 11:44:24 +01:00
David Pilato 345125237d Merge pull request #22296 from gameldar/patch-2
Add link to foreach processor to ingest-attachment.asciidoc
2016-12-21 11:41:28 +01:00
David Pilato 2ec5deef08 Link for setting page in elasticsearch.yml is outdated
Closes #22299.
2016-12-21 11:28:07 +01:00
Christoph Büscher bdecbb529f Factor out sort values from InternalSearchHit (#22080)
This adds fromXContent method and unit test for sort values that are part of 
InternalSearchHit. In order to centralize serialisation and xContent parsing and
rendering code, move all relevant parts to a new class which can be unit tested
much better in isolation.This is part of the preparation for parsing search
responses on the client side.
2016-12-21 11:19:47 +01:00
Clinton Gormley 9b22ec1c6a Add ID for percolate query to Java API docs 2016-12-21 10:54:27 +01:00
Boaz Leskes 6249f1092f x_refresh.yaml tests should use unique index names and doc ids to ease debugging
This is to make it easier to grep the node logs
2016-12-21 10:25:33 +01:00