Commit Graph

29634 Commits

Author SHA1 Message Date
Andrew Banchich 3b58582059 [Docs] Grammatical fix in important-settings.asciidoc (#27654) 2017-12-19 17:08:08 +01:00
kel 192b263e31 Make AbstractQueryBuilder.declareStandardFields to be protected (#27894) 2017-12-19 16:34:08 +01:00
Nik Everett 32669ca265
Test: Change randomValueOtherThan(null, supplier) (#27901)
When the first parameter of `ESTestCase#randomValueOtherThan` is `null`
then run the supplier until it returns non-`null`. Previously,
`randomValueOtherThan` just ran the supplier one time which was
confusing.

Unexpectedly, it looks like not tests rely on the original `null`
handling.

Closes #27821
2017-12-19 10:23:38 -05:00
David Turner b1039164a1 Mute MigrationDocumentationIT#testClusterHealth()
Relates #27754
2017-12-19 12:16:49 +00:00
David Turner b26cc36928 Mute testRetentionPolicyChangeDuringRecovery
Relates #27861.
2017-12-19 12:05:45 +00:00
Albert Zaharovits 01a47baa10
Retain originalIndex info when rewriting FieldCapabilities requests (#27761)
A FieldCapabilities request can cover multiple indices (or aliases pointing to multiple indices).
When rewriting the request for each index, store the original requested indices.
2017-12-19 13:38:41 +02:00
David Turner 1383cab267 Add @AwaitsFix for #27890 2017-12-19 08:41:42 +00:00
Boaz Leskes bea9471b2f
Use port 0 InternalTestCluster nodes (#27859)
We currently have a complicated port assignment scheme to make sure that the nodes span off by the internal test cluster will be assigned fixed port ranges that will also not collide between clusters. The port ranges need to be fixed in advance so that the nodes will be able to find each other via `UnicastZenPing`.

This approach worked well for the last few years but we are now at a point that our testing has grown beyond it and we exceed the 5 reusable ranges per JVM. This means that nodes are not always assigned the first 5 ports in their range which causes cluster formation issues. On top of that, most of the clusters that are span up don't even rely on `UnicastZenPing` but rather `MockZenPings` that uses in memory maps for discovery (with the down side that they are not influenced by network disruption simulations).

This PR changes `InternalTestCluster` to use port 0 as a fixed assignment. This will allow the OS to manage ports and will ensure we don't have collisions. For tests that need to simulate network disruptions (and thus can't use `MockZenPings`), a new `UnicastHostProvider` is introduced that is based on the current state of the test cluster. Since that is only resolved at run time, it is aware of the port assignments of the OS.

Closes #27818
Closes #27762
2017-12-19 08:43:03 +01:00
Jason Tedor aebdb2a646 Filter current version from compatible versions
We need to filter the current version from the list of compatible
versions to match how we calculate the list of compatible versions in
Gradle.
2017-12-18 17:37:22 -05:00
Alexander Kazakov d9a0b50893 Using DocValueFormat::parseBytesRef for parsing missing value parameter (#27855) 2017-12-18 20:50:58 +01:00
Jason Tedor 054711dd88 More refinement of version is compatible test
This test further refines the version is compatible test, adding
additional handling of edge cases.
2017-12-18 13:20:22 -05:00
Nhat Nguyen 0d99fadd2a Put back lastSyncedGlobalCheckpoint in deletion policy
The PR #27837 unintentionally changed to an in memory global checkpoint.
2017-12-18 11:54:50 -05:00
Jason Tedor fa1159a854 Tighten version compatibility test
This commit tightens the version compatibility test, generalizing it
beyond a hard-coded 7.0.0.
2017-12-18 10:40:06 -05:00
Yannick Welsch a5e8a221ec
Move GlobalCheckpointTracker and remove SequenceNumbersService (#27837)
This commit moves GlobalCheckpointTracker from the engine to IndexShard, where it better fits logically: Tracking the global checkpoint based on the local checkpoints of all shards in the replication group is not a property of the engine, but rather a property fulfilled by the current primary shard. The LocalCheckpointTracker on the other hand is driven by the contents of the local translog. By moving GlobalCheckpointTracker to IndexShard, it makes little sense to keep the SequenceNumbersService class around - it would only wrap the LocalCheckpointTracker. This commit therefore removes the class and replaces occurrences of SequenceNumbersService in the engine directly by LocalCheckpointTracker.
2017-12-18 15:27:44 +01:00
Alan Woodward af3f63616b
Allow TrimFilter to be used in custom normalizers (#27758)
AnalysisFactoryTestCase checks that the ES custom token filter multi-term
awareness matches the underlying lucene factory.  For the trim filter this
won't be the case until LUCENE-8093 is released in 7.3, so we add a
temporary exclusion

Closes #27310
2017-12-18 14:27:03 +00:00
Jason Tedor 76771242e8 Fix version tests for release tests
This commit fixes the version tests for release tests. The problem here
is that during release tests all version should be treated as released
so the assertions must be modified accordingly.

Relates #27815
2017-12-18 08:51:37 -05:00
Boaz Leskes 9cd69e7ec1
recovery from snapshot should fill gaps (#27850)
When snapshotting the primary we capture a lucene commit at an arbitrary moment from a sequence number perspective. This means that it is possible that the commit misses operations and that there is a gap between the local checkpoint in the commit and the maximum sequence number.

When we restore, this will create a primary that "misses" operations and currently will mean that the sequence number system is stuck (i.e., the local checkpoint will be stuck). To fix this we should fill in gaps when we restore, in a similar fashion to normal store recovery.
2017-12-18 13:33:39 +01:00
kel 26fc717ddd Remove unused class PreBuiltTokenFilters (#27839) 2017-12-18 11:48:38 +01:00
David Turner f0b21e3182
Make randomNonNegativeLong() draw from a uniform distribution (#27856)
Currently randomNonNegativeLong() returns 0 half as often as any positive long,
but random number generators are typically expected to return
uniformly-distributed values unless otherwise specified. This fixes this issue
by mapping Long.MIN_VALUE directly onto 0 rather than resampling.
2017-12-18 09:57:40 +00:00
kel 7a27a2770b Reject scroll query if size is 0 (#22552) (#27842) 2017-12-18 10:38:41 +01:00
Jim Ferenczi 55b71a871b Adapt rest test after backport. Relates #27833 2017-12-18 10:36:44 +01:00
David Turner e6da564eb1
Handle case where the hole vertex is south of the containing polygon(s) (#27685)
Normally the hole is assigned to the component of the first edge to the south
of one of its vertices, but if the chosen hole vertex is south of everything
then the binary search returns -1 yielding an ArrayIndexOutOfBoundsException.
Instead, assign the vertex to the component of the first edge to its north.
Subsequent validation catches the fact that the hole is outside its component.

Fixes #25933
2017-12-18 08:50:40 +00:00
Jason Tedor 75c0cd0672
Move range field mapper back to core
This commit moves the range field mapper back to core so that we can
remove the compile-time dependency of percolator on mapper-extras which
compilcates dependency management for the percolator client JAR, and
modules should not be intertwined like this anyway.

Relates #27854
2017-12-17 14:27:10 -05:00
Jason Tedor fade828c50
Fix publication of elasticsearch-cli to Maven
This commit addresses the publication of the elasticsearch-cli to
Maven. For now for simplicity we publish this to Maven so that it is
available as a transitive dependency for any artifacts that depend on
the core elasticsearch artifact. It is possible that in the future we
can simply exclude this dependency but for now this is the safest and
simplest approach that can happen in a patch release.

Relates #27853
2017-12-17 11:51:18 -05:00
Nhat Nguyen 4f62b51c87
Use lastSyncedGlobalCheckpoint in deletion policy (#27826)
Today we use the in-memory global checkpoint from SequenceNumbersService
to clean up unneeded commit points, however the latest global checkpoint
may haven't fsynced to the disk yet. If the translog checkpoint fsync
failed and we already use a higher global checkpoint to clean up commit
points, then we may have removed a safe commit which we try to keep for
recovery.

This commit updates the deletion policy using lastSyncedGlobalCheckpoint
from Translog rather the in memory global checkpoint.

Relates #27606
2017-12-16 11:03:31 -05:00
Tal Levy 43ff38c5da
update ingest-attachment to use Tika 1.17 and newer deps (#27824)
- this pr updates tika and its dependencies
- updates the SHAs
- updates the class excludes
2017-12-15 13:47:26 -08:00
Jay Modi 717e2ddf42
Do not use system properties when building the HttpAsyncClient (#27829)
This commit removes the usage of system properties for the HttpAsyncClient as this overrides some
defaults that we intentionally change. In order to set the default SSLContext to the system context
we set the SSLContext on the builder explicitly.

Closes #27827
2017-12-15 12:54:22 -07:00
jaymode 541e142a6d
Revert "Increase logging on qa:mixed-cluster tests"
This reverts commit e04e5ab037 as we no longer need the increased
logging for the mixed cluster tests. This will reduce the size of logs for some build failures.
2017-12-15 10:44:35 -07:00
Tim Brooks 916e7dbe29
Add NioGroup for use in different transports (#27737)
This commit is related to #27260. It adds a base NioGroup for use in
different transports. This class creates and starts the underlying
selectors. Different protocols or transports are established by passing
the ChannelFactory to the bindServerChannel or openChannel
methods. This allows a TcpChannelFactory to be passed which will
create and register channels that support the elasticsearch tcp binary
protocol or a channel factory that will create http channels (or other).
2017-12-15 10:42:00 -07:00
kel f5e0932c8d Add version support for inner hits in field collapsing (#27822) (#27833)
Add version support for inner hits in field collapsing
2017-12-15 18:00:40 +01:00
sandstrom e542c03bea Clarify that number of threads is set by packages
This commit clarifies that on systemd the packages already set the number of
threads for the Elasticsearch process.

Relates #27840
2017-12-15 11:29:13 -05:00
Jason Tedor 7945848dd6 Register HTTP read timeout setting
This commit registers the HTTP read timeout setting so that it can
actually be set.
2017-12-15 10:56:00 -05:00
Simon Willnauer 481d98b8d5
Remove `operationThreaded` from Java API (#27836)
This option is completely unused. Some places set it but we never read
the value neither respect it.
2017-12-15 15:20:55 +01:00
Colin Goodheart-Smithe c93cc1bb8f
Fix ByteSizeValue serialisation test 2017-12-15 12:10:10 +00:00
Simon Willnauer d941c64edb
Optimize version map for append-only indexing (#27752)
Today we still maintain a version map even if we only index append-only
or in other words, documents with auto-generated IDs. We can instead maintain
an un-safe version map that will be swapped to a safe version map only if necessary
once we see the first document that requires access to the version map. For instance:
 * a auto-generated id retry
 * any kind of deletes
 * a document with a foreign ID (non-autogenerated

In these cases we forcefully refresh then internal reader and start maintaining
a version map until such a safe map wasn't necessary for two refresh cycles.
Indices / shards that never see an autogenerated ID document will always meintain a version
map and in the case of a delete / retry in a pure append-only index the version map will be
de-optimized for a short amount of time until we know it's safe again to swap back. This
will also minimize the requried refeshes.

Closes #19813
2017-12-15 12:13:10 +01:00
Martijn van Groningen cef7bd2079
docs: add best practises for wildcard queries inside percolator queries 2017-12-15 10:49:59 +01:00
Christoph Büscher f3293879b5 [Docs] Improve rendering of ranking evaluation docs 2017-12-15 10:45:44 +01:00
Alex Benusovich 5f74e81f29 [Docs] Clarify version for restoring old indices. (#27800)
Sentence should say that an index created in 2.x cannot be restored in a 6.x cluster even if 5.x cluster created the snapshot.
2017-12-15 09:42:36 +01:00
Simon Willnauer 1e5d3787e5 [TEST] Don't start thread before checking for pending refresh
If we start the thread too early it registers a refresh listener and
that causes out assertion to fail if there is a zero timeout.
Closes #27769
2017-12-15 09:28:50 +01:00
Christoph Büscher 54b1fed5b3 Corrected ByteSizeValue bwc serialization version after backport to 6.x 2017-12-15 08:56:59 +01:00
Martijn van Groningen e9160fc014
percolator: also extract match_all queries
I've seen several cases where match_all queries were being used inside percolator queries,
because these queries were created generated by other systems.

Extracting these queries will allow the percolator at query time in a filter context
to skip over these queries without parsing or validating that these queries actually
match with the document being percolated.
2017-12-15 08:50:29 +01:00
Alex Crome af8bd8bbcf Fix custom service names when installing on windows
We document that users can set custom service names on Windows. Alas,
the functionality does not work. This commit fixes the issue by passing
the environment variable SERVICE_ID as the service name otherwise
defaulting to elasticsearch-service-x64.

Relates #25255
2017-12-14 20:41:39 -05:00
Tim Brooks f33f9612a7
Remove potential nio selector leak (#27825)
When an ESSelector is created an underlying nio selector is opened. This
selector is closed by the event loop after close has been signalled by
another thread.

However, there is a possibility that an ESSelector is created and some
exception in the startup process prevents it from ever being started
(however, close will still be called). The allows the selector to leak.

This commit addresses this issue by having the signalling thread close
the selector if the event loop is not running when close is signalled.
2017-12-14 14:37:41 -07:00
Christoph Büscher c541a0c60e Add skip versions for rank_eval yaml tests 2017-12-14 22:18:37 +01:00
Jack Conradson 1de927c80d
Painless: Clean Up Painless Cast Object (#27794)
Added static methods to make creating Painless casts obvious as to what is
being boxed/unboxed.
2017-12-14 09:08:10 -08:00
Adrien Grand 1b660821a2
Allow `_doc` as a type. (#27816)
Allowing `_doc` as a type will enable users to make the transition to 7.0
smoother since the index APIs will be `PUT index/_doc/id` and `POST index/_doc`.
This also moves most of the documentation to `_doc` as a type name.

Closes #27750
Closes #27751
2017-12-14 17:47:53 +01:00
Christoph Büscher bb14b8f7c5 Merge branch 'rankeval'
This commit adds a new module that provides an endpoint that can be used to
evaluate search ranking results.

Closes #19195
2017-12-14 16:45:03 +01:00
Jason Tedor 10d2667498
Clarify using supported LTS versions of the Java
This commit clarifies that we recommended using supported LTS versions
of Java as opposed to supporting a minimum version and any version above
that.

Relates #27795
2017-12-14 07:47:50 -05:00
Colin Goodheart-Smithe 579d1fea57
Fixes ByteSizeValue to serialise correctly (#27702)
* Fixes ByteSizeValue to serialise correctly

This fix makes a few fixes to ByteSizeValue to make it possible to perform round-trip serialisation:
* Changes wire serialisation to use Zlong methods instead of VLong methods. This is needed because the value `-1` is accepted but previously if `-1` is supplied it cannot be serialised using the wire protocol.
* Limits the supplied size to be no more than Long.MAX_VALUE when converted to bytes. Previously values greater than Long.MAX_VALUE bytes were accepted but would be silently interpreted as Long.MAX_VALUE bytes rather than erroring so the user had no idea the value was not being used the way they had intended. I consider this a bug and so fine to include this bug fix in a minor version but I am open to other points of view.
* Adds a `getStringRep()` method that can be used when serialising the value to JSON. This will print the bytes value if the size is positive, `”0”` if the size is `0` and `”-1”` if the size is `-1`.
* Adds logic to detect fractional values when parsing from a String and emits a deprecation warning in this case.
* Modifies hashCode and equals methods to work with long values rather than doubles so they don’t run into precision problems when dealing with large values. Previous to this change the equals method would not detect small differences in the values (e.g. 1-1000 bytes ranges) if the actual values where very large (e.g. PBs). This was due to the values being in the order of 10^18 but doubles only maintaining a precision of ~10^15.

Closes #27568

* Fix bytes settings default value to not use fractional values

* Fixes test

* Addresses review comments

* Modifies parsing to preserve unit

This should be bwc since in the case that the input is fractional it reverts back to the old method of parsing it to the bytes value.

* Addresses more review comments

* Fixes tests

* Temporarily changes version check to 7.0.0

This will be changed to 6.2 when the fix has been backported
2017-12-14 12:17:17 +00:00
Daniel Mitterdorfer d26b33dea2 Mute VersionUtilsTest#testGradleVersionsMatchVersionUtils
Relates #27815
2017-12-14 12:33:41 +01:00