Commit Graph

186 Commits

Author SHA1 Message Date
Jim Ferenczi 7a7658707a
Upgrade to Lucene release 8.0.0 (#39998)
This commit upgrades to the GA release of Lucene 8

Closes #39640
2019-03-13 18:11:50 +01:00
Alan Woodward 71b8494181
Upgrade to lucene 8.0.0-snapshot-ff9509a8df (#39444)
Backport of #39350

Contains the following:

* LUCENE-8635: Move terms dictionary off-heap for non-primary-key fields in `MMapDirectory`
* LUCENE-8292: `TermsEnum` is fully abstract
* LUCENE-8679: Return WITHIN in `EdgeTree#relateTriangle` only when polygon and triangle share one edge
* LUCENE-8676: Nori tokenizer deals correctly with large buffers
* LUCENE-8697: `GraphTokenStreamFiniteStrings` better handles side paths with gaps
* LUCENE-8664: Add `equals` and `hashCode` to `TotalHits`
* LUCENE-8660: `TopDocsCollector` returns accurate hit counts if the total equals the threshold
* LUCENE-8654: `Polygon2D#relateTriangle` fix for when the polygon is inside the triangle
* LUCENE-8645: `Intervals#fixField` can merge intervals from different fields
* LUCENE-8585: Create jump-tables for DocValues at index time
2019-02-27 14:36:08 +00:00
Julie Tibshirani c2e9d13ebd
Default include_type_name to false in the yml test harness. (#38058)
This PR removes the temporary change we made to the yml test harness in #37285
to automatically set `include_type_name` to `true` in index creation requests
if it's not already specified. This is possible now that the vast majority of
index creation requests were updated to be typeless in #37611. A few additional
tests also needed updating here.

Additionally, this PR updates the test harness to set `include_type_name` to
`false` in index creation requests when communicating with 6.x nodes. This
mirrors the logic added in #37611 to allow for typeless document write requests
in test set-up code. With this update in place, we can remove many references
to `include_type_name: false` from the yml tests.
2019-02-01 11:44:13 -08:00
Colin Goodheart-Smithe 21e392e95e
Removes typed calls from YAML REST tests (#37611)
This PR attempts to remove all typed calls from our YAML REST tests. The PR adds include_type_name: false to create index requests that use a mapping and also to put mapping requests. It also removes _type from index requests where they haven't already been removed. The PR ignores tests named *_with_types.yml since this are specifically testing typed API behaviour.

The change also includes changing the test harness to add the type _doc to index, update, get and bulk requests that do not specify the document type when the test is running against a mixed 7.x/6.x cluster.
2019-01-30 16:32:58 +00:00
Adrien Grand e9fcb25a28
Upgrade to lucene-8.0.0-snapshot-83f9835. (#37668)
This snapshot uses a new file format for doc-values which is expected to make
advance/advanceExact perform faster on sparse fields:
https://issues.apache.org/jira/browse/LUCENE-8585
2019-01-22 11:44:29 +01:00
Nick Knize b2aa655f46
Upgrade master to lucene-8.0.0-snapshot-a1c6e642aa (#37091)
Updates the master branch to the latest snapshot of Lucene 8.0.
2019-01-02 20:18:19 -06:00
Alan Woodward c7ac9ef826
Upgrade to lucene snapshot 774e9aefbc (#36637)
Includes LUCENE-8607: improvement to MatchAllDocsQuery
2018-12-14 20:30:07 +00:00
Alan Woodward 9ac7359643
Update lucene to snapshot-7e4555a2fd (#36563)
Includes the following:

* Reversion of doc-values changes in LUCENE-8374; we are interested in seeing if this 
  has an effect on benchmarks for node-stats and index-stats
* More improvements to docvalues updates
2018-12-12 20:18:32 +00:00
Nhat Nguyen 3fb5a12b30 Upgrade to Lucene-8.0.0-snapshot-61e448666d (#36518)
Includes:
- LUCENE-8602: Share TermsEnum if possible while applying DV updates
2018-12-12 06:47:40 +01:00
Nhat Nguyen 2a7edca59f
Upgrade to Lucene-8.0.0-snapshot-ef61b547b1 (#36450)
Includes:

- LUCENE-8598: Improve field updates packed values
- LUCENE-8599: Use sparse bitset to store docs in SingleValueDocValuesFieldUpdates
2018-12-10 16:33:49 -05:00
Nhat Nguyen 10feb75eb7
Upgrade to Lucene-8.0.0-snapshot-aaa64d70159 (#36335)
Includes:

LUCENE-8594: DV update are broken for updates on new field
LUCENE-8590: Optimize DocValues update datastructures
LUCENE-8593: Specialize single value numeric DV updates

Relates #36286
2018-12-06 20:33:25 -05:00
Jim Ferenczi 18866c4c0b
Make hits.total an object in the search response (#35849)
This commit changes the format of the `hits.total` in the search response to be an object with
a `value` and a `relation`. The `value` indicates the number of hits that match the query and the
`relation` indicates whether the number is accurate (in which case the relation is equals to `eq`)
or a lower bound of the total (in which case it is equals to `gte`).
This change also adds a parameter called `rest_total_hits_as_int` that can be used in the
search APIs to opt out from this change (retrieve the total hits as a number in the rest response).
Note that currently all search responses are accurate (`track_total_hits: true`) or they don't contain
`hits.total` (`track_total_hits: true`). We'll add a way to get a lower bound of the total hits in a
follow up (to allow numbers to be passed to `track_total_hits`).

Relates #33028
2018-12-05 19:49:06 +01:00
Alan Woodward 73ceaad03a
Update to lucene-8.0.0-snapshot-c78429a554 (#36212)
Includes:

* A fix for a bug in Intervals.or() (https://issues.apache.org/jira/browse/LUCENE-8586)
* The ability to disable offset mangling in WordDelimiterGraphFilter
        (https://issues.apache.org/jira/browse/LUCENE-8509)
* BM25Similarity no longer multiplies scores by k1 + 1
2018-12-05 12:43:56 +00:00
Jim Ferenczi e37a0ef844
Upgrade to lucene-8.0.0-snapshot-67cdd21996 (#35816) 2018-11-22 15:42:59 +01:00
Jack Conradson d62bbca56d
[Scripting] Use Number as a return value for BucketAggregationScript (#35653)
This change fixes #35351. Users were no longer able to return types of numbers other than doubles for bucket aggregation scripts. This change reverts to the previous behavior of being able to return any type of number and having it converted to a double outside of the script.
2018-11-16 12:33:01 -08:00
Nick Knize 2591f66a33
upgrade to lucene-8.0.0-snapshot-6d9c714052 (#35428) 2018-11-12 10:48:27 -06:00
Nick Knize a5e1f4d3a2 Upgrade to lucene-8.0.0-snapshot-31d7dfe6b1 (#35224) 2018-11-06 11:55:23 +01:00
Nik Everett e28509fbfe
Core: Less settings to AbstractComponent (#35140)
Stop passing `Settings` to `AbstractComponent`'s ctor. This allows us to
stop passing around `Settings` in a *ton* of places. While this change
touches many files, it touches them all in fairly small, mechanical
ways, doing a few things per file:
1. Drop the `super(settings);` line on everything that extends
`AbstractComponent`.
2. Drop the `settings` argument to the ctor if it is no longer used.
3. If the file doesn't use `logger` then drop `extends
AbstractComponent` from it.
4. Clean up all compilation failure caused by the `settings` removal
and drop any now unused `settings` isntances and method arguments.

I've intentionally *not* removed the `settings` argument from a few
files:
1. TransportAction
2. AbstractLifecycleComponent
3. BaseRestHandler

These files don't *need* `settings` either, but this change is large
enough as is.

Relates to #34488
2018-10-31 21:23:20 -04:00
Alpar Torok 59536966c2
Add a new "contains" feature (#34738)
The contains syntax was added in #30874 but the skips were not properly
put in place.
The java runner has the feature so the tests will run as part of the
build, but language clients will be able to support it at their own
pace.
2018-10-25 08:50:50 +03:00
Ryan Ernst 687dc1eb11
Scripting: Remove SearchScript (#34730)
This commit removes the last non context based script class.
2018-10-24 15:03:38 -07:00
Zachary Tong 299d044bfc
Collapse pipeline aggs into single package (#34658)
- Restrict visibility of Aggregators and Factories
- Move PipelineAggregatorBuilders up a level so it is consistent with
AggregatorBuilders
- Checkstyle line length fixes for a few classes
- Minor odds/ends (swapping to method references, formatting, etc)
2018-10-23 16:01:01 -04:00
Ryan Ernst 222652dfce
Scripting: Convert script fields to use script context (#34164)
This commit removes the use of SearchScript for script fields and adds
a new FieldScript.
2018-10-20 16:33:49 -07:00
Nik Everett 1452d55155
Expressions: Wrap at 140 columns (#34493)
Apply our standard line length to all of the java files in the
`lang-expression` plugin.
2018-10-17 16:12:21 -04:00
Armin Braun 3954d041a0
SCRIPTING: Move sort Context to its Own Class (#33717)
* SCRIPTING: Move sort Context to its own Class
2018-10-17 10:02:44 +01:00
Armin Braun ebca27371c
SCRIPTING: Move Aggregation Script Context to its own class (#33820)
* SCRIPTING: Move Aggregation Script Context to its own class
2018-10-15 17:28:05 +01:00
Armin Braun 511526250b
MINOR: Remove Deadcode in ExpressionTermSetQuery (#34442) 2018-10-15 17:26:08 +01:00
eray daf88335d7 Add max_children limit to nested sort (#33587)
Add an option to `nested` sort to limit the number of children to visit when picking the sort value
of the root document. 

Closes #33592
2018-10-05 12:02:47 +02:00
Armin Braun 3ccfc3de58
SCRIPTING: Terms set query expression (#33856)
* SCRIPTING: Add Expr. Compile for TermSetQuery Ctx.

* Follow up to #33602 adding the ability to compile TermsSetQuery
scripts with the expressions engine in the same way we support
SearchScript in Expressions
   * Duplicated the code here for now to make the change less complex,
 the only difference to SearchScript is that `_score` and `_value` are not handled for TermsSetQuery
* remove redundant check
2018-10-04 16:03:57 +02:00
Ryan Ernst 47cbae9b26
Scripting: Remove ExecutableScript (#34154)
This commit removes the legacy ExecutableScript, which was no longer
used except in tests. All uses have previously been converted to script
contexts.
2018-09-28 17:13:08 -07:00
Armin Braun c6462057a1
MINOR: Remove Some Dead Code in Scripting (#33800)
* The is default check method is not used in ScriptType
* The removed vars on ExpressionSearchScript are unused
2018-09-18 20:43:31 +02:00
Jim Ferenczi 241c74efb2
upgrade to a new snapshot of Lucene 8 (7d0a7782fa) (#33812) 2018-09-18 18:16:40 +02:00
Armin Braun 615f494c77
MINOR: Drop Redundant Ctx. Check in ScriptService (#33782)
* MINOR: Drop Redundant Ctx. Check in ScriptService

* This check is completely redundant, the expression script
engine will throw anyway (and with a similar message) for
those contexts that it cannot compile. Moreover, the update context
is not the only context that is not suported by the expression engine
at this point so handling the update context separately here makes
no sense.
2018-09-18 07:25:22 +02:00
Alan Woodward 39c3234c2f
Upgrade to latest Lucene snapshot (#33505)
* LeafCollector.setScorer() now takes a Scorable
* Scorers may not have null Weights
* IndexWriter.getFlushingBytes() reports how much memory is being used by IW threads writing to disk
2018-09-10 20:51:55 +01:00
Jim Ferenczi 79cd6385fe
Collapse package structure for metrics aggs (#33463)
This change collapses all metrics aggregations classes into a single package `org.elasticsearch.aggregations.metrics`.
It also restricts the visibility of some classes (aggregators and factories) that should not be used outside of the package.

Relates #22868
2018-09-07 10:58:06 +02:00
Jim Ferenczi 7ad71f906a
Upgrade to a Lucene 8 snapshot (#33310)
The main benefit of the upgrade for users is the search optimization for top scored documents when the total hit count is not needed. However this optimization is not activated in this change, there is another issue opened to discuss how it should be integrated smoothly.
Some comments about the change:
* Tests that can produce negative scores have been adapted but we need to forbid them completely: #33309

Closes #32899
2018-09-06 14:42:06 +02:00
Armin Braun d412230cda
SCRIPTING: Support BucketAggScript return null (#32811)
* As explained in #32790, `BucketAggregationScript` must support `null` as a return value
* Closes #32790
2018-08-13 20:08:26 +02:00
Nicholas Knize e162127ff3 Upgrade to Lucene-7.5.0-snapshot-13b9e28f9d
The main feature is the inclusion of bkd backed geo_shape with
INTERSECT, DISJOINT, WITHIN bounding box and polygon query support.
2018-08-09 11:15:02 -05:00
Armin Braun 6fa7016bbf
SCRIPTING: Move Aggregation Scripts to their own context (#32068)
* SCRIPTING: Move Aggregation Scripts to their own context
2018-08-04 10:37:07 +02:00
Jim Ferenczi 53ff06e621
Upgrade to Lucene-7.5.0-snapshot-608f0277b0 (#32390)
The main highlight is the removal of the reclaim_deletes_weight in the TieredMergePolicy.
The es setting index.merge.policy.reclaim_deletes_weight is deprecated in this commit and the value is ignored. The new merge policy setting setDeletesPctAllowed should be added in a follow up.
2018-07-27 08:28:51 +02:00
Julie Tibshirani 15ff3da653
Add support for field aliases. (#32172)
* Add basic support for field aliases in index mappings. (#31287)
* Allow for aliases when fetching stored fields. (#31411)
* Add tests around accessing field aliases in scripts. (#31417)
* Add documentation around field aliases. (#31538)
* Add validation for field alias mappings. (#31518)
* Return both concrete fields and aliases in DocumentFieldMappers#getMapper. (#31671)
* Make sure that field-level security is enforced when using field aliases. (#31807)
* Add more comprehensive tests for field aliases in queries + aggregations. (#31565)
* Remove the deprecated method DocumentFieldMappers#getFieldMapper. (#32148)
2018-07-18 09:33:09 -07:00
Alpar Torok 08b8d11e30
Add support for switching distribution for all integration tests (#30874)
* remove left-over comment

* make sure of the property for plugins

* skip installing modules if these exist in the distribution

* Log the distrbution being ran

* Don't allow running with integ-tests-zip passed externally

* top level x-pack/qa can't run with oss distro

* Add support for matching objects in lists

Makes it possible to have a key that points to a list and assert that a
certain object is present in the list. All keys have to be present and
values have to match. The objects in the source list may have additional
fields.

example:
```
  match:  { 'nodes.$master.plugins': { name: ingest-attachment }  }
```

* Update plugin and module tests to work with other distributions

Some of the tests expected that the integration tests will always be ran
with  the `integ-test-zip` distribution so that there will be no other
plugins loaded.

With this change, we check for the presence of the plugin without
assuming exclusivity.

* Allow modules to run on other distros as well

To match the behavior of tets.distributions

* Add and use a new `contains` assertion

Replaces the  previus changes that caused `match` to do a partial match.

* Implement PR review comments
2018-06-26 06:49:03 -07:00
Adrien Grand f023e95ae0
Upgrade to Lucene 7.4.0. (#31529)
This moves Elasticsearch from a recent 7.4.0 snapshot to the GA release.
2018-06-22 16:17:17 +02:00
Nhat Nguyen 8453ca638d
Upgrade to Lucene-7.4.0-snapshot-518d303506 (#31360) 2018-06-15 10:58:21 -04:00
Adrien Grand cc55235030
Decouple MultiValueMode. (#31075)
Currently this class takes care of moth selecting the relevant value, and
replacing missing values if any. This is fine for sorting, which always needs
to do both at the same time, but we also have a number of aggregations and
script utils that need to retain information about missing values so this change
proposes to decouple selection of the relevant value and replacement of missing
values.
2018-06-05 08:51:20 +02:00
Nhat Nguyen abe61159a8
Upgrade to Lucene-7.4.0-snapshot-0a7c3f462f (#31073)
This snapshot includes:
- LUCENE-8341: Record soft deletes in SegmentCommitInfo which will resolve #30851
- LUCENE-8335: Enforce soft-deletes field up-front
2018-06-04 14:18:46 -04:00
Nhat Nguyen 363f1e84ca
Upgrade to Lucene-7.4-snapshot-1cbadda4d3 (#30928)
This snapshot includes LUCENE-8328 which is needed to stabilize CCR builds.
2018-05-29 12:29:52 -04:00
Martijn van Groningen ae2f021f1c
Move score script context from SearchScript to its own class (#30816) 2018-05-25 07:17:50 +02:00
Nhat Nguyen 1918a30237
Upgrade to Lucene-7.4.0-snapshot-cc2ee23050 (#30778)
The new snapshot includes LUCENE-8324 which fixes missing checkpoint
after a fully deletes segment is dropped on flush. This snapshot should
resolves failed tests in the CorruptedFileIT suite.

Closes #30741
Closes #30577
2018-05-22 13:11:48 -04:00
Nhat Nguyen 67d8fc222d
Upgrade to Lucene-7.4.0-snapshot-59f2b7aec2 (#30726)
This snapshot resolves issues related to ShrinkIndexIT.
2018-05-18 18:21:39 -04:00
Nhat Nguyen 519768b5d3
Upgrade to Lucene-7.4-snapshot-6705632810 (#30519)
This snapshot is to include LUCENE-8298 which allows DocValues updates
to reset a value. This is needed for the Lucene rollback work.
2018-05-10 12:31:45 -04:00