Commit Graph

2906 Commits

Author SHA1 Message Date
Simon Willnauer 31740e279f Resolve index names to Index instances early
Today index names are often resolved lazily, only when they are really
needed. This can be problematic especially when it gets to mapping updates
etc. when a node sends a mapping update to the master but while the request
is in-flight the index changes for whatever reason we would still apply the update
since we use the name of the index to identify the index in the clusterstate.
The problem is that index names can be reused which happens in practice and sometimes
even in a automated way rendering this problem as realistic.
In this change we resolve the index including it's UUID as early as possible in places
where changes to the clusterstate are possible. For instance mapping updates on a node use a
concrete index rather than it's name and the master will fail the mapping update iff
the index can't be found by it's <name, uuid> tuple.

Closes #17048
2016-03-14 11:08:48 +01:00
Adrien Grand 5596e31068 Upgrade to lucene-6.0.0-f0aa4fc. #17075 2016-03-14 07:58:52 +01:00
Ryan Ernst 5bd7da5659 Addressed PR feedback
* Fix tests still referring to -E
* add comment about missing classes
* rename writer constant
2016-03-11 11:46:23 -08:00
Ryan Ernst 591fb8f028 Merge branch 'master' into cli-parsing 2016-03-11 10:45:05 -08:00
Yannick Welsch 04e55ecf6b Make logging message String constant to allow static checks 2016-03-11 10:30:59 +01:00
Yannick Welsch 718876a941 Fix wrong placeholder usage in logging statements 2016-03-11 10:30:59 +01:00
Nik Everett ebc12690bc [reindex] Move refresh tests to unit test
The refresh tests were failing rarely due to refreshes happening
automatically on indexes with -1 refresh intervals. This commit moves
the refresh test into a unit test where we can check if it was attempted
so we never get false failures from background refreshes.

It also stopped refresh from being run if the reindex request was canceled.
2016-03-10 17:48:22 -05:00
Nik Everett b2eec96045 [reindex] Make search failure cause rest failure
Indexing failures have caused the reindex http request to fail for a while
now. Both search and indexing failures cause it to abort. But search
failures didn't cause a non-200 response code from the http api. This
fixes that.

Also slips in a fix to some infrequently failing rest tests.

Closes #16037
2016-03-10 13:47:49 -05:00
Nik Everett b8d931d23c [reindex] Timeout if sub-requests timeout
Sadly, it isn't easy to simulate a timeout during an integration test, you
just have to cause one. Groovy's sleep should do the job.
2016-03-10 13:05:23 -05:00
Nik Everett 38241a5d8b [reindex] Implement CompositeIndicesRequest
Implements CompositeIndicesRequest on UpdateByQueryRequest and
ReindexRequest so that plugins can reason about the request. In both cases
this implementation is imperfect but useful because instead of listing
all requests that make up the request it instead attempts to make dummy
requests that represent the requests that it will later make.
2016-03-09 16:29:23 -05:00
Simon Willnauer 016bc3ea2b Merge pull request #17001 from s1monw/cut_over_to_uuid
Use index UUID to lookup indices on IndicesService

Today we use the index name to lookup index instances on the IndicesService
which applied to search requests but also to index deletion etc. This commit
moves the interface to expect an Index instance which is a tuple
and looks up the index by uuid rather than by name. This prevents accidental modification
of the wrong index if and index is recreated or searching from the wrong index in such a case.
Accessing an index that has the same name but different UUID will now result in an IndexNotFoundException.
2016-03-09 19:44:36 +01:00
Simon Willnauer e72dac91b3 Use index UUID to lookup indices on IndicesService
Today we use the index name to lookup index instances on the IndicesService
which applied to search reqeusts but also to index deletion etc. This commit
moves the interface to expcet and `Index` instance which is a <name, uuid> tuple
and looks up the index by uuid rather than by name. This prevents accidential modificaiton
of the wrong index if and index is recreated or searching from the _wrong_ index in such a case.
Accessing an index that has the same name but different UUID will now result in an IndexNotFoundException.

Closes #17001
2016-03-09 19:42:15 +01:00
Ryan Ernst 1dafead2eb Fix precommit 2016-03-08 22:55:24 -08:00
Nik Everett 6d0efae713 Teach list tasks api to wait for tasks to finish
_wait_for_completion defaults to false. If set to true then the API will
wait for all the tasks that it finds to stop running before returning. You
can use the timeout parameter to prevent it from waiting forever. If you
don't set a timeout parameter it'll default to 30 seconds.

Also adds a log message to rest tests if any tasks overrun the test. This
is just a log (instead of failing the test) because lots of tasks are run
by the cluster on its own and they shouldn't cause the test to fail. Things
like fetching disk usage from the other nodes, for example.

Switches the request to getter/setter style methods as we're going that
way in the Elasticsearch code base. Reindex is all getter/setter style.

Closes #16906
2016-03-08 11:53:57 -05:00
Christoph Büscher 5ff413074a Adding tests for `time_zone` parameter for date range aggregation 2016-03-07 15:38:24 +01:00
Adrien Grand c69fc008d4 Merge pull request #16972 from alexshadow007/fix-16812
Build empty extended stats aggregation if no docs collected for bucket
2016-03-07 10:44:20 +01:00
Robert Muir 54018a5d37 upgrade to lucene 6.0.0-snapshot-bea235f
Closes #16964

Squashed commit of the following:

commit a23f9d2d29220991aa498214530753d7a5a148c6
Merge: eec9c4e 0b0a251
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Mar 7 04:12:02 2016 -0500

    Merge branch 'master' into lucene6

commit eec9c4e5cd11e9c3e0b426f04894bb2a6dae4f21
Merge: bc67205 675d940
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 13:45:00 2016 -0500

    Merge branch 'master' into lucene6

commit bc67205bdfe1526eae277ab7856fc050ecbdb7b2
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 09:56:31 2016 -0500

    fix test bug

commit a60723b007ff12d97b1810cef473bd7b553a0327
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:35:35 2016 +0100

    Fix SimpleValidateQueryIT to put braces around boosted terms

commit ae3a49d7ba7ced448d2a5262e5d8ec98671a9090
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:27:25 2016 +0100

    fix multimatchquery

commit ae23fdb88a8f6d3fb7ba60fd1aaf3fd72d899aa5
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:20:49 2016 +0100

    Rewrite DecayFunctionScoreIT to be independent of the similarity used

    This test relied a lot on the term scoring and compared scores
    that are dependent on the similarity. This commit changes the base query
    to be a predictable constant score query.

commit 366c2d518c35d31251033f1b6f6a93f6e2ae327d
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 14:06:14 2016 +0100

    Fix scoring in tests due to changes to idf calculation.

    Lucene 6 uses a different default similarity as well as a different
    way to calculate IDF. In contrast to older version lucene 6 uses docCount per field
    to calculate the IDF not the # of docs in the index to overcome the sparse field
    cases.

commit dac99fd64ac2fa71b8d8d106fe68825e574c49f8
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 08:21:57 2016 -0500

    don't hardcoded expected termquery score

commit 6e9f340ba49ab10eed512df86d52a121aa775b0f
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 08:04:45 2016 -0500

    suppress deprecation warning until migrated to points

commit 3ac8908424b3fdad44a90a4f7bdb3eff7efd077d
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:21:43 2016 -0500

    Remove invalid test: all commits have IDs, and its illegal to do this.

commit c12976288124ad1a26467e7e848fb810548e7eab
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:06:14 2016 -0500

    don't test with unsupported back compat

commit 18bbfe76128570bc70883bf91ff4c44c82d27817
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:02:18 2016 -0500

    remove now invalid lucene 4 backcompat test

commit 7e730e572886f0ef2d3faba712e4256216ff01ec
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:58:52 2016 -0500

    remove now invalid lucene 4 backwards test

commit 244d2ab6868ba5ac9e0bcde3c2833743751a25ec
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:47:23 2016 -0500

    use 6.0 codec

commit 5f64d4a431a6fdaa1234adca23f154c2a1de8284
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:43:08 2016 -0500

    compile, javadocs, forbidden-apis, etc

commit 1f273cd62a7fe9ca8f8944acbbfc5cbdd3d81ccb
Merge: cd33921 29e3443
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 10:45:29 2016 +0100

    Merge branch 'master' into lucene6

commit cd33921ac742ef9fb351012eff35f3c7dbda7264
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:58:37 2016 -0500

    fix hunspell dictionary loading

commit c7fdbd837b01f7defe9cb1c24e2ec65604b0dc96
Merge: 4d4190f d8948ba
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:41:53 2016 -0500

    Merge branch 'master' into lucene6

commit 4d4190fd82601aaafac6b8254ccb3edf218faa34
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:39:14 2016 -0500

    remove nocommit

commit 77ca69e288b1a41aa9595c921ed166c272a00ea8
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:38:24 2016 -0500

    clean up numericutils vs legacynumericutils

commit a466d696fbaad04b647ffbc0857a9439b583d0bf
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:32:43 2016 -0500

    upgrade spatial4j

commit 5412c747a8cfe638bacedbc8233163cb75cc3dc5
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:19:28 2016 -0500

    move to 6.0.0-snapshot-8eada27

commit b32bfe924626b87e540692375ece09e7c2edb189
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:30:09 2016 +0100

    Fix some test compile errors.

commit 6ccde35e9840b03c68d1a2cd47c7923a06edf64a
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:25:51 2016 +0100

    Current Lucene version is 6.0.0.

commit f62e1015d931b4cc04c778298a8fa1ba65e97ad9
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:20:48 2016 +0100

    Fix compile errors in NGramTokenFilterFactory.

commit 6837c6eabf96075f743649da9b9b52dd39611c58
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:50:59 2016 +0100

    Fix the edge ngram tokenizer/filter.

commit ccd7f070de5efcdfbeb34b9555c65c4990bf1ba6
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:42:44 2016 +0100

    The missing value is now accessible through a getter.

commit bd3b77f9b28e5b05daa3d49683a9922a6baf2963
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:41:51 2016 +0100

    Remove IndexCacheableQuery.

commit 05f3091c347aeae80eeb16349ac51d2b53cf86f7
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:39:43 2016 +0100

    Fix compilation of function_score queries.

commit 81cda79a2431ac78f56b0cc5a5765387f662d801
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:35:02 2016 +0100

    Fix compile errors in BlendedTermQuery.

commit 70994ce8dd1eca0b995870974a38e20f26f96a7b
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 23:33:03 2016 -0500

    add bug ID

commit 29d4f1a71f36f646b5a6060bed3db019564a279d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 21:02:32 2016 -0500

    easy .store changes

commit 5e1a1e6fd665fa455e88d3a8987362fad5f44bb1
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 20:47:24 2016 -0500

    cleanups mostly around boosting

commit 333a669ec6c305ada5645d13ed1da0e19ec1d053
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 20:27:56 2016 -0500

    more simple fixes

commit bd5cd98a1e089c866b6b4a5e159400b110140ce6
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 19:49:38 2016 -0500

    more easy fixes and removal of ancient cruft

commit a68f419ee47da5f9c9ce5b372f01d707e902474c
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 19:35:02 2016 -0500

    cutover numerics

commit 4ca5dc1fa47dd5892db00899032133318fff3116
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:34:18 2016 -0500

    fix some constants

commit 88710a17817086e477c6c021ec346d0534b7fb88
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:14:25 2016 -0500

    Add spatial-extras jar as a core dependency

commit c8cd6726583e5ce3f546ed355d4eca037164a30d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:03:33 2016 -0500

    update to lucene 6 jars
2016-03-07 04:12:23 -05:00
Alexander Kazakov 3674774b73 Build empty extended stats aggregation if no docs are collected for bucket #16812 2016-03-05 22:50:54 +03:00
Nik Everett 4d6cb34417 [reindex] Add ingest support 2016-03-04 10:05:13 -05:00
Lee Hinman 6adbbff97c Fix organization rename in all files in project
Basically a query-replace of "https://github.com/elasticsearch/" with "https://github.com/elastic/"
2016-03-03 12:04:13 -07:00
Clinton Gormley 529f7cb42c Fixed bad YAML in msearch test 2016-03-03 14:45:02 +01:00
Adrien Grand 2b545df372 Fix modules/reindex to not use the string field anymore. 2016-03-03 11:11:00 +01:00
Adrien Grand eef19be072 Deprecate string in favor of text/keyword. #16877
This commit removes the ability to use string fields on indices created on or
after 5.0. Dynamic mappings now generate text fields by default for strings
but there are plans to also add a sub keyword field (in a future PR).

Most of the changes in this commit are just about replacing string with
keyword or text. Some tests have been removed because they existed because of
corner cases of string mappings like setting ignore-above on a text field or
enabling term vectors on a keyword field which are now impossible.

The plan is to remove strings entirely in 6.0.
2016-03-03 10:20:56 +01:00
Nik Everett 18e5bb83c5 Disable problematic reindex test
This should get the builds back to normal while we wait on #16914 or
something like it to fix the test properly.
2016-03-02 13:02:29 -05:00
Nik Everett 942eb70956 Revert "Silence reindex's rest tests"
This reverts commit aa0ef84f5a.
2016-03-02 09:17:06 -05:00
Nik Everett aa0ef84f5a Silence reindex's rest tests
They are failing sporadically in CI.
2016-03-02 08:32:30 -05:00
Nik Everett c7c8bb357a Merge pull request #16861 from nik9000/reindex_is_ready
Reindex required some parsing changes for search requests to support
differing defaults from the regular search api.
2016-03-01 10:02:48 -05:00
javanna b7597d7aea Rename SearchServiceTransportAction to SearchTransportService
The suffix TransportAction is misleading as it may make think that it extends TransportAction, but it does not. This class makes accessible the different search operations exposed by SearchService through the transport layer. Also resolved few compiler warnings in the class itself.
2016-03-01 11:35:04 +01:00
Nik Everett aeed7ee218 Reindex: rename source to searchRequest
This makes the code easier to read for those familiar with the
Elasticsearch code base.
2016-02-29 14:57:16 -05:00
Nik Everett 16934d50b6 Reindex isn't a plugin
Remove another mention of it being a plugin.
2016-02-29 10:21:10 -05:00
Nik Everett 356b7cd12b Merge reindex to master because it is ready! 2016-02-29 10:08:29 -05:00
Nik Everett d587a74533 Fix reindex to work with master branch
Stuff changed, reindex's got to change.
2016-02-29 10:04:16 -05:00
Yannick Welsch ab4e671bfd Remove suppressAccessChecks permission for Groovy script plugin
Closes #16527
2016-02-27 18:58:00 +01:00
Yannick Welsch af918ebf2b Upgrade groovy dependency in lang-groovy module to version 2.4.6
Closes #16830
2016-02-27 17:41:15 +01:00
Nik Everett c38119bae9 Merge branch 'master' into feature/reindex 2016-02-26 16:59:54 -05:00
Jack Conradson 7986770e5f Moved Painless from a plugin to a module.
Closes #16755
2016-02-21 16:50:54 -08:00
Mike McCandless 5fffede2b0 Upgrade to Lucene 5.5.0 official release 2016-02-20 17:34:16 -05:00
Colin Goodheart-Smithe ebc745211d Moved AggregatorBuilder implementations into their own class files
Also renamed histogram.AbstractBuilcer to AbstractHistogramBuilder, range.AbstractBuilder to AbstractRangeBuilder and org.elasticsearch.search.aggregations.pipeline.having to org.elasticsearch.search.aggregations.pipeline.bucketselector
2016-02-15 14:09:36 +00:00
Colin Goodheart-Smithe 1a46628daa Merge branch 'master' into feature/aggs-refactoring
# Conflicts:
#	core/src/test/java/org/elasticsearch/search/builder/SearchSourceBuilderTests.java
2016-02-15 10:37:16 +00:00
Jason Tedor 1fb1ffbac6 Class permission for Groovy references
This commit adds a class permission for groovy.lang.Reference so they
can be used in scripts.

Closes #16660
2016-02-14 09:45:40 -05:00
Simon Willnauer 297fb24112 apply fixes after review 2016-02-12 12:53:18 +01:00
Colin Goodheart-Smithe ed3f7903f4 Merge branch 'master' into feature/aggs-refactoring 2016-02-12 09:48:59 +00:00
Simon Willnauer 685bee3081 Merge branch 'master' into add_rewrite_infra 2016-02-12 09:46:01 +01:00
Nik Everett 821a20f582 Merge branch 'master' into feature/reindex 2016-02-11 17:41:05 -05:00
Nik Everett 18808b7576 Move reindex from a plugin to a module 2016-02-11 17:39:49 -05:00
Nicholas Knize 52ee4c7027 upgrade to lucene 5.5.0-snapshot-850c6c2 2016-02-11 14:28:50 -06:00
Adrien Grand bc47c577d2 Add a new `keyword` field.
The `keyword` field is intended to replace `not_analyzed` string fields. It is
indexed and has doc values by default, and doesn't support enabling term
vectors.

Although it doesn't support setting an analyzer for now, there are plans for
it to support basic normalization in the future such as case folding.
2016-02-11 18:19:53 +01:00
Simon Willnauer 8bdc7362a6 Add infrastructure to rewrite query buidlers
QueryBuilders today do all their heavy lifting in toQuery() which
can be too late for several operations. For instance if we want to fetch geo shapes
on the coordinating node we need to do all this before we create the actual lucene query
which happens on the shard itself. Also optimizations for request caching need to be done
to the query builder rather than the query which then in-turn needs to be serialized again.
This commit adds the basic infrastructure for query rewriting and moves the heavy lifting into
the rewrite method for the following queries:

 * `WrapperQueryBuilder`
 * `GeoShapeQueryBuilder`
 * `TermsQueryBuilder`
 * `TemplateQueryBuilder`

Other queries like `MoreLikeThisQueryBuilder` still need to be fixed / converted. The nice
sideeffect of this is that queries like template queries will now also match the request cache
if their non-template equivalent has been cached befoore. In the future this will allow to
add optimizataion like rewriting time-based queries into primitives like `match_all_docs` or `match_no_docs`
based on the currents shards bounds. This is especially appealing for indices that are read-only ie. never change.
2016-02-11 10:45:49 +01:00
Colin Goodheart-Smithe 1f760bd1bd Merge branch 'master' into feature/aggs-refactoring 2016-02-10 12:16:26 +00:00
Colin Goodheart-Smithe 54e2f31189 Parse Aggregations on Coordinating node 2016-02-10 10:00:16 +00:00