Commit Graph

2745 Commits

Author SHA1 Message Date
Simon Willnauer 7f16a1d9a7 Improve upgrade experience of node level index settings
In 5.0 we don't allow index settings to be specified on the node level ie.
in yaml files or via commandline argument. This can cause problems during
upgrade if this was used extensively. For instance if analyzers where
specified on a node level this might cause the index to be closed when
imported (see #17187). In such a case all indices relying on this
must be updated via `PUT /${index}/_settings`. Yet, this API has slightly
different semantics since it overrides existing settings. To make this less
painful this change adds a `preserve_existing` parameter on that API to ensure
we have the same semantics as if the setting was applied on the node level.

This change also adds a better error message and a change to the migration guide
to ensure upgrades are smooth if index settings are specified on the node level.

If a index setting is detected this change fails the node startup and prints a message
like this:
```
*************************************************************************************
Found index level settings on node level configuration.

Since elasticsearch 5.x index level settings can NOT be set on the nodes
configuration like the elasticsearch.yaml, in system properties or command line
arguments.In order to upgrade all indices the settings must be updated via the
/${index}/_settings API. Unless all settings are dynamic all indices must be closed
in order to apply the upgradeIndices created in the future should use index templates
to set default values.

Please ensure all required values are updated on all indices by executing:

curl -XPUT 'http://localhost:9200/_all/_settings?preserve_existing=true' -d '{
  "index.number_of_shards" : "1",
  "index.query.default_field" : "main_field",
  "index.translog.durability" : "async",
  "index.ttl.disable_purge" : "true"
}'
*************************************************************************************
```
2016-03-21 20:12:18 +01:00
javanna bf390a935e Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-21 17:18:23 +01:00
Martijn van Groningen 000e419795 docs: fix link 2016-03-21 14:09:30 +01:00
Martijn van Groningen e3b7e5d75a percolator: Replace percolate api with the new percolator query
Also replaced the PercolatorQueryRegistry with the new PercolatorQueryCache.

The PercolatorFieldMapper stores the rewritten form of each percolator query's xcontext
in a binary doc values field. This make sure that the query rewrite happens only during
indexing (some queries for example fetch shapes, terms in remote indices) and
the speed up the loading of the queries in the percolator query cache.

Because the percolator now works inside the search infrastructure a number of features
(sorting fields, pagination, fetch features) are available out of the box.

The following feature requests are automatically implemented via this refactoring:

Closes #10741
Closes #7297
Closes #13176
Closes #13978
Closes #11264
Closes #10741
Closes #4317
2016-03-21 12:21:50 +01:00
Clinton Gormley 830e2e049a Docs: Build release notes 2016-03-18 15:04:49 +01:00
Clinton Gormley e07b6a2641 Docs: Added 5.0.0-alpha1 release notes 2016-03-18 14:51:49 +01:00
Clinton Gormley 4506b7ad82 Docs: Fixed bad asciidoc link 2016-03-18 13:26:01 +01:00
Clinton Gormley dc21ab7576 Docs: Corrected behaviour of max_token_length in standard tokenizer 2016-03-18 10:58:16 +01:00
Alexander Reelsen 2dffad9ec3 Docs: Display reindex/update by query API and fix build doc issue
The documentation existed, but was not linked anywhere.
Also fixed the docs to make sure they build with this enabled.
2016-03-18 10:44:16 +01:00
Martijn van Groningen 3b17ddcd46 Removed old 1.x parent/child logic that should have been removed.
`0` really means, don't match any child docs.
2016-03-18 10:07:27 +01:00
Martijn van Groningen 1dd2be81c3 nested / parent child: Removed `total` score mode in favour of `sum` score mode.
Closes #17083
2016-03-18 10:07:26 +01:00
Clinton Gormley bd059b8cc3 Clarify how `-Djava.security.policy=someURL` must be passed
Closes #17160
2016-03-17 14:13:19 +01:00
Areek Zillur da165f425f update migration doc for removing gateway.format setting 2016-03-16 18:48:02 -04:00
Alexander Kazakov 51ac97000b Fix column aliases in _cat/indices, _cat/nodes and _cat/shards APIs #17101 2016-03-16 19:18:43 +03:00
Christoph Büscher 6ddf9ae92f Merge branch 'master' into feature-suggest-refactoring 2016-03-16 15:27:02 +01:00
Clinton Gormley d83e12094e Docs: Added redirect entries for multicast plugin and the cloud plugins 2016-03-16 12:31:00 +01:00
Christoph Büscher 39667b5793 Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	docs/reference/migration/migrate_5_0/java.asciidoc
2016-03-16 12:06:42 +01:00
Jason Tedor 618441aea3 Merge pull request #17088 from jasontedor/simplify-bootstrap-settings
Bootstrap does not set system properties
2016-03-15 19:25:16 -04:00
Clinton Gormley 432f0cc193 Docs: Added the ingest node to the modules/nodes page
Closes #17113
2016-03-15 19:03:18 +01:00
Jason Tedor 2f7e181318 Fix typo inadvertently introduced 2016-03-15 10:05:28 -04:00
Christoph Büscher bc84cdfed1 Using SortMode enum in all sort builders 2016-03-15 12:43:19 +01:00
Christoph Büscher b4b874f0d8 Merge branch 'master' into feature-suggest-refactoring 2016-03-15 12:11:39 +01:00
Areek Zillur 35f7cfb6c0 Add upgrader to upgrade old indices to new naming convention 2016-03-14 23:24:05 -04:00
Christoph Büscher 40f3501d7f Merge branch 'master' into feature-suggest-refactoring 2016-03-14 14:57:57 +01:00
Clinton Gormley 0ed0fea558 Updated link to Joda time zones 2016-03-14 12:24:58 +01:00
Christoph Büscher 97638c95fc Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	docs/reference/migration/migrate_5_0.asciidoc
2016-03-14 11:13:47 +01:00
Clinton Gormley c3cd8564df Corrected regexp syntax docs for COMPLEMENT 2016-03-14 10:46:31 +01:00
Clinton Gormley c90b4f3bae Docs: Added note about upgrading from 1.x to 5.x 2016-03-14 09:58:46 +01:00
Jason Tedor 8a05c2a2be Bootstrap does not set system properties
Today, certain bootstrap properties are set and read via system
properties. This action-at-distance way of managing these properties is
rather confusing, and completely unnecessary. But another problem exists
with setting these as system properties. Namely, these system properties
are interpreted as Elasticsearch settings, not all of which are
registered. This leads to Elasticsearch failing to startup if any of
these special properties are set. Instead, these properties should be
kept as local as possible, and passed around as method parameters where
needed. This eliminates the action-at-distance way of handling these
properties, and eliminates the need to register these non-setting
properties. This commit does exactly that.

Additionally, today we use the "-D" command line flag to set the
properties, but this is confusing because "-D" is a special flag to the
JVM for setting system properties. This creates confusion because some
"-D" properties should be passed via arguments to the JVM (so via
ES_JAVA_OPTS), and some should be passed as arguments to
Elasticsearch. This commit changes the "-D" flag for Elasticsearch
settings to "-E".
2016-03-13 20:09:15 -04:00
Jason Tedor 8ac5a98b87 Remove links to nonexistent migration docs 2016-03-13 19:12:06 -04:00
Clinton Gormley 5f48b9c86a Removed breaking changes docs for < 5.0 2016-03-13 21:18:44 +01:00
Clinton Gormley 5c845f8bb5 Reworked 5.0 breaking changes docs 2016-03-13 21:17:48 +01:00
Ryan Ernst 5f3d0067f8 Merge pull request #17024 from rjernst/cli-parsing
Cli: Switch to jopt-simple
2016-03-11 12:35:39 -08:00
Ryan Ernst 3f44e1d429 Remove old reference to site plugins example in docs 2016-03-11 11:53:20 -08:00
Ryan Ernst 591fb8f028 Merge branch 'master' into cli-parsing 2016-03-11 10:45:05 -08:00
Clinton Gormley a5a9bbfe88 Update compound-word-tokenfilter.asciidoc
Only FOP v1.2 compatible hyphenation files are supported by the hyphenation decompounder
2016-03-11 15:08:36 +01:00
Jason Tedor f465d98eb3 Add raw recovery progress to cat recovery API
This commit adds fields bytes_recovered and files_recovered to the cat
recovery API. These fields, respectively, indicate the total number of
bytes and files recovered. Additionally, for consistency, some totals
fields and translog recovery fields have been renamed.

Closes #17064
2016-03-11 08:27:09 -05:00
Christoph Büscher daeffb149c Merge branch 'master' into feature-suggest-refactoring 2016-03-11 10:37:28 +01:00
Daniel Mitterdorfer 94aa025b93 Document breaking change in ClusterHealthResponse in 2.2 2016-03-11 09:47:53 +01:00
Ed Winn c4934f5250 Current link returns 404. Updated 2016-03-10 16:52:30 -07:00
Lee Hinman b3bd189cd7 [DOCS] Fix missing table end in function-score-query.asciidoc 2016-03-10 13:35:33 -07:00
Lee Hinman 982a369efc [DOCS] Document the modifiers for `field_value_factor`
Resolves #13511
2016-03-10 11:34:34 -07:00
Yannick Welsch 295d33c2a6 Merge pull request #17021 from ywelsch/fix/block-delete-on-snapshot
Fail closing or deleting indices during a full snapshot
2016-03-10 18:51:04 +01:00
Yannick Welsch 266394c3ab Fail closing or deleting indices during a full snapshot
Closes #16321
2016-03-10 18:11:47 +01:00
Lee Hinman 22e716551b Add -XX+AlwaysPreTouch JVM flag
Enables the touching of all memory pages used by the JVM heap spaces
during initialization of the HotSpot VM, which commits all memory pages
at initialization time. By default, pages are committed only as they are
needed.
2016-03-10 10:11:32 -07:00
Clinton Gormley 2fa573bc58 Missing word in docs 2016-03-10 14:34:05 +01:00
Robin Clarke 046212035c Clarification about precedence of settings
Closes #14559
2016-03-10 14:29:51 +01:00
Nicholas Knize f7a2dbfcaf fixing silly typo in docs 2016-03-10 07:28:13 -06:00
Andrew Cholakian b8db32b7fd Improved transport sniffing docs
Closes #15204
2016-03-10 14:20:09 +01:00
Clinton Gormley 4219f8e062 Updated Java API docs with version recommendations
Closes #15188
2016-03-10 14:09:15 +01:00
Clinton Gormley 6b4a6be2df Merge pull request #14800 from MaineC/doc-fix/query-dsl
Fix minor typos in query dsl docs
2016-03-10 13:45:13 +01:00
Martijn van Groningen 2fa33d5c47 Added ingest statistics to node stats API
The ingest stats include the following statistics:
* `ingest.total.count`- The total number of document ingested during the lifetime of this node
* `ingest.total.time_in_millis` - The total time spent on ingest preprocessing documents during the lifetime of this node
* `ingest.total.current` - The total number of documents currently being ingested.
* `ingest.total.failed` - The total number ingest preprocessing operations failed during the lifetime of this node

Also these stats are returned on a per pipeline basis.
2016-03-10 13:21:43 +01:00
Clinton Gormley a8c7ae7809 Fixed bad docs link 2016-03-10 13:08:29 +01:00
Clinton Gormley f9622f9acc Docs: Added a note about the update API not supporting external versioning
Closes #12820
2016-03-10 12:33:39 +01:00
Christoph Büscher 69c83b3459 Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/action/suggest/TransportSuggestAction.java
        core/src/main/java/org/elasticsearch/search/suggest/SuggestParseElement.java
	core/src/main/java/org/elasticsearch/search/suggest/SuggestionSearchContext.java
	core/src/main/java/org/elasticsearch/search/suggest/phrase/PhraseSuggester.java
2016-03-10 11:44:36 +01:00
Nicholas Knize 55635d5de1 update coerce and breaking changes documentation 2016-03-09 16:09:44 -06:00
Nicholas Knize 61f39e6c92 GeoPointV2 update docs and query builders
This commit updates the documentation for GeoPointField by removing all references to the coerce and doc_values parameters. DocValues are enabled in lucene GeoPointField by default (required for boundary filtering). The QueryBuilders are updated to automatically normalize points (ignoring the coerce parameter) for any index created onOrAfter version 2.2.
2016-03-09 16:09:44 -06:00
Clinton Gormley 06929f8ed4 Merge pull request #17030 from 36degrees/patch-1
Fix typo in clear cache documentation
2016-03-09 15:34:15 +01:00
Ryan Ernst fdce9d7c4d Merge branch 'master' into cli-parsing 2016-03-08 14:18:20 -08:00
Ryan Ernst 3836f3a736 Remove reference to standalonerunner 2016-03-08 13:40:39 -08:00
Jason Tedor c0572c631d Note to configuration docs on number of threads
This commit adds a note to the configuration docs regarding the number
of threads necessary for the Elasticsearch user.

Relates #17003
2016-03-08 09:14:55 -05:00
Christoph Büscher 1264f37a1b Merge branch 'master' into feature-suggest-refactoring 2016-03-08 11:10:08 +01:00
Christoph Büscher ff46303f15 Simplify mock scripts 2016-03-07 15:39:35 +01:00
Christoph Büscher 6b0f63e1a6 Adding `time_zone` parameter to daterange-aggregation docs 2016-03-07 15:38:24 +01:00
Martijn van Groningen 82d01e4315 Added ingest info to node info API, which contains a list of available processors.
Internally the put pipeline API uses this information in node info API to validate if all specified processors in a pipeline exist on all nodes in the cluster.
2016-03-07 14:44:50 +01:00
Christopher Taylor 93adddc61b Document `sum` as supported scoring type
the examples all use `sum` for the `"score_mode"` field, but it isn't listed in the list of supported modes.
2016-03-07 10:48:58 +01:00
Christoph Büscher 7ec5075a87 Merge branch 'master' into feature-suggest-refactoring 2016-03-07 10:45:43 +01:00
Clinton Gormley a5eca17b4e Merge pull request #16971 from clintongormley/merge-settings-docs
Redocument the `index.merge.scheduler.max_thread_count` setting
2016-03-07 10:42:26 +01:00
Robert Muir 54018a5d37 upgrade to lucene 6.0.0-snapshot-bea235f
Closes #16964

Squashed commit of the following:

commit a23f9d2d29220991aa498214530753d7a5a148c6
Merge: eec9c4e 0b0a251
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Mar 7 04:12:02 2016 -0500

    Merge branch 'master' into lucene6

commit eec9c4e5cd11e9c3e0b426f04894bb2a6dae4f21
Merge: bc67205 675d940
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 13:45:00 2016 -0500

    Merge branch 'master' into lucene6

commit bc67205bdfe1526eae277ab7856fc050ecbdb7b2
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 09:56:31 2016 -0500

    fix test bug

commit a60723b007ff12d97b1810cef473bd7b553a0327
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:35:35 2016 +0100

    Fix SimpleValidateQueryIT to put braces around boosted terms

commit ae3a49d7ba7ced448d2a5262e5d8ec98671a9090
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:27:25 2016 +0100

    fix multimatchquery

commit ae23fdb88a8f6d3fb7ba60fd1aaf3fd72d899aa5
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:20:49 2016 +0100

    Rewrite DecayFunctionScoreIT to be independent of the similarity used

    This test relied a lot on the term scoring and compared scores
    that are dependent on the similarity. This commit changes the base query
    to be a predictable constant score query.

commit 366c2d518c35d31251033f1b6f6a93f6e2ae327d
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 14:06:14 2016 +0100

    Fix scoring in tests due to changes to idf calculation.

    Lucene 6 uses a different default similarity as well as a different
    way to calculate IDF. In contrast to older version lucene 6 uses docCount per field
    to calculate the IDF not the # of docs in the index to overcome the sparse field
    cases.

commit dac99fd64ac2fa71b8d8d106fe68825e574c49f8
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 08:21:57 2016 -0500

    don't hardcoded expected termquery score

commit 6e9f340ba49ab10eed512df86d52a121aa775b0f
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 08:04:45 2016 -0500

    suppress deprecation warning until migrated to points

commit 3ac8908424b3fdad44a90a4f7bdb3eff7efd077d
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:21:43 2016 -0500

    Remove invalid test: all commits have IDs, and its illegal to do this.

commit c12976288124ad1a26467e7e848fb810548e7eab
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:06:14 2016 -0500

    don't test with unsupported back compat

commit 18bbfe76128570bc70883bf91ff4c44c82d27817
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:02:18 2016 -0500

    remove now invalid lucene 4 backcompat test

commit 7e730e572886f0ef2d3faba712e4256216ff01ec
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:58:52 2016 -0500

    remove now invalid lucene 4 backwards test

commit 244d2ab6868ba5ac9e0bcde3c2833743751a25ec
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:47:23 2016 -0500

    use 6.0 codec

commit 5f64d4a431a6fdaa1234adca23f154c2a1de8284
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:43:08 2016 -0500

    compile, javadocs, forbidden-apis, etc

commit 1f273cd62a7fe9ca8f8944acbbfc5cbdd3d81ccb
Merge: cd33921 29e3443
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 10:45:29 2016 +0100

    Merge branch 'master' into lucene6

commit cd33921ac742ef9fb351012eff35f3c7dbda7264
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:58:37 2016 -0500

    fix hunspell dictionary loading

commit c7fdbd837b01f7defe9cb1c24e2ec65604b0dc96
Merge: 4d4190f d8948ba
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:41:53 2016 -0500

    Merge branch 'master' into lucene6

commit 4d4190fd82601aaafac6b8254ccb3edf218faa34
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:39:14 2016 -0500

    remove nocommit

commit 77ca69e288b1a41aa9595c921ed166c272a00ea8
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:38:24 2016 -0500

    clean up numericutils vs legacynumericutils

commit a466d696fbaad04b647ffbc0857a9439b583d0bf
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:32:43 2016 -0500

    upgrade spatial4j

commit 5412c747a8cfe638bacedbc8233163cb75cc3dc5
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:19:28 2016 -0500

    move to 6.0.0-snapshot-8eada27

commit b32bfe924626b87e540692375ece09e7c2edb189
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:30:09 2016 +0100

    Fix some test compile errors.

commit 6ccde35e9840b03c68d1a2cd47c7923a06edf64a
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:25:51 2016 +0100

    Current Lucene version is 6.0.0.

commit f62e1015d931b4cc04c778298a8fa1ba65e97ad9
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:20:48 2016 +0100

    Fix compile errors in NGramTokenFilterFactory.

commit 6837c6eabf96075f743649da9b9b52dd39611c58
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:50:59 2016 +0100

    Fix the edge ngram tokenizer/filter.

commit ccd7f070de5efcdfbeb34b9555c65c4990bf1ba6
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:42:44 2016 +0100

    The missing value is now accessible through a getter.

commit bd3b77f9b28e5b05daa3d49683a9922a6baf2963
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:41:51 2016 +0100

    Remove IndexCacheableQuery.

commit 05f3091c347aeae80eeb16349ac51d2b53cf86f7
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:39:43 2016 +0100

    Fix compilation of function_score queries.

commit 81cda79a2431ac78f56b0cc5a5765387f662d801
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:35:02 2016 +0100

    Fix compile errors in BlendedTermQuery.

commit 70994ce8dd1eca0b995870974a38e20f26f96a7b
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 23:33:03 2016 -0500

    add bug ID

commit 29d4f1a71f36f646b5a6060bed3db019564a279d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 21:02:32 2016 -0500

    easy .store changes

commit 5e1a1e6fd665fa455e88d3a8987362fad5f44bb1
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 20:47:24 2016 -0500

    cleanups mostly around boosting

commit 333a669ec6c305ada5645d13ed1da0e19ec1d053
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 20:27:56 2016 -0500

    more simple fixes

commit bd5cd98a1e089c866b6b4a5e159400b110140ce6
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 19:49:38 2016 -0500

    more easy fixes and removal of ancient cruft

commit a68f419ee47da5f9c9ce5b372f01d707e902474c
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 19:35:02 2016 -0500

    cutover numerics

commit 4ca5dc1fa47dd5892db00899032133318fff3116
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:34:18 2016 -0500

    fix some constants

commit 88710a17817086e477c6c021ec346d0534b7fb88
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:14:25 2016 -0500

    Add spatial-extras jar as a core dependency

commit c8cd6726583e5ce3f546ed355d4eca037164a30d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:03:33 2016 -0500

    update to lucene 6 jars
2016-03-07 04:12:23 -05:00
Clinton Gormley 6d7e8814d6 Redocument the `index.merge.scheduler.max_thread_count` setting
Closes #16961
2016-03-05 16:28:43 +01:00
javanna e5d9328a2d [DOCS] adapt docs to node.client setting removal 2016-03-05 10:55:19 +01:00
javanna 9c4a5bbe7e adapt cluster stats api to node.client setting removal
The cluster stats api now returns counts for each node role. The `master_data`, `master_only`, `data_only` and `client` fields have been removed from the response in favour of `master`, `data`, `ingest` and `coordinating_only`. The same node can have multiple roles, hence contribute to multiple roles counts. Every node is implicitly a coordinating node, so whenever a node has no explicit roles, it will be counted as coordinating only.
2016-03-05 10:55:19 +01:00
javanna f786e9866c adapt _cat/nodes to node.client removal
_cat/nodes used to return `c` for client node or `d` for data node as part of the node.role column. This commit changes it to return `m` for master eligible, `d` for data and/or `i` for ingest. A node with no explicit roles will be a coordinating only node and marked with `-`. A node can obviously have multiple roles. The master column has been adapted to return only whether a node is the current master (`*`) or not (`-`).
2016-03-05 10:55:19 +01:00
Nik Everett d079830f10 [docs] Fix bad link in reindex docs 2016-03-04 14:34:30 -05:00
Nik Everett 22557621fe [docs] Docs for reindex using ingest pipeline 2016-03-04 10:05:13 -05:00
Jim Ferenczi 9a203dcb66 Fix wrong include in docs 2016-03-04 12:44:28 +01:00
Jim Ferenczi cd950a34da Merge pull request #16900 from jimferenczi/mapping_field_level_boost
Change the field mapping index time boost into a query time boost
2016-03-04 11:50:29 +01:00
Martijn van Groningen 116acee1dd Merge pull request #16946 from dedemorton/ingest_doc_edit
Improve the ingest documentation.
2016-03-04 11:49:19 +01:00
Jim Ferenczi 927303e7a9 Change the field mapping index time boost into a query time boost.
Index time boost will still be applied for indices created before 5.0.0.
2016-03-04 11:47:35 +01:00
Clinton Gormley 29e3443917 Update api-conventions.asciidoc
Docs: Fix bad date math index expression
2016-03-04 10:10:29 +01:00
DeDe Morton 4d0124e65c Edits to ingest plugin docs 2016-03-03 22:49:31 -08:00
DeDe Morton 6b52b0bdc3 Ingest node edits 2016-03-03 22:29:27 -08:00
Vector241-Eric d8948bae5b Fix a quick documentation typo.
Fix the object of the sentence to agree with the plural verb.
2016-03-03 16:36:21 -07:00
James Kerr 719c862675 Typo in verb tense of "to gather" 2016-03-03 16:21:11 -07:00
Christoph Büscher 30a788d87c Suggestions: Make field name mandatory ctor argument
The field name is a required argument for all suggesters, but
it was specified via a field() setter in SuggestionBuilder so far.
This changes field name to being a mandatory constructor argument
and lets suggestion builders throw an error if field name is missing
or the empty string.
2016-03-03 21:35:53 +01:00
Lee Hinman 6adbbff97c Fix organization rename in all files in project
Basically a query-replace of "https://github.com/elasticsearch/" with "https://github.com/elastic/"
2016-03-03 12:04:13 -07:00
Geoff Wagstaff 06e2a8262d Fix use of apostrophe 2016-03-03 11:56:56 -07:00
Adrien Grand 2c3e4840f2 Store _all payloads on 1 byte instead of 4. #16899
This changes the `_all` field to store per-field boosts using a single byte
similarly to norms.
2016-03-03 15:00:23 +01:00
Christoph Büscher ef4db5c1e4 Merge branch 'master' into feature-suggest-refactoring 2016-03-03 12:18:03 +01:00
Michael McCandless 4df83dc9a5 Merge pull request #16661 from camilojd/feature/disk-segments-stats
Segments stats in indices stats API now optionally includes aggregated file sizes by file extension / index component
2016-03-02 13:58:20 -05:00
Clinton Gormley 69b5b1920f Merge pull request #16907 from centic9/patch-2
Elasticsearch monitoring support for Dynatrace Application Monitoring
2016-03-02 15:25:54 +01:00
Christoph Büscher c0e5e40e02 Merge branch 'master' into feature-suggest-refactoring 2016-03-02 14:22:03 +01:00
Clinton Gormley 5bb744bfde Changed v3.0.0 to v5.0.0 in plugin docs 2016-03-02 11:57:42 +01:00
Clinton Gormley d1e6e2d1df Merge pull request #16741 from blachniet/patch-1
Update aggregations.asciidoc
2016-03-02 10:57:56 +01:00
Clinton Gormley 05e3cd6b97 Merge pull request #16878 from peschlowp/patch-8
Update index-options.asciidoc
2016-03-02 10:52:44 +01:00
Clinton Gormley 73529d551c Merge pull request #16875 from peschlowp/patch-7
Update configuration.asciidoc
2016-03-02 10:50:57 +01:00
Camilo Díaz Repka 356364810c Implementation of Segment disk stats aggregating sizes by index file extension.
Use 'includeSegmentFileSizes' as the flag name to report disk usage.
Added test that verifies reported segment disk usage is growing accordingly after adding a document.
Documentation: Reference the new parameter as part of indices stats.
2016-03-01 23:16:49 -03:00
Jason Tedor aa8ee74c6c Bump Elasticsearch version to 5.0.0-SNAPSHOT
This commit bumps the Elasticsearch version to 5.0.0-SNAPSHOT in line
with the alignment of versions across the stack.

Closes #16862
2016-03-01 17:03:47 -05:00
Christoph Büscher aecf51cb42 Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	core/src/main/java/org/elasticsearch/rest/action/search/RestSearchAction.java
2016-03-01 17:20:15 +01:00
Nik Everett c7c8bb357a Merge pull request #16861 from nik9000/reindex_is_ready
Reindex required some parsing changes for search requests to support
differing defaults from the regular search api.
2016-03-01 10:02:48 -05:00
javanna 3d2a50c8ea [DOCS] Cat nodes api: document recently added http column 2016-03-01 15:36:55 +01:00
Clinton Gormley d928352df5 Update count.asciidoc
Added a note clarifying that deleted documents are not returned by `_cat/count`

Closes https://github.com/elastic/elasticsearch/issues/16008
2016-03-01 13:40:32 +01:00
Christoph Büscher a7053afdb9 Merge branch 'master' into feature-suggest-refactoring
Conflicts:
	docs/reference/migration/migrate_5_0.asciidoc
2016-03-01 12:37:05 +01:00
Clinton Gormley 3fe9c7736c Update zen.asciidoc
Remove docs saying that unicast hosts supports port ranges.

Relates to #15816
2016-03-01 12:19:34 +01:00
Clinton Gormley 9674cbbe62 Documented [] syntax for buckets_path
Closes #15707
2016-03-01 09:55:01 +01:00
Andrey Ryaguzov f744c3f724 Docs: Added migration description for custom analysis file path
Closes #15597
Closes #15556
2016-02-29 20:56:19 +01:00
David Pilato 26863a4d75 Fix: Change docs on "node client" to not use an in-memory node
Currently we suggesting users create a Node (using NodeBuilder in 2.x) to have a client that is capable of keeping up-to-date information. This is generally a bad idea as it means elasticsearch has no control over eg max heap size or gc settings, and is also problematic for users because they must deal with dependency collisions (and in 2.x+ dependencies of elasticsearch itself).

A better alternative, and what we should document, is to run a local elasticsearch server using bin/elasticsearch, and then use the transport client to connect to that local node. This local connection is virtually free, and allows the client code to be completely isolated from the elasticsearch process. Plugins are then also easy to deal with: just install them in elasticsearch as usual.

Related to #16679
2016-02-29 17:29:24 +01:00
Nik Everett fd729604ab [docs] Add breaking change notes for logging
ba5be0332d removed support for degrading
to slf4j and j.u.l but didn't document this as a breaking change because
it is only breaking for folks using Elasticsearch's jar as a java client.
People do that so this counts as a breaking change.

Also, if anyone was brave enough to try and replace log4j on an installed
version of Elasticsearch that will no longer work and this documents that
as well. It doens't get a full heading and instead lives with the java
client notes. Mostly because I can't imagine it worked consistently enough
for anyone to actually do it in the first place. We just never tested it
well enough to make sure we didn't break it after it was implemented.
2016-02-29 11:28:18 -05:00
Lee Hinman 8619df6311 [DOCS] Use https for rpm packages, add dnf instructions 2016-02-29 09:15:48 -07:00
Dominik Stadler eeb47a081b Add note about size being per shard here
It seems the current note via the comment is too less intrusive for people to notice and it leads to strange results otherwise.
2016-02-29 09:11:08 -07:00
Jason Tedor 0de66da2d4 Fix formatting in migration 5.0 docs 2016-02-29 11:05:42 -05:00
David Pilato b83b9d69c9 Change docs on "node client" to not use an in-memory node
Currently we suggesting users create a Node (using NodeBuilder in 2.x) to have a client that is capable of keeping up-to-date information. This is generally a bad idea as it means elasticsearch has no control over eg max heap size or gc settings, and is also problematic for users because they must deal with dependency collisions (and in 2.x+ dependencies of elasticsearch itself).

A better alternative, and what we should document, is to run a local elasticsearch server using bin/elasticsearch, and then use the transport client to connect to that local node. This local connection is virtually free, and allows the client code to be completely isolated from the elasticsearch process. Plugins are then also easy to deal with: just install them in elasticsearch as usual.

Closes #15383
2016-02-29 16:14:16 +01:00
Nik Everett 356b7cd12b Merge reindex to master because it is ready! 2016-02-29 10:08:29 -05:00
Nik Everett 1e1645d22b Move migrate_3_0 to migrate_5_0 2016-02-29 09:43:11 -05:00
Jason Tedor 5d8631848b Add note on Groovy dependencies to migration docs
This commit adds a note to the migration docs regarding the reduction of
the Groovy dependencies from the groovy-all artifact to the groovy
artifact that was previously done in
180ab2493e.

Closes #16858
2016-02-29 09:39:01 -05:00
David Pilato 7a42014909 Upgrade Azure Storage client to 4.0.0
We are using `2.0.0` today but Azure team now recommends:

```xml
<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>azure-storage</artifactId>
    <version>4.0.0</version>
</dependency>
```

This new version fix the timeout issues we have seen with azure storage although #15080 adds a timeout support.
Azure storage client 2.0.0 was not passing correctly this value when it was calling Azure services.

Note that the timeout is a server side timeout and not client side timeout.
It means that it will raise only a timeout when:

* upload of blob is complete
* if azure service is not able to process the blob (and store it) within a given time range.

In which case it will raise an exception which elasticsearch can deal with:

```
java.io.IOException
    at __randomizedtesting.SeedInfo.seed([91BC11AEF16E073F:6886FA5308FCE4D8]:0)
    at com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:643)
    at com.microsoft.azure.storage.blob.BlobOutputStream.writeBlock(BlobOutputStream.java:444)
    at com.microsoft.azure.storage.blob.BlobOutputStream.access$000(BlobOutputStream.java:53)
    at com.microsoft.azure.storage.blob.BlobOutputStream$1.call(BlobOutputStream.java:388)
    at com.microsoft.azure.storage.blob.BlobOutputStream$1.call(BlobOutputStream.java:385)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.microsoft.azure.storage.StorageException: Operation could not be completed within the specified time.
    at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
    at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:305)
    at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:175)
    at com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlockInternal(CloudBlockBlob.java:1006)
    at com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlock(CloudBlockBlob.java:978)
    at com.microsoft.azure.storage.blob.BlobOutputStream.writeBlock(BlobOutputStream.java:438)
    ... 9 more
```

The following code was used to test this against Azure platform:

```java
public void testDumb() throws URISyntaxException, StorageException, IOException, InvalidKeyException {
    String connectionString = "MY-AZURE-STRING";

    CloudStorageAccount storageAccount = CloudStorageAccount.parse(connectionString);
    CloudBlobClient client = storageAccount.createCloudBlobClient();
    client.getDefaultRequestOptions().setTimeoutIntervalInMs(1000);
    CloudBlobContainer container = client.getContainerReference("dumb");
    container.createIfNotExists();
    CloudBlockBlob blob = container.getBlockBlobReference("blob");

    File sourceFile = File.createTempFile("sourceFile", ".tmp");

    try {
        int fileSize = 10000000;

        byte[] buffer = new byte[fileSize];
        Random random = new Random();
        random.nextBytes(buffer);

        logger.info("Generate local file");
        FileOutputStream fos = new FileOutputStream(sourceFile);
        fos.write(buffer);
        fos.close();
        logger.info("End generate local file");

        FileInputStream fis = new FileInputStream(sourceFile);

        logger.info("Start uploading");
        blob.upload(fis, fileSize);
        logger.info("End uploading");

    }
    finally {
        if (sourceFile.exists()) {
            sourceFile.delete();
        }
    }
}
```

With 2.0.0, the above code was not raising any exception. With 4.0.0, the exception is now thrown correctly.

The default timeout is 5 minutes. See https://github.com/Azure/azure-storage-java/blob/master/microsoft-azure-storage/src/com/microsoft/azure/storage/core/Utility.java#L352-L375

Closes #12567.

Release notes from 2.0.0:

 * Removed deprecated table AtomPub support.
 * Removed deprecated constructors which take service clients in favor of constructors which take credentials.
 * Added support for "Add" permissions on Blob SAS.
 * Added support for "Create" permissions on Blob and File SAS.
 * Added support for IP Restricted SAS and Protocol SAS.
 * Added support for Account SAS to all services.
 * Added support for Minute and Hour Metrics to FileServiceProperties and added support for File Metrics to CloudAnalyticsClient.
 * Removed deprecated startCopyFromBlob() on CloudBlob. Use startCopy() instead.
 * Removed deprecated Credentials and StorageKey classes. Please use the appropriate methods on StorageCredentialsAccountAndKey instead.

 * Fixed a bug in table where a select on a non-existent field resulted in a null reference exception if the corresponding field in the TableEntity was not nullable.
 * Fixed a bug in table where JsonParser was automatically closing the response stream before it was completely drained causing socket exhaustion.
 * Fixed a bug in StorageCredentialsAccountAndKey.updateKey(String) which prevented valid keys from being set.
 * Added CloudBlobContainer.listBlobs(final String, final boolean) method.
 * Fixed a bug in blob where using AccessConditions on block blob uploads larger than 64MB done with the upload* methods or block blob uploads done openOutputStream with would fail if the blob did not already exist.
 * Added support for setting a proxy per request. Proxy can be set on an OperationContext instance and will be used when that instance is passed to the request method.

 * Added support for SAS to the Azure File service.
 * Added support for Append Blob.
 * Added support for Access Control Lists (ACL) to File Shares.
 * Added support for getting and setting of CORS rules to File service.
 * Added support for ShareStats to File Shares.
 * Added support for copying an Azure File to another Azure File or a Block Blob asynchronously, and aborting Azure File copy operations asynchronously.
 * Added support for copying a Blob to an Azure File asynchronously.
 * Added support for setting a maximum quota property on a File Share.
 * Removed deprecated AuthenticationScheme and its getter and setter. In the future only SharedKey will be used.
 * Removed deprecated getter/setters for all request option properties on the service clients. Please use the default request options getter/setters instead.
 * Removed getSubDirectoryReference() for blob directories and file directories. Use getDirectoryReference() instead.
 * Removed getEntityClass() in TableQuery. Please use getClazzType() instead.
 * Added client-side verification for lease duration and break periods.
 * Deprecated the setters in table for timestamp as this property is only modifiable by the service.
 * Deprecated startCopyFromBlob() on CloudBlob. Use startCopy() instead.
 * Deprecated the Credentials and StorageKey classes. Please use the appropriate methods on StorageCredentialsAccountAndKey instead.
 * Deprecated constructors which take service clients in favor of constructors which take credentials.
 * Fixed a bug where the DateBackwardCompatibility flag was not applied if set on the CloudTableClient default request options.
 * Changed library behavior to retry all exceptions thrown when parsing a response object.
 * Changed behavior to stop removing query parameters passed in with the resource URI if that URI contains a SAS token. Some query parameters such as comp, restype, snapshot and api-version will still be removed.
 * Added support for logging StringToSign to SharedKey and SAS.
 * **Added a connect timeout to prevent hangs when establishing the network connection.**
 * **Made performance enhancements to the BlobOutputStream class.**

 * Fixed a bug where maximum execution time was ignored for file, queue, and table services.
 * **Changed the socket timeout to be set to the service side timeout plus 5 minutes when maximum execution time is not set.**
 * **Changed the socket timeout to default to 5 minutes rather than infinite when neither service side timeout or maximum execution time are set.**
 * Fixed a bug where MD5 was calculated for commitBlockList even though UseTransactionalMD5 was set to false.
 * Fixed a bug where selecting fields that did not exist returned an error rather than an EntityProperty with a null value.
 * Fixed a bug where table entities with a single quote in their partition or row key could be inserted but not operated on in any other way.

 * Fixed a bug for all listing API's where next() would sometimes throw an exception if hasNext() had not been called even if there were more elements to iterate on.
 * Added sequence number to the blob properties. This is populated for page blobs.
 * Creating a page blob sets its length property.
 * Added support for page blob sequence numbers and sequence number access conditions.
 * Fixed a bug in abort copy where the lease access condition was not sent to the service.
 * Fixed an issue in startCopyFromBlob where if the URI of the source blob contained certain non-ASCII characters they would not be encoded appropriately. This would result in Authorization failures.
 * Fixed a small performance issue in XML serialization.
 * Fixed a bug in BlobOutputStream and FileOutputStream where flush added data to a request pool rather than immediately committing it to the Azure service.
 * Refactored to remove the blob, queue, and file package dependency on table in the error handling code.
 * Added additional client-side logging for REST requests, responses, and errors.

Closes #15976.
2016-02-29 15:00:34 +01:00
Christoph Büscher 379bd94c2a Merge branch 'master' into feature-suggest-refactoring 2016-02-29 14:06:30 +01:00
Clinton Gormley 812f03a33f Merge pull request #16842 from anhlqn/patch-1
Fix minor spelling
2016-02-29 01:32:42 +01:00
Clinton Gormley 8830817fa3 Merge pull request #16827 from ayushsangani/patch-3
Modify path of Servlet Transport
2016-02-29 00:59:44 +01:00
Clinton Gormley 2d56eed306 Merge pull request #16785 from dsem/patch-1
Fix python script filename extension
2016-02-28 23:04:41 +01:00
Clinton Gormley cd4b6d5e78 Update client.asciidoc
Remove comment about configuring the transport client in the config file

Relates to https://github.com/elastic/elasticsearch/pull/13383#issuecomment-185341750
2016-02-28 22:48:24 +01:00
Clinton Gormley 380ecd7604 Merge pull request #16777 from radar/patch-1
Add example for require_field_match to highlighting docs
2016-02-28 21:33:47 +01:00
Clinton Gormley 300554841e Merge pull request #16738 from robertlyson/patch-1
Update to serial differencing aggregation doc
2016-02-28 20:09:14 +01:00
Itamar Syn-Hershko 8ea6264f55 Format settings in discovery-ec2 docs
Closes #16846
2016-02-28 11:15:22 -05:00
Ali Beyad a7f6488216 Merge remote-tracking branch 'upstream/master' into
feature-suggest-refactoring
2016-02-26 17:21:12 -05:00
Nik Everett c38119bae9 Merge branch 'master' into feature/reindex 2016-02-26 16:59:54 -05:00
evanfreed 7ed30a9c00 Spelling
Corrected spelling.
2016-02-26 13:39:25 -05:00
Konstantin Delchev b451d5eb07 Fix typo 2016-02-26 13:36:13 -05:00
Martijn van Groningen ddc2b60c4b docs: fix incorrect grok pattern parameter names
Closes #16819
2016-02-26 06:38:25 -08:00
Jason Tedor c978335968 Remove es.useLinkedTransferQueue
This commit removes the system property "es.useLinkedTransferQueue" that
defaulted to false and was used to control the queue implementation used
in a few places.

Closes #16786
2016-02-23 17:29:15 -08:00
Christoph Büscher 78534847a3 Merge branch 'master' into feature-suggest-refactoring 2016-02-23 16:47:13 -08:00
Henrik Nordvik a996e21859 Document cpu usage in _cat/nodes
Closes #16775
2016-02-23 16:41:19 -08:00
Jason Tedor 6e840b39f5 Remove ability to disable Netty gathering writes
Java NIO has the notion of gathering writes. These are writes that
gather data from multiple buffers into a single channel. These gathering
writes in Netty have been enabled by default with the possibility to
disable them using "es.netty.gathering". This flag was added in case
having gathering writes on by default did not work out. We have not
published this ability and sufficient time has passed to render
judgement that using gathering writes is okay.

Closes #16774
2016-02-22 16:17:44 -08:00
David Pilato a55ad665da Merge pull request #16743 from dadoonet/pr/9209-cat-recovery-timeunit
[cat/recovery] Make recovery time a TimeValue()
2016-02-22 12:38:04 -08:00
Lee Hinman 99052c3fef Limit the accepted length of the _id
Elasticsearch should reject ids that are this long, to ensure a document
always remains retrievable for clients that impose a maximum URI length

Closes #16034
2016-02-22 12:34:18 -07:00
Christoph Büscher d888b5c267 Merge branch 'master' into feature-suggest-refactoring 2016-02-22 11:22:28 -08:00
Boaz Leskes 4a7980f96c Merge pull request #16766 from rstruber/patch-1
fix grammar in Total Shards Per Node docs
2016-02-22 08:42:42 -08:00
Jason Tedor fa885f2e96 Remove es.max-open-files flag
This commit removes the es.max-open-files flag as the same information
can be obtained from the cluster nodes info API, and is warn logged on
startup if it's set too low anyway.

Closes #16757
2016-02-21 21:01:08 -08:00
Lee Hinman 5451763935 [DOCS] Add blurb about `data_path` not needing to include index name
Resolves #11497
(node.enable_custom_paths is no longer used)
2016-02-21 13:11:45 -07:00
David Pilato dc32aaa8c8 [cat/recovery] Make recovery time a TimeValue(): add doc
Recovery `time` should be a TimeValue() to match other cat APIs.

Closes #9209
2016-02-20 16:30:54 -08:00
Robert 7844804874 Update to serial differencing aggregation doc
Hi,

`thirtieth_difference` should use `the_sum` metric as the `buckets_path`.
2016-02-20 12:13:02 +01:00
David Pilato 6c3f6778df Merge branch 'fix/16720-move-gce-settings' 2016-02-19 17:02:15 -08:00
David Pilato 90fba97a30 Moves GCE settings to the new infra
Closes #16720.
2016-02-19 17:00:39 -08:00
Clinton Gormley b59d3613df Merge pull request #16691 from gaelL/doc
snapshots doc: Repository Verification `verify` example
2016-02-19 11:12:27 -08:00
Yannick Welsch 4ea83740a2 Updates to resiliency documentation
Closes #16658
2016-02-19 09:48:33 -08:00
gaelL 58355430f6 snapshots doc: Repository Verification `verify` example fix
Replace `my_backup` in the example line by `s3_repository` to match
the example above.
2016-02-16 14:15:21 +01:00
Clinton Gormley aedfa10b43 Change docs example with script on/off to true/false
Closes #15469
2016-02-15 17:53:59 +01:00
Ali Beyad 5320d538f1 Merge remote-tracking branch 'upstream/master' into feature-suggest-refactoring 2016-02-15 11:03:33 -05:00
David Pilato 89a2505d01 Merge branch 'doc/16650-deprecate-attachments' 2016-02-15 16:40:47 +01:00
David Pilato 55d9b6878b Deprecate Mapper Attachment Plugin
Now that we have the ingest-attachment plugin (https://github.com/elastic/elasticsearch/pull/16490)  we should deprecate the mapper-attachment plugin.

Closes #16650.
2016-02-15 16:40:12 +01:00