Commit Graph

416 Commits

Author SHA1 Message Date
Boaz Leskes f7eb517eb8 Update the Flush API documentation (#33551)
The semantics of the API changed considerably since the documentation was written.

The main change is to remove references to memory reduction (this is related to refresh).
Instead, flush refers to recovery times. I also removed the references to trimming the translog
as the translog may be required for other purposes (operation history for ops based recovery
and complement ongoing file based recoveries).

Closes #32869
2019-01-15 11:45:10 +01:00
Julie Tibshirani 36a3b84fc9
Update the default for include_type_name to false. (#37285)
* Default include_type_name to false for get and put mappings.

* Default include_type_name to false for get field mappings.

* Add a constant for the default include_type_name value.

* Default include_type_name to false for get and put index templates.

* Default include_type_name to false for create index.

* Update create index calls in REST documentation to use include_type_name=true.

* Some minor clean-ups around the get index API.

* In REST tests, use include_type_name=true by default for index creation.

* Make sure to use 'expression == false'.

* Clarify the different IndexTemplateMetaData toXContent methods.

* Fix FullClusterRestartIT#testSnapshotRestore.

* Fix the ml_anomalies_default_mappings test.

* Fix GetFieldMappingsResponseTests and GetIndexTemplateResponseTests.

We make sure to specify include_type_name=true during xContent parsing,
so we continue to test the legacy typed responses. XContent generation
for the typeless responses is currently only covered by REST tests,
but we will be adding unit test coverage for these as we implement
each typeless API in the Java HLRC.

This commit also refactors GetMappingsResponse to follow the same appraoch
as the other mappings-related responses, where we read include_type_name
out of the xContent params, instead of creating a second toXContent method.
This gives better consistency in the response parsing code.

* Fix more REST tests.

* Improve some wording in the create index documentation.

* Add a note about types removal in the create index docs.

* Fix SmokeTestMonitoringWithSecurityIT#testHTTPExporterWithSSL.

* Make sure to mention include_type_name in the REST docs for affected APIs.

* Make sure to use 'expression == false' in FullClusterRestartIT.

* Mention include_type_name in the REST templates docs.
2019-01-14 13:08:01 -08:00
Jason Tedor 03be4dbaca
Introduce retention lease persistence (#37375)
This commit introduces the persistence of retention leases by persisting
them in index commits and recovering them when recovering a shard from
store.
2019-01-12 14:43:19 -08:00
Christoph Büscher c149bb8cc2
Support 'include_type_name' in RestGetIndicesAction (#37149)
This change adds support for the 'include_type_name' parameter for the
indices.get API. This parameter, which defaults to `false` starting in 7.0,
changes the response to not include the indices type names any longer.

If the parameter is set in the request, we additionally emit a deprecation
warning since using the parameter should be only temporarily necessary while
adapting to the new response format and we will remove it with the next major
version.
2019-01-09 14:17:17 +01:00
Josh Soref edb48321ba [DOCS] Various spelling corrections (#37046) 2019-01-07 14:44:12 +01:00
lcawl 32bed098bb [DOCS] Synchs titles of X-Pack APIs 2018-12-20 10:27:24 -08:00
Armin Braun c5b3ac5578
SNAPSHOTS: Allow Parallel Restore Operations (#36397)
* Enable parallel restore operations
* Add uuid to restore in progress entries to uniquely identify them
* Adjust restore in progress entries to be a map in cluster state
* Added tests for:
   * Parallel restore from two different snapshots
   * Parallel restore from a single snapshot to different indices to test uuid identifiers are correctly used by `RestoreService` and routing allocator
   * Parallel restore with waiting for completion to test transport actions correctly use uuid identifiers
2018-12-14 11:39:23 +01:00
Ryan Ernst 254d1e8f22
Core: Add backcompat for joda time formats (#36531)
This commit adds deprecation warnings when using format specifiers with
joda data formats that will change with java time. It also adds the "8"
prefix which may be used to force the new java time format parsing.
2018-12-13 12:26:51 -08:00
Nhat Nguyen 51800de2a8
Enable soft-deletes by default on 7.0.0 or later (#36141)
This change enables soft-deletes by default on ES 7.0.0 or later.

Relates #33222

Co-authored-by: Jason Tedor <jason@tedor.me>
2018-12-11 18:58:49 -05:00
Peter Dyson 63a9571c4c
Explain nondeterministic template merging (#35405) 2018-11-30 14:56:32 +10:00
Simon Willnauer 29ef442841
Add a `_freeze` / `_unfreeze` API (#35592)
This commit adds a rest endpoint for freezing and unfreezing an index.
Among other cleanups mainly fixing an issue accessing package private APIs
from a plugin that got caught by integration tests this change also adds
documentation for frozen indices.
Note: frozen indices are marked as `beta` and available as a basic feature.

Relates to #34352
2018-11-20 08:03:24 +01:00
Senthil e27a0d5b0d Docs: Align prose with snippet (#34839)
Modified example text to include multiple index patterns used in the code
2018-10-25 16:51:18 -04:00
Frederik Bosch 183c32d4c3 [DOCS] Force Merge: clarify execution and storage requirements (#33882) 2018-10-23 12:25:04 +02:00
Julie Tibshirani f854330e06
Make sure to use the type _doc in the REST documentation. (#34662)
* Replace custom type names with _doc in REST examples.
* Avoid using two mapping types in the percolator docs.
* Rename doc -> _doc in the main repository README.
* Also replace some custom type names in the HLRC docs.
2018-10-22 11:54:04 -07:00
Julie Tibshirani fbb9ac34f9
Deprecate type exists requests. (#34663) 2018-10-22 08:46:11 -07:00
Christoph Büscher 7cc3ac0fc4
[Docs] Add _cache/clear parameter examples (#34614)
Add example for selectively clearing just the request, query or fielddata cache
and for selectively clearing the cache for specific fields.

Closes #34287
2018-10-19 00:44:37 +02:00
Tal Levy 1e0fcebfe1
update rollover to leverage write-alias semantics (#32216)
Rollover should not swap aliases when `is_write_index` is set to `true`.
Instead, both the new and old indices should have the rollover alias,
with the newly created index as the new write index

Updates Rollover to leverage the ability to preserve aliases and swap which is the write index.

Historically, Rollover would swap which index had the designated alias for writing documents against. This required users to keep a separate read-alias that enabled reading against both rolled over and newly created indices, whiles the write-alias was being re-assigned at every rollover.

With the ability for aliases to designate a write index, Rollover can be a bit more flexible with its use of aliases.

Updates include:

- Rollover validates that the target alias has a write index (the index that is being rolled over). This means that the restriction that aliases only point to one index is no longer necessary.
- Rollover explicitly (and atomically) swaps which index is the write-index by explicitly assigning the existing index to have `is_write_index: false` and have the newly created index have its rollover alias as `is_write_index: true`. This is only done when `is_write_index: true` on the write index. Default behavior of removing the alias from the rolled over index stays when `is_write_index` is not explicitly set

Relevant things that are staying the same:

- Rollover is rejected if there exist any templates that match the newly-created index and configure the rollover-alias
   - I think this existed to prevent the situation where an alias pointed to two indices for a short while. Although this can technically be relaxed, the specific cases that are safe are really particular and difficult to reason, so leaving the broad restriction sounds good
2018-07-30 14:32:55 -07:00
Julie Tibshirani 15ff3da653
Add support for field aliases. (#32172)
* Add basic support for field aliases in index mappings. (#31287)
* Allow for aliases when fetching stored fields. (#31411)
* Add tests around accessing field aliases in scripts. (#31417)
* Add documentation around field aliases. (#31538)
* Add validation for field alias mappings. (#31518)
* Return both concrete fields and aliases in DocumentFieldMappers#getMapper. (#31671)
* Make sure that field-level security is enforced when using field aliases. (#31807)
* Add more comprehensive tests for field aliases in queries + aggregations. (#31565)
* Remove the deprecated method DocumentFieldMappers#getFieldMapper. (#32148)
2018-07-18 09:33:09 -07:00
Piotr Prądzyński 99030e7af5 Docs: Inconsistency between description and example (#31858) 2018-07-06 12:44:20 -04:00
Tal Levy 3b70e943eb
add is-write-index flag to aliases (#30942)
This commit adds the is-write-index flag for aliases.
It allows requests to set the flag, and responses to display the flag.
It does not validate and/or affect any indexing/getting/updating behavior
of Elasticsearch -- this will be done in a follow-up PR.
2018-06-15 08:45:29 -07:00
Itamar Syn-Hershko 526f906927 Remove usage of explicit type in docs (#29667) 2018-06-04 09:42:29 -04:00
lipsill a8c643e833 [Docs] Fix a typo in Create Index naming limitation (#30891) 2018-06-01 18:15:24 +02:00
Zachary Tong 6909a05f3d
[DOCS] Document index name limitations (#30826)
Also tidy up the docs a bit, there's no yaml example anymore, etc
2018-05-25 10:21:09 -04:00
Adam Chalkley 7cc38ab45a Fix default shards count in create index docs (#30747)
Update the default number of primary shards to match doc update work
done in #30539.
2018-05-20 14:59:28 -04:00
Jason Tedor d68c44b76c
Default copy settings to true and deprecate on the REST layer (#30598)
This commit defaults the copy_settings REST parameter to the shrink and
split APIs to true, and deprecates the parameter.
2018-05-18 10:12:08 -04:00
Jason Tedor 4a4e3d70d5
Default to one shard (#30539)
This commit changes the default out-of-the-box configuration for the
number of shards from five to one. We think this will help address a
common problem of oversharding. For users with time-based indices that
need a different default, this can be managed with index templates. For
users with non-time-based indices that find they need to re-shard with
the split API in place they no longer need to resort only to
reindexing.

Since this has the impact of changing the default number of shards used
in REST tests, we want to ensure that we still have coverage for issues
that could arise from multiple shards. As such, we randomize (rarely)
the default number of shards in REST tests to two. This is managed via a
global index template. However, some tests check the templates that are
in the cluster state during the test. Since this template is randomly
there, we need a way for tests to skip adding the template used to set
the number of shards to two. For this we add the default_shards feature
skip. To avoid having to write our docs in a complicated way because
sometimes they might be behind one shard, and sometimes they might be
behind two shards we apply the default_shards feature skip to all docs
tests. That is, these tests will always run with the default number of
shards (one).
2018-05-14 12:22:35 -04:00
Jason Tedor 901436148b Adjust copy settings versions
This commit adjusts the versions on the copy settings behavior now
that the default behavior is configured in 7.0.0.
2018-05-13 22:23:13 -04:00
Jason Tedor 593fdd40ed
Deprecate not copy settings and explicitly disallow (#30404)
We want copying settings to be the default behavior. This commit
deprecates not copying settings, and disallows explicitly not copying
settings. This gives users a transition path to the future default
behavior.
2018-05-13 10:30:05 -04:00
Mueed Chaudhry bf141a3fd1 [docs] add warning for read-write indices in force merge documentation (#28869) 2018-05-09 18:53:55 +02:00
javanna c9f5a7893b [DOCS] convert forcemerge snippet
Relates to #30113
2018-05-07 16:09:03 +02:00
Matija Bruncic e5653e635d Update forcemerge.asciidoc (#30113) 2018-05-07 14:56:12 +02:00
Jason Tedor 5de6f4ff7b Adjust copy settings on resize BWC version
This commit adjusts the BWC version for copy settings on resize
operations after the behavior was backported to 6.x.
2018-05-01 08:49:16 -04:00
Jason Tedor 50535423ff
Allow copying source settings on resize operation (#30255)
Today when an index is created from shrinking or splitting an existing
index, the target index inherits almost none of the source index
settings. This is surprising and a hassle for operators managing such
indices. Given this is the default behavior, we can not simply change
it. Instead, we start by introducing the ability to copy settings. This
flag can be set on the REST API or on the transport layer and it has the
behavior that it copies all settings from the source except non-copyable
settings (a property of a setting introduced in this
change). Additionally, settings on the request will always override.

This change is the first step in our adventure:
 - this flag is added here in 7.0.0 and immediately deprecated
 - this flag will be backported to 6.4.0 and remain deprecated
 - then, we will remove the ability to set this flag to false in 7.0.0
 - finally, in 8.0.0 we will remove this flag and the only behavior will
   be for settings to be copied
2018-05-01 08:48:19 -04:00
Jason Tedor 811f5b4efc
Do not ignore request analysis/similarity on resize (#30216)
Today when a resize operation is performed, we copy the analysis,
similarity, and sort settings from the source index. It is possible for
the resize request to include additional index settings including
analysis, similarity, and sort settings. We reject sort settings when
validating the request. However, we silently ignore analysis and
similarity settings on the request that are already set on the source
index. Since it is possible to change the analysis and similarity
settings on an existing index, this should be considered a bug and the
sort of leniency that we abhor. This commit addresses this bug by
allowing the request analysis/similarity settings to override the
existing analysis/similarity settings on the target.
2018-04-30 07:31:36 -04:00
Adrien Grand 6a6c0ea5e6
Add an `include_type_name` option. (#29453)
This adds an `include_type_name` option to the `indices.create`,
`indices.get_mapping` and `indices.put_mapping` APIs, which defaults to `true`.
When set to `false`, then mappings will be returned directly in the body of
the `indices.get_mapping` API, without keying them by the type name, the
`indices.create` will expect mappings directly under the `mappings` key, and
the `indices.put_mapping` will use `_doc` as a type name and fail if a `type`
is provided explicitly.

Relates #15613
2018-04-11 15:54:16 +02:00
Boaz Leskes eb8b31746a Move trimming unsafe commits from engine ctor to store (#29260)
As follow up to #28245 , this PR removes the logic for selecting the 
right start commit from the Engine constructor in favor of explicitly
trimming them in the Store, before the engine is opened. This makes the
constructor in engine follow standard Lucene semantics and use the last
commit.

Relates #28245
Relates #29156
2018-03-29 13:35:57 -04:00
Boaz Leskes f5d4550e93
Fold EngineDiskUtils into Store, for better lock semantics (#29156)
#28245 has introduced the utility class`EngineDiskUtils` with a set of methods to prepare/change
translog and lucene commit points. That util class bundled everything that's needed to create and
empty shard, bootstrap a shard from a lucene index that was just restored etc. 

In order to safely do these manipulations, the util methods acquired the IndexWriter's lock. That
would sometime fail due to concurrent shard store fetching or other short activities that require the
files not to be changed while they read from them. 

Since there is no way to wait on the index writer lock, the `Store` class has other locks to make
sure that once we try to acquire the IW lock, it will succeed. To side step this waiting problem, this
PR folds `EngineDiskUtils` into `Store`. Sadly this comes with a price - the store class doesn't and
shouldn't know about the translog. As such the logic is slightly less tight and callers have to do the
translog manipulations on their own.
2018-03-26 14:08:03 +02:00
Adrien Grand 8f9d2ee4e2
Reject updates to the `_default_` mapping. (#29165)
This will reject mapping updates to the `_default_` mapping with 7.x indices
and still emit a deprecation warning with 6.x indices.

Relates #15613
Supersedes #28248
2018-03-21 10:44:11 +01:00
Ryan Ernst 52a517e216
Docs: Add example of resetting index setting (#29048)
This commit adds an example using `null` to reset an index settings.

closes #22870
2018-03-19 20:04:58 -07:00
Adrien Grand 708c06896b
Explain why Elasticsearch doesn't support incremental resharding. (#29082)
I have seen this question a couple times already, most recently at
https://twitter.com/dimosr7/status/973872744965332993

I tried to keep the explanation as simple as I could, which is not always easy
as this is a matter of trade-offs.
2018-03-16 16:50:58 +01:00
Boaz Leskes bf65cb4914
Untangle Engine Constructor logic (#28245)
Currently we have a fairly complicated logic in the engine constructor logic to deal with all the 
various ways we want to mutate the lucene index and translog we're opening.

We can:
1) Create an empty index
2) Use the lucene but create a new translog
3) Use both
4) Force a new history uuid in all cases.

This leads complicated code flows which makes it harder and harder to make sure we cover all the 
corner cases. This PR tries to take another approach. Constructing an InternalEngine always opens 
things as they are and all needed modifications are done by static methods directly on the 
directory, one at a time.
2018-03-14 20:59:47 +01:00
Joost Rothweiler 02a611663a Indices PUT Mapping API docs: Remove mapping type user and rephrase first sentence (#28998)
The current docs on [Indices APIs: PUT Mapping](https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-put-mapping.html) suggests that a having number of different mapping types per index is still possible in elasticsearch versions > 6.0.0 although they have been [removed](https://www.elastic.co/guide/en/elasticsearch/reference/current/removal-of-types.html). The console code has already been updated accordingly but notes (2) and (3) on the console code still name the `user` mapping type.

This PR updates the list with notes after the console code, as well as the first sentence of the docs
to avoid confusion. Also, I have removed the second command from the console code as it no
longer holds any value if the docs are solely on the `_doc` mapping.
2018-03-13 09:58:47 +01:00
Boaz Leskes beb55d148a
Simplify the Translog constructor by always expecting an existing translog (#28676)
Currently the Translog constructor is capable both of opening an existing translog and creating a
new one (deleting existing files). This PR separates these two into separate code paths. The
constructors opens files and a dedicated static methods creates an empty translog.
2018-02-15 09:24:09 +01:00
Arne Neumann 3b41b98ee4 documentation fix: index exists is ambiguous (#28460) 2018-02-01 13:22:32 -08:00
Adrien Grand 1b660821a2
Allow `_doc` as a type. (#27816)
Allowing `_doc` as a type will enable users to make the transition to 7.0
smoother since the index APIs will be `PUT index/_doc/id` and `POST index/_doc`.
This also moves most of the documentation to `_doc` as a type name.

Closes #27750
Closes #27751
2017-12-14 17:47:53 +01:00
Mayya Sharipova c6b73239ae
Limit the number of tokens produced by _analyze (#27529)
Add an index level setting `index.analyze.max_token_count` to control
the number of generated tokens in the  _analyze endpoint.
Defaults to 10000.

Throw an error if the number of generated tokens exceeds this limit.

Closes #27038
2017-11-30 11:54:39 -05:00
Simon Willnauer fadbe0de08
Automatically prepare indices for splitting (#27451)
Today we require users to prepare their indices for split operations.
Yet, we can do this automatically when an index is created which would
make the split feature a much more appealing option since it doesn't have
any 3rd party prerequisites anymore.

This change automatically sets the number of routinng shards such that
an index is guaranteed to be able to split once into twice as many shards.
The number of routing shards is scaled towards the default shard limit per index
such that indices with a smaller amount of shards can be split more often than
larger ones. For instance an index with 1 or 2 shards can be split 10x
(until it approaches 1024 shards) while an index created with 128 shards can only
be split 3x by a factor of 2. Please note this is just a default value and users
can still prepare their indices with `index.number_of_routing_shards` for custom
splitting.

NOTE: this change has an impact on the document distribution since we are changing
the hash space. Documents are still uniformly distributed across all shards but since
we are artificually changing the number of buckets in the consistent hashign space
document might be hashed into different shards compared to previous versions.

This is a 7.0 only change.
2017-11-23 09:48:54 +01:00
Simon Willnauer bd7efa908a Add ability to split shards (#26931)
This change adds a new `_split` API that allows to split indices into a new
index with a power of two more shards that the source index.  This API works
alongside the `_shrink` API but doesn't require any shard relocation before
indices can be split.

The split operation is conceptually an inverse `_shrink` operation since we
initialize the index with a _syntetic_ number of routing shards that are used
for the consistent hashing at index time. Compared to indices created with
earlier versions this might produce slightly different shard distributions but
has no impact on the per-index backwards compatibility.  For now, the user is
required to prepare an index to be splittable by setting the
`index.number_of_routing_shards` at index creation time.  The setting allows the
user to prepare the index to be splittable in factors of
`index.number_of_routing_shards` ie. if the index is created with
`index.number_of_routing_shards: 16` and `index.number_of_shards: 2` it can be
split into `4, 8, 16` shards. This is an intermediate step until we can make
this the default. This also allows us to safely backport this change to 6.x.

The `_split` operation is implemented internally as a DeleteByQuery on the
lucene level that is executed while the primary shards execute their initial
recovery. Subsequent merges that are triggered due to this operation will not be
executed immediately. All merges will be deferred unti the shards are started
and will then be throttled accordingly.

This change is intended for the 6.1 feature release but will not support pre-6.1
indices to be split unless these indices have been shrunk before. In that case
these indices can be split backwards into their original number of shards.
2017-11-06 11:37:55 +01:00
Nhat c7ce5a07f2
Add size-based condition to the index rollover API (#27160)
This is to add a max_size condition to the index rollover API. We use
a totalSizeInBytes from DocsStats to evaluate this condition.

Closes #27004
2017-11-04 19:51:48 -04:00
Jason Tedor a7895839a0 Reformat paragraph in template docs to 80 columns
This commit reformats a paragraph in the template docs to fit in 80
columns as for the rest of the doc, and as-is a standard that we loosely
adhere to.
2017-10-12 17:52:43 -04:00
Pius 1125bc635c Clarify settings and template on create index
This commit clarifies the interaction between settings specified in a
create index request, and those that would come from any templates that
apply to the create index request.

Relates #26994
2017-10-12 17:48:57 -04:00
Alexander Kazakov ff737a880c Add wait_for_active_shards parameter to index open command (#26682)
Adds the wait_for_active_shards parameter to the index open command. Similar to the index creation command, the index open command will now, by default, wait until the primaries have been allocated.

Closes #20937
2017-09-22 11:15:03 +02:00
Boaz Leskes 1ca0b5e9e4 Introduce a History UUID as a requirement for ops based recovery (#26577)
The new ops based recovery, introduce as part of  #10708, is based on the assumption that all operations below the global checkpoint known to the replica do not need to be synced with the primary. This is based on the guarantee that all ops below it are available on primary and they are equal. Under normal operations this guarantee holds. Sadly, it can be violated when a primary is restored from an old snapshot. At the point the restore primary can miss operations below the replica's global checkpoint, or even worse may have total different operations at the same spot. This PR introduces the notion of a history uuid to be able to capture the difference with the restored primary (in a follow up PR).

The History UUID is generated by a primary when it is first created and is synced to the replicas which are recovered via a file based recovery. The PR adds a requirement to ops based recovery to make sure that the history uuid of the source and the target are equal. Under normal operations, all shard copies will stay with that history uuid for the rest of the index lifetime and thus this is a noop. However, it gives us a place to guarantee we fall back to file base syncing in special events like a restore from snapshot (to be done as a follow up) and when someone calls the truncate translog command which can go wrong when combined with primary recovery (this is done in this PR).

We considered in the past to use the translog uuid for this function (i.e., sync it across copies) and thus avoid adding an extra identifier. This idea was rejected as it removes the ability to verify that a specific translog really belongs to a specific lucene index. We also feel that having a history uuid will serve us well in the future.
2017-09-14 21:25:02 +03:00
Tanguy Leroux f95dec797d [Docs] Convert more doc snippets (#26359)
This commit converts some remaining doc snippets so that they are now
testable.
2017-08-28 11:23:09 +02:00
markwalkom 746487c3f3 Update templates.asciidoc (#26036)
Dropped in a few links to index settings and mappings to make things easier to jump to.
2017-08-08 11:29:11 +02:00
Clinton Gormley ff4a2519f2 Update experimental labels in the docs (#25727)
Relates https://github.com/elastic/elasticsearch/issues/19798

Removed experimental label from:
* Painless
* Diversified Sampler Agg
* Sampler Agg
* Significant Terms Agg
* Terms Agg document count error and execution_hint
* Cardinality Agg precision_threshold
* Pipeline Aggregations
* index.shard.check_on_startup
* index.store.type (added warning)
* Preloading data into the file system cache
* foreach ingest processor
* Field caps API
* Profile API

Added experimental label to:
* Moving Average Agg Prediction


Changed experimental to beta for:
* Adjacency matrix agg
* Normalizers
* Tasks API
* Index sorting

Labelled experimental in Lucene:
* ICU plugin custom rules file
* Flatten graph token filter
* Synonym graph token filter
* Word delimiter graph token filter
* Simple pattern tokenizer
* Simple pattern split tokenizer

Replaced experimental label with warning that details may change in the future:
* Analysis explain output format
* Segments verbose output format
* Percentile Agg compression and HDR Histogram
* Percentile Rank Agg HDR Histogram
2017-07-18 14:06:22 +02:00
Clinton Gormley 0170e0e8d3 Remove usage of multi-types from the docs and added a page explaining type removal (#25543)
Closes #25401
2017-07-05 12:30:19 +02:00
Wesley Merkel 7994703903 [Docs] Fix some errors in the index aliases docs (#25449) 2017-07-05 11:16:17 +02:00
Jun Ohtani 6894ef6057 [Analysis] Support normalizer in request param (#24767)
* [Analysis] Support normalizer in request param

Support normalizer param
Support custom normalizer with char_filter/filter param

Closes #23347
2017-07-04 19:16:56 +09:00
javanna df5640efd7 [DOCS] delete index no longer supports specifying aliases 2017-06-19 10:30:05 +02:00
Jason Tedor dcf57f296e Fix get mappings HEAD requests
Get mappings HEAD requests incorrectly return a content-length header of
0. This commit addresses this by removing the special handling for get
mappings HEAD requests, and just relying on the general mechanism that
exists for handling HEAD requests in the REST layer.

Relates #23192
2017-06-11 14:58:56 -04:00
Sergey Novikov 7c8657df0e Return the index name on a create index response
This commit modifies the create index response so that it includes the
index name.

Relates #25139
2017-06-09 13:47:47 -04:00
Lee Hinman a32d1b91fa Remove comma-separated feature parsing for GetIndicesAction
This removes the parsing of things like `GET /idx/_aliases,_mappings`, instead,
a user must choose between retriving all index metadata with `GET /idx`, or only
a specific form such as `GET /idx/_settings`.

Relates to (and is a prerequisite of) #24437
2017-06-02 14:43:38 -06:00
Brian Lesperance 959990728b Docs: Fix grammar in aliases doc (#24852) 2017-05-24 10:18:25 -04:00
glefloch 59dd4d288a documentation of preserve existing settings 2017-05-08 12:05:01 +02:00
Zachary Tong 735986c140
[DOCS] Tweak doc test to sync_flush
The response is attempting to illustrate the sync_id marker, but in
the test the index is too "fresh" to have a sync marker. So the test
needs to execute a sync flush behind the scenes so that the marker
is present
2017-05-01 15:15:32 -04:00
Zachary Tong 38273709b5 CONSOLEify some more Indices APIs (#24375)
* CONSOLEify doc testing for some more Indices APIs

Related to #18160
2017-05-01 13:56:39 -04:00
Adrien Grand 1be2800120 Only allow one type on 7.0 indices (#24317)
This adds the `index.mapping.single_type` setting, which enforces that indices
have at most one type when it is true. The default value is true for 6.0+ indices
and false for old indices.

Relates #15613
2017-04-27 08:43:20 +02:00
Emir Beganović f4f6ce4b91 Docs: fix some case problems in aliases.asciidoc (#23657)
Make more things uppercase.
2017-04-17 21:45:19 -04:00
Himanshu Bhandoh c53a2b870e Fixing typo in spelling of rollover (#24146)
rolllover -> rollover
2017-04-17 21:31:11 -04:00
Lee Hinman 5cace8e48a Remove shadow replicas
Resolves #22024
2017-04-11 11:26:26 -06:00
Clinton Gormley 5eba90f37c Doc values can no longer be updated
Closes #23820
2017-03-31 15:29:01 +02:00
Lee Hinman ac3cc9641b
[DOCS] Console-ify shadow replica documentation
Relates to #23001
2017-02-15 11:58:53 -07:00
Ryan Ernst c91848e6a7 Docs: Consoleify cluster and indices settings docs (#23030)
relates #23001
2017-02-10 14:57:43 -08:00
Yannick Welsch 9154686623 Remove legacy primary shard allocation mode based on versions (#23016)
Elasticsearch v5.0.0 uses allocation IDs to safely allocate primary shards whereas prior versions of ES used a version-based mode instead. Elasticsearch v5 still has support for version-based primary shard allocation as it needs to be able to load 2.x shards. ES v6 can drop the legacy support.
2017-02-08 10:00:55 +01:00
Jim Ferenczi bbf62e3472 CONSOLify indices/analyze.asciidoc and search/field-stats.asciidoc
Relates #23001
2017-02-07 20:15:09 +01:00
Jim Ferenczi 4876448e39 Consilify get-field-mapping docs (#22936)
This change also removes the reference to the difference bewteen full name and index name.
They are always the same since 2.x and `name` does not refer anymore to `author.name` automatically.
A simple pattern must be used instead.
Remove redundant code that checks the field name twice.
2017-02-03 10:04:31 +01:00
Shubham Aggarwal d80e05f4d2 Minor fix in Create Index desc (#22877) 2017-01-31 12:53:15 +01:00
Boaz Leskes 1227044ddd Add a deprecation notice to shadow replicas (#22647)
Relates to #22024

On top of documentation, the PR adds deprecation loggers and deals with the resulting warning headers.

The yaml test is set exclude versions up to 6.0. This is need to make sure bwc tests pass until this is backported to 5.2.0 . Once that's done, I will change the yaml test version limits
2017-01-18 12:28:09 +01:00
Boaz Leskes f88ab76067 Revert "Add a deprecation notice to shadow replicas (#22025)"
This reverts commit 0da190234c.
2017-01-16 16:15:41 +01:00
Boaz Leskes 0da190234c Add a deprecation notice to shadow replicas (#22025)
Also adds deprecation logging.

See #22024
2017-01-16 15:40:05 +01:00
Jason Tedor 2713549533 Use reader for doc stats
Today we try to pull stats from index writer but we do not get a
consistent view of stats. Under heavy indexing, this inconsistency can
be very skewed indeed. In particular, it can lead to the number of
deleted docs being reported as negative and this leads to serialization
issues. Instead, we should provide a consistent view of the stats by
using an index reader.

Relates #22317
2016-12-23 09:44:56 -05:00
Clinton Gormley 5555e85619 Document that the PUT mapping API with the _default_ type overwrites instead of merging
Closes #8215
2016-11-26 12:43:56 +01:00
Clinton Gormley 284dedfb5f Update get-settings.asciidoc
Fix syntax for filtering settings by name

Closes https://github.com/elastic/elasticsearch/issues/21713
2016-11-22 15:10:09 +01:00
Jeffery Bradberry 9322c4fe62 The alias is switched to point to the new index (#21512)
If the index satisfies the specified conditions then a new index is created and the alias is switched to point to the new index.
2016-11-19 16:47:39 +01:00
Sakthipriyan Vairamani 52d4cd504a minor grammatical improvements (#21640) 2016-11-18 08:55:04 -07:00
Jason Tedor f5ac0e5076 Remove lenient stats parsing
Today when parsing a stats request, Elasticsearch silently ignores
incorrect metrics. This commit removes lenient parsing of stats requests
for the nodes stats and indices stats APIs.

Relates #21417
2016-11-15 12:17:26 -05:00
Alexander Lin 0219a211d3 Allows multiple patterns to be specified for index templates (#21009)
* Allows for an array of index template patterns to be provided to an
index template, and rename the field from 'template' to 'index_pattern'.

Closes #20690
2016-11-10 18:00:30 -05:00
Clinton Gormley 860efb2ad2 Replace all index date-math examples with the URI encoded form
Closes #21454
2016-11-10 12:23:19 +01:00
Clinton Gormley ef2453360e Fixed rollover docs test for date math indices 2016-11-03 13:48:34 +01:00
Clinton Gormley 6902eafbfe Specify date math with URI encoding in rollover API docs
Closes #21181
2016-11-03 10:48:50 +01:00
Jun Ohtani a66c76eb44 Merge pull request #20704 from johtani/remove_request_params_in_analyze_api
Removing request parameters in _analyze API
2016-10-27 17:43:18 +09:00
Chris Earle c2894d2362 Add missing '// CONSOLE' to code snippet 2016-10-18 12:57:03 -04:00
Chris Earle 9cf7214380 [DOCS] Add "version" to template and pipeline docs (#20407)
* [DOCS] Add "version" to template and pipeline docs

This adds details about the "version" to both the template and pipeline pages.
2016-10-18 11:56:18 -04:00
Clinton Gormley 43ad8de75b Fixed typo in index rollover docs
Closes #20935
2016-10-17 17:56:54 +02:00
Clinton Gormley a7d8de14bb Removed the upgrade API docs
Relates to #20675
2016-10-11 12:21:46 +02:00
Lucas Bremgartner 4bf45e53a5 Documentation: comments in index templates (#20701)
Fixes #20513
2016-10-08 12:38:45 +02:00
Clinton Gormley eb62f90578 Document date math use in the rollover API (#20780)
* Document date math use in the rollover API

Relates to #20709
2016-10-07 15:49:28 +02:00
Jun Ohtani 370f0b885e Removing request parameters in _analyze API
Remove request params in _analyze API without index param
Change rest-api-test using JSON
Change docs using JSON

Closes #20246
2016-10-07 16:23:24 +09:00
Nik Everett bfd072bc10 Switch more docs to CONSOLE
Related to #18160
2016-09-02 18:22:30 -04:00
Jun Ohtani c4759bcc02 Merge pull request #20285 from johtani/fix/remove_token_filter_param_in_analyze_api
Remove `token_filter` in _analyze API
2016-09-03 02:03:51 +09:00
Clinton Gormley 4885709e10 Update rollover-index.asciidoc
Fixed weirdly formatted callouts in rollover docs
2016-09-02 16:08:34 +02:00
Nik Everett c8b984aee9 Add CONSOLE to more docs
Relates to #18160
2016-09-01 17:08:18 -04:00
Nik Everett 4b219d15d2 Add CONSOLE to a few snippets in reference docs
This allows them to be run in Console and adds them to the list of
docs that are automatically tested as part of the build.

Relates to #18160
2016-09-01 13:08:36 -04:00
Jun Ohtani 3d9f8ed764 Remove `token_filter` in _analyze API
Remove the param and change docs

Closes #20283
2016-09-02 01:36:45 +09:00
Nik Everett 1128776b3c [docs] Remove coming in 2.0.0
2.0.0's been out for a long time.

Closes #20268
2016-08-31 14:43:04 -04:00
Ali Beyad 4641254ea6 Parameter improvements to Cluster Health API wait for shards (#20223)
* Params improvements to Cluster Health API wait for shards

Previously, the cluster health API used a strictly numeric value
for `wait_for_active_shards`. However, with the introduction of
ActiveShardCount and the removal of write consistency level for
replication operations, `wait_for_active_shards` is used for
write operations to represent values for ActiveShardCount. This
commit moves the cluster health API's usage of `wait_for_active_shards`
to be consistent with its usage in the write operation APIs.

This commit also changes `wait_for_relocating_shards` from a
numeric value to a simple boolean value `wait_for_no_relocating_shards`
to set whether the cluster health operation should wait for
all relocating shards to complete relocation.

* Addresses code review comments

* Don't be lenient if `wait_for_relocating_shards` is set
2016-08-31 11:58:19 -04:00
Nik Everett df73292256 Add an alias action to delete an index
While removing an index isn't actually an alias action, if we add
an alias action that deletes an index then we can delete and index
and add an alias with the same name as the index atomically, in
the same cluster state update.

Closes #20064
2016-08-30 10:15:21 -04:00
Adrien Grand a4ea7e7223 Switch indices.exists_type from `{index}/{type}` to `{index}/_mapping/{type}`. #20055
This will help remove types as we will need `{index}/{id}` to tell whether a
document exists.

Relates #15613
2016-08-19 09:18:24 +02:00
Adrien Grand d894db1590 Only use `PUT` for index creation, not POST. #20001
Currently both `PUT` and `POST` can be used to create indices. This commit
removes support for `POST index_name` so that we can use it to index documents
with auto-generated ids once types are removed.

Relates #15613
2016-08-17 10:15:42 +02:00
polyfractal 2c1b9b67db [DOCS] Add missing index to test setup
#19884 fixed the aliases, but didn't add `index2` to the test setup which causes the
alias command to fail
2016-08-16 11:55:56 -04:00
Ludo eea1bc719b Swap a index for the same alias (#19884)
In the example there was a alias removed and then a different alias created for the same index, but I think actually swapping a index by another one for the same alias would make more sense as an example here.
2016-08-16 15:24:42 +02:00
Nik Everett 1e587406d8 Fail yaml tests and docs snippets that get unexpected warnings
Adds `warnings` syntax to the yaml test that allows you to expect
a `Warning` header that looks like:
```
    - do:
        warnings:
            - '[index] is deprecated'
            - quotes are not required because yaml
            - but this argument is always a list, never a single string
            - no matter how many warnings you expect
        get:
            index:    test
            type:    test
            id:        1
```

These are accessible from the docs with:
```
// TEST[warning:some warning]
```

This should help to force you to update the docs if you deprecate
something. You *must* add the warnings marker to the docs or the build
will fail. While you are there you *should* update the docs to add
deprecation warnings visible in the rendered results.
2016-08-04 15:23:05 -04:00
Ali Beyad a21dd80f1b Documentation changes for wait_for_active_shards (#19581)
Documentation changes and migration doc changes for introducing 
wait_for_active_shards and removing write consistency level.

Closes #19581
2016-08-02 09:15:01 -04:00
Glen Smith bf51247ec0 Add note to put mapping API docs on index creation
This commit clarifies that the put mapping API can also apply when
creating an index.

Closes #19703
2016-07-31 15:58:24 -04:00
Areek Zillur 4e3602a790 Add zero-padding to auto-generated rollover index name increment
closes #19484
2016-07-27 10:50:47 -04:00
Nik Everett 3c0288ee98 Consolify term and phrase suggester docs
This includes a working example of reverse filters to support
correcting prefix errors.
2016-07-26 12:28:31 -04:00
Folusho Oladipo 1e7495a7fa corrected the use of two synonymous words (#19498)
Two synonyms were jointly used in the sentence(i.e "problems" and "issues"), so I deleted one of them.
2016-07-21 12:21:12 +02:00
Jun Ohtani cebad703fe Analyze: Specify anonymous char_filters/tokenizer/token_filters in the analyze API
Add parser for anonymous char_filters/tokenizer/token_filters
Using Settings in AnalyzeRequest for anonymous definition
Add breaking changes document

Closed #8878
2016-07-21 11:06:36 +09:00
Sho Minagawa 6aa598e3fb Fix typo on analyze.asciidoc (#19354) 2016-07-11 15:49:39 +02:00
Glen Smith d7099f05b9 slight clarification 2016-07-07 20:46:18 -04:00
Christoph Wurm d1727653dd Update shrink-index.asciidoc
Fix half-finished sentence
2016-07-05 13:34:58 +02:00
Boaz Leskes 6861d3571e Persistent Node Ids (#19140)
Node IDs are currently randomly generated during node startup. That means they change every time the node is restarted. While this doesn't matter for ES proper, it makes it hard for external services to track nodes. Another, more minor, side effect is that indexing the output of, say, the node stats API results in creating new fields due to node ID being used as keys.

The first approach I considered was to use the node's published address as the base for the id. We already [treat nodes with the same address as the same](https://github.com/elastic/elasticsearch/blob/master/core/src/main/java/org/elasticsearch/discovery/zen/NodeJoinController.java#L387) so this is a simple change (see [here](https://github.com/elastic/elasticsearch/compare/master...bleskes:node_persistent_id_based_on_address)). While this is simple and it works for probably most cases, it is not perfect. For example, if after a node restart, the node is not able to bind to the same port (because it's not yet freed by the OS), it will cause the node to still change identity. Also in environments where the host IP can change due to a host restart, identity will not be the same. 

Due to those limitation, I opted to go with a different approach where the node id will be persisted in the node's data folder. This has the upside of connecting the id to the nodes data. It also means that the host can be adapted in any way (replace network cards, attach storage to a new VM). I

It does however also have downsides - we now run the risk of two nodes having the same id, if someone copies clones a data folder from one node to another. To mitigate this I changed the semantics of the protection against multiple nodes with the same address to be stricter - it will now reject the incoming join if a node exists with the same id but a different address. Note that if the existing node doesn't respond to pings (i.e., it's not alive) it will be removed and the new node will be accepted when it tries another join.

Last, and most importantly, this change requires that *all* nodes persist data to disk. This is a change from current behavior where only data & master nodes store local files. This is the main reason for marking this PR as breaking.

Other less important notes:
- DummyTransportAddress is removed as we need a unique network address per node. Use `LocalTransportAddress.buildUnique()` instead.
- I renamed `node.add_lid_to_custom_path` to `node.add_lock_id_to_custom_path` to avoid confusion with the node ID which is now part of the `NodeEnvironment` logic.
- I removed the `version` paramater from `MetaDataStateFormat#write` , it wasn't really used and was just in the way :)
- TribeNodes are special in the sense that they do start multiple sub-nodes (previously known as client nodes). Those sub-nodes do not store local files but derive their ID from the parent node id, so they are generated consistently.
2016-07-04 21:09:25 +02:00
debadair 084b35c08b Docs: Fixed code callout error. 2016-06-17 14:31:03 -07:00
Areek Zillur 615920df2e update docs 2016-06-17 00:03:38 -04:00
Areek Zillur 6adffa6b7b Merge branch 'master' into enhancement/rollover_api 2016-06-16 17:27:32 -04:00
Areek Zillur eb9b4437b2 update docs 2016-06-15 14:57:17 -04:00
Areek Zillur 41d31541a6 Allow users to override the name for the rollover index 2016-06-09 13:43:19 -04:00
Areek Zillur a9f24ea2dc fail rollover request if rollover index already exists 2016-06-09 12:38:12 -04:00
Areek Zillur 9027e8a719 renamed simulated mode to dry_run mode 2016-06-09 11:55:10 -04:00
Areek Zillur 94a7978ef6 add documentation 2016-06-08 18:38:02 -04:00
Simon Willnauer b2c4c323e1 Allow `_shrink` to N shards if source shards is a multiple of N (#18699)
Today we allow to shrink to 1 shard but that might not be possible due to
too many document or a single shard doesn't meet the requirements for the index.
The logic can be expanded to N shards if the source index shards is a multiple of N.
This guarantees that there are not hotspots created due to different number of shards
being shrunk into one.
2016-06-07 10:06:41 +02:00
Clinton Gormley e6aaaf11ed Reworked docs for index-shrink API (#18705) 2016-06-03 09:50:51 +02:00
Simon Willnauer 22dfc41521 Only filter intial recovery (post API) when shrinking an index (#18661)
Today we use `index.routing.allocation.include._id` to filter the allocation
for the shrink target index. That has the sideeffect that the user has to
delete that setting / change it once the primary has been recovered (shrink is done)
This PR adds a dedicated filter that can only be set internally that only filters
allocation for unassigned shards.
2016-06-02 15:38:51 +02:00
Simon Willnauer 502a775a7c Add primitive to shrink an index into a single shard (#18270)
This adds a low level primitive operations to shrink an existing
index into a new index with a single shard. This primitive expects
all shards of the source index to allocated on a single node. Once the target index is initializing on the shrink node it takes a snapshot of the source index shards and copies all files into the target indices data folder. An [optimization](https://issues.apache.org/jira/browse/LUCENE-7300) coming in Lucene 6.1 will also allow for optional constant time copy if hard-links are supported by the filesystem. All mappings are merged into the new indexes metadata once the snapshots have been taken on the merge node.

To shrink an existing index all shards must be moved to a single node (one instance of each shard) and the index must be read-only:

```BASH
$ curl -XPUT 'http://localhost:9200/logs/_settings' -d '{
    "settings" : {
        "index.routing.allocation.require._name" : "shrink_node_name",
        "index.blocks.write" : true 
    }
}
```
once all shards are started on the shrink node. the new index can be created via:

```BASH
$ curl -XPUT 'http://localhost:9200/logs/_shrink/logs_single_shard' -d '{
    "settings" : {
        "index.codec" : "best_compression",
        "index.number_of_replicas" : 1
    }
}'
```

This API will perform all needed check before the new index is created and selects the shrink node based on the allocation of the source index. This call returns immediately, to monitor shrink progress the recovery API should be used since all copy operations are reflected in the recovery API with byte copy progress etc.

The shrink operation does not modify the source index, if a shrink operation should
be canceled or if the shrink failed, the target index can simply be deleted and
all resources are released.
2016-05-31 10:41:44 +02:00
Shane Connelly dd825c8730 Add TESTRESPONSE code 2016-05-12 18:44:08 -07:00
Shane Connelly 73c9cfbe8d Adds CONSOLE tests to aliases documentation 2016-05-12 11:25:13 -07:00
Zack Dever 5a7edf992c Add missing comma in JSON data in curl example
Relates #18227
2016-05-09 21:20:37 -04:00
Clinton Gormley 3f594089c2 Renamed all AUTOSENSE snippets to CONSOLE (#18210) 2016-05-09 15:42:23 +02:00
Nik Everett 4b1c116461 Generate and run tests from the docs
Adds infrastructure so `gradle :docs:check` will extract tests from
snippets in the documentation and execute the tests. This is included
in `gradle check` so it should happen on CI and during a normal build.

By default each `// AUTOSENSE` snippet creates a unique REST test. These
tests are executed in a random order and the cluster is wiped between
each one. If multiple snippets chain together into a test you can annotate
all snippets after the first with `// TEST[continued]` to have the
generated tests for both snippets joined.

Snippets marked as `// TESTRESPONSE` are checked against the response
of the last action.

See docs/README.asciidoc for lots more.

Closes #12583. That issue is about catching bugs in the docs during build.
This catches *some* bugs in the docs during build which is a good start.
2016-05-05 13:58:03 -04:00
Josef Salyer b2ce2f5afa Fixed the documentation formatting. (#17990)
The source code example for the initial example was missing the correct JSON object formatting and syntax.  That has been fixed with my change.
2016-04-29 18:07:48 +02:00
Jun Ohtani 9eb242a5fe Analyze API : Rename filters/token_filters/char_filter to filter/token_filter/char_filter
Closes #15189
2016-04-21 18:05:11 +09:00
Mahmoud Lababidi 20b398daea Add a more descriptive example to Index Template
Closes #17513
The current example in the documentation for Index Templates lacks any properties values. This is helpful to many devs that aren't sure how to take a regular Index Mapping and convert it to a template.
2016-04-06 12:18:10 +02:00
Clinton Gormley 637da86e5f Docs: Updated the upgrade API docs to explain that the reindex API should be used instead 2016-03-31 14:34:31 +02:00
Clinton Gormley 18c5ea8599 Update templates.asciidoc
Relates to #17375
2016-03-31 11:15:57 +02:00
Areek Zillur 442a6e0009 document suggest stats being merged with search stats 2016-03-23 16:37:57 -04:00
Adrien Grand b42f66c8ac Document 5.0 mapping changes. 2016-03-22 16:22:58 +01:00
Clinton Gormley 06929f8ed4 Merge pull request #17030 from 36degrees/patch-1
Fix typo in clear cache documentation
2016-03-09 15:34:15 +01:00
Camilo Díaz Repka 356364810c Implementation of Segment disk stats aggregating sizes by index file extension.
Use 'includeSegmentFileSizes' as the flag name to report disk usage.
Added test that verifies reported segment disk usage is growing accordingly after adding a document.
Documentation: Reference the new parameter as part of indices stats.
2016-03-01 23:16:49 -03:00
Lee Hinman 5451763935 [DOCS] Add blurb about `data_path` not needing to include index name
Resolves #11497
(node.enable_custom_paths is no longer used)
2016-02-21 13:11:45 -07:00