Commit Graph

602 Commits

Author SHA1 Message Date
Jason Tedor 5a9fc8aa2a Remove path.conf setting
This commit removes path.conf as a valid setting and replaces it with a
command-line flag for specifying a non-default path for configuration.

Relates #25392
2017-06-26 15:18:29 -04:00
Jason Tedor fde6f72cb5 Fix queries in cross-cluster search docs
This commit fixes two queries in the cross-cluster search docs; they
were missing the query object wrapping the actual query.

Relates #25282
2017-06-16 21:15:35 -04:00
debadair c161d90524 [DOCS] Defined es-test-dir and plugins-examples-dir in index.asciidoc. (#25232)
Use these attributes when specifying the location of included tests.
2017-06-15 08:54:10 -07:00
Lisa Cawley d181761aeb [DOCS] Add ML node to node.asciidoc (#24495)
* [DOCS] Add ML node to node.asciidoc

* [DOCS] Clarify ML node in node.asciidoc

* [DOCS] Add X-Pack icon for admonition blocks

* [DOCS] Formatting X-Pack blocks in node.asciidoc

* [DOCS] Add xpack icon images to node.asciidoc

* [DOCS] Add final xpack role attributes

* [DOCS] Remove unnecssary xpackicon image

* [DOCS] Add link to X-Pack node settings

* [DOCS] Fix path to X-Pack repository

* [DOCS] Add links to X-Pack node settings

* [DOCS] Fixed text for links to X-Pack node settings

* [DOCS] Change standalone node to dedicated node
2017-06-13 14:03:42 -07:00
Ryan Ernst a03b6c2fa5 Scripting: Change keys for inline/stored scripts to source/id (#25127)
This commit adds back "id" as the key within a script to specify a
stored script (which with file scripts now gone is no longer ambiguous).
It also adds "source" as a replacement for "code". This is in an attempt
to normalize how scripts are specified across both put stored scripts and script usages, including search template requests. This also deprecates the old inline/stored keys.
2017-06-09 08:29:25 -07:00
Luca Cavanna 018c6c38fe [DOCS] Clarify connections and gateway nodes selection in cross cluster search docs (#24859)
Closes #24836
2017-06-02 11:13:47 +02:00
Clinton Gormley 086abe6216 Marked `fixed_auto_queue_size` as experimental
Relates to https://github.com/elastic/elasticsearch/pull/23884
2017-05-22 10:03:31 +02:00
Jack Conradson 1196dfb6bb Remove Deprecated Script Settings (#24756)
Removes all fine-grained script settings replaced by scripts.types_allowed and scripts.contexts_allowed.
2017-05-18 13:32:46 -07:00
Ryan Ernst 463fe2f4d4 Scripting: Remove file scripts (#24627)
This commit removes file scripts, which were deprecated in 5.5.

closes #21798
2017-05-17 14:42:25 -07:00
Jack Conradson b7f0df626a [DOCS] Added Painless Language Spec content 2017-05-16 12:46:56 -07:00
Lee Hinman d09e64323f Add ability to automatically adjust search threadpool queue_size
This PR adds a new thread pool type: `fixed_auto_queue_size`. This thread pool
behaves like a regular `fixed` threadpool, except that every
`auto_queue_frame_size` operations (default: 10,000) in the thread pool,
[Little's Law](https://en.wikipedia.org/wiki/Little's_law) is calculated and
used to adjust the pool's `queue_size` either up or down by 50. A minimum and
maximum is taken into account also. When the min and max are the same value, a
regular fixed executor is used instead.

The `SEARCH` threadpool is changed to use this new type of thread pool. However,
the min and max are both set to 1000, meaning auto adjustment is opt-in rather
than opt-out.

Resolves #3890
2017-05-16 11:13:16 -06:00
Ryan Ernst 97d2657e18 Remove script access to term statistics (#19462)
In scripts (at least some of the languages), the terms dictionary and
postings can be access with the special _index variable. This is for
very advanced use cases which want to do their own scoring. The problem
is segment level statistics must be recomputed for every document.
Additionally, this is not friendly to the terms index caching as the
order of looking up terms should be controlled by lucene.

This change removes _index from scripts. Anyone using it can and should
instead write a Similarity plugin, which is explicitly designed to allow
doing the calculations needed for a relevance score.

closes #19359
2017-05-16 09:10:09 -07:00
Simon Willnauer 1cae850cf5 Add a cluster block that allows to delete indices that are read-only (#24678)
Today when an index is `read-only` the index is also blocked from
being deleted which sometimes is undesired since in-order to make
changes to a cluster indices must be deleted to free up space. This is
a likely scenario in a hosted environment when disk-space is limited to switch
indices read-only but allow deletions to free up space.
2017-05-16 17:34:37 +02:00
Jack Conradson 43292979fd Add New Security Script Settings (#24637)
Settings are simplified to allowed_types and allowed_contexts.  If a setting is not specified the default is to enable all for that setting.
2017-05-15 13:37:46 -07:00
Ryan Ernst c1f1f66509 Scripting: Replace advanced and native scripts with ScriptEngine docs (#24603)
This commit documents how to write a `ScriptEngine` in order to use
expert internal apis, such as using Lucene directly to find index term
statistics. These documents prepare the way to remove both native
scripts and IndexLookup.

The example java code is actually compiled and tested under a new gradle
subproject for example plugins. This change does not yet breakup
jvm-example into the new examples dir, which should be done separately.

relates #19359
relates #19966
2017-05-11 12:15:16 -07:00
Chris Ivens d447b79e16 Docs: Tiny typo to Painless dispatch justification (#24588) 2017-05-10 22:05:19 -04:00
Ali Beyad 743217a430 Enhances get snapshots API to allow retrieving repository index only (#24477)
Currently, the get snapshots API (e.g. /_snapshot/{repositoryName}/_all)
provides information about snapshots in the repository, including the
snapshot state, number of shards snapshotted, failures, etc.  In order
to provide information about each snapshot in the repository, the call
must read the snapshot metadata blob (`snap-{snapshot_uuid}.dat`) for
every snapshot.  In cloud-based repositories, this can be expensive,
both from a cost and performance perspective.  Sometimes, all the user
wants is to retrieve all the names/uuids of each snapshot, and the
indices that went into each snapshot, without any of the other status
information about the snapshot.  This minimal information can be
retrieved from the repository index blob (`index-N`) without needing to
read each snapshot metadata blob.

This commit enhances the get snapshots API with an optional `verbose`
parameter.  If `verbose` is set to false on the request, then the get
snapshots API will only retrieve the minimal information about each
snapshot (the name, uuid, and indices in the snapshot), and only read
this information from the repository index blob, thereby giving users
the option to retrieve the snapshots in a repository in a more
cost-effective and efficient manner.

Closes #24288
2017-05-10 15:48:40 -04:00
Clinton Gormley 2486086980 Deprecate the Tribe node
The Tribe node is deprecated in favour of Cross Cluster Search and will be removed in 7.0.
2017-05-10 14:05:12 +02:00
Clinton Gormley 9f08a553d9 Fixed docs syntax for for-in loop in painless 2017-05-05 16:07:20 +02:00
Nik Everett a01f846226 CONSOLEify a few more docs
Adds CONSOLE to cross-cluster-search docs but skips them for testing
because we don't have a second cluster set up. This gets us the
`VIEW IN CONSOLE` and `COPY AS CURL` links and makes sure that they
are valid yaml (not json, technically) but doesn't get testing.
Which is better than we had before.

Adds CONSOLE to the dynamic templates docs and ingest-node docs.
The ingest-node docs contain a *ton* of non-console snippets. We
might want to convert them to full examples later, but that can be
a separate thing.

Relates to #18160
2017-05-04 21:01:14 -04:00
Till Backhaus b744dc3bcc Link to minimum master nodes docs from Zen docs
This commit adds a link to the minimum master nodes section of the
important settings docs from the Zen discovery docs to clarify the
meaning and importance of setting minimum master nodes to a quorum of
master-eligible nodes.

Relates #24311
2017-04-25 16:53:05 -04:00
Nik Everett 5fbc86e2aa Allow painless to load stored fields (#24290)
We document that painless can load stored fields but it can't
because the classes that make that work aren't whitelisted.
2017-04-24 14:22:39 -04:00
farisk 931198688c Document that painless doesn't support the "advanced text scoring" (#24271)
I just spent ages debugging a script I wrote after following the documentation. It was not clear to me that _index is not defined when using painless; if it was mentioned on this page I would have saved myself a lot of time.
2017-04-24 10:29:49 -04:00
Clinton Gormley 710cd05253 Added examples to cross cluster search of using cluster settings 2017-04-20 15:02:52 +02:00
Iliiaz Akhmedov 688fa309bc Changing some grammar in docs (#24164) 2017-04-19 08:49:13 -06:00
Nik Everett c17bee571f Update scripts/security docs for sandboxed world (#23977)
Drops any mention of non-sandboxed scripting languages other than a
brief "we don't support them and we shouldn't because A and B"
statement.

Relates to #23930
2017-04-11 14:55:57 -04:00
Nik Everett 7fad7c675d Rewrite the scripting security docs (#23930)
They needed to be updated now that Painless is the default and
the non-sandboxed scripting languages are going away or gone.

I dropped the entire section about customizing the classloader
whitelists. In master this barely does anything (exposes more
things to expressions).
2017-04-07 11:46:41 -04:00
wyukawa 3274eab41d Fix bulk queue size in thread pool docs
This commit fixes an incorrect specification for the default queue size
for the bulk thread pool in the thread pool docs.

Relates #23870
2017-04-03 22:39:24 -04:00
Nik Everett 9d2293b381 CONSOLEify the "using scripts" documentation
I found an error in one of the Painless scripts as part of
the conversion.

Relates to #18160
2017-04-03 10:15:29 -04:00
Ali Beyad a4b37bf7fe [DOCS] Multiple clusters connected to the same repository (#23807) 2017-03-30 13:08:41 -04:00
Shane Connelly 16a8d5245f Reflect cross-cluster search in "dedicated" terminology (#23771)
* Reflects cross-cluster search in dedicated node settings

* Fix space issue
2017-03-27 23:14:34 -04:00
Igor Motov 63e1403017 Docs: add description of possible snapshot states 2017-03-23 15:20:38 -04:00
Stefan Gorgiovski 798c19dd7f Deprecate request_cache for clear-cache (#23638)
It is called `request` now.
2017-03-22 08:28:04 -04:00
Jay Modi b234644035 Enforce Content-Type requirement on the rest layer and remove deprecated methods (#23146)
This commit enforces the requirement of Content-Type for the REST layer and removes the deprecated methods in transport
requests and their usages.

While doing this, it turns out that there are many places where *Entity classes are used from the apache http client
libraries and many of these usages did not specify the content type. The methods that do not specify a content type
explicitly have been added to forbidden apis to prevent more of these from entering our code base.

Relates #19388
2017-02-17 14:45:41 -05:00
Ali Beyad 71739623d3 Consolify snapshot documentation (#23189)
This commit brings the snapshot documentation in conformity
with the CONSOLE format, and fixes the docs so that the documentation
tests can be run against them.
2017-02-15 18:13:27 -05:00
Boaz Leskes 70a3ac1767 Add a note about `cluster.routing.allocation.node_concurrent_recoveries` (#23160)
Closes #23152
2017-02-14 14:14:41 +02:00
Clinton Gormley d43417ef47 Docs: Deleted redundant word in scripting 2017-02-09 22:02:42 +01:00
Igor Motov 1fc4fa5729 Docs: CONSOLEify native script docs
Relates #23001
2017-02-08 13:30:39 -05:00
Tim Brooks ad4bfa2307 Docs: CONSOLEify transport docs (#23027)
This is related to #23001.
2017-02-07 20:06:28 -06:00
Nik Everett a2ed676862 Docs: Explain painless's method dispatch (#23021)
Painless uses Ruby-like method dispatch (reciever type, method name,
and arity) rather than Java-like (reciever type, method name, and
argument compile time types) or Groovy-like method dispatch (receiver
type, method name, and argument run time types). We do this for
mostly good reasons but we never documented it.

Relates to #22720
2017-02-07 12:09:22 -05:00
Simon Willnauer dc659feeb4 Add a setting to disable remote cluster connections on a node (#23005)
Today either all nodes in the cluster connect to remote clusters of only nodes
that have remote clusters configured in their node config. To allow global remote
cluster configuration but restrict connections to a set of nodes in the cluster
this change adds a new setting `search.remote.connect` (defaults to `true`) to allow
to disable remote cluster connections on a per node basis.
2017-02-07 09:59:24 +01:00
Nik Everett 0d6e622242 Make dates be ReadableDateTimes in scripts (#22948)
Instead of longs. If you want millis since epoch you can call doc.date_field.value.millis.

Relates to #22875
2017-02-06 16:44:56 -05:00
javanna b9cf6333bd [TEST] fix typo in cross cluster search docs 2017-02-05 15:56:45 +01:00
Jay Modi 7520a107be Optionally require a valid content type for all rest requests with content (#22691)
This change adds a strict mode for xcontent parsing on the rest layer. The strict mode will be off by default for 5.x and in a separate commit will be enabled by default for 6.0. The strict mode, which can be enabled by setting `http.content_type.required: true` in 5.x, will require that all incoming rest requests have a valid and supported content type header before the request is dispatched. In the non-strict mode, the Content-Type header will be inspected and if it is not present or not valid, we will continue with auto detection of content like we have done previously.

The content type header is parsed to the matching XContentType value with the only exception being for plain text requests. This value is then passed on with the content bytes so that we can reduce the number of places where we need to auto-detect the content type.

As part of this, many transport requests and builders were updated to provide methods that
accepted the XContentType along with the bytes and the methods that would rely on auto-detection have been deprecated.

In the non-strict mode, deprecation warnings are issued whenever a request with body doesn't provide the Content-Type header.

See #19388
2017-02-02 14:07:13 -05:00
Nik Everett dacc150934 Expose multi-valued dates to scripts and document painless's date functions (#22875)
Implemented by wrapping an array of reused `ModuleDateTime`s that
we grow when needed. The `ModuleDateTime`s are reused when we
move to the next document.

Also improves the error message returned when attempting to modify
the `ScriptdocValues`, removes a couple of allocations, and documents
that the date functions are available in Painless.

Relates to #22162
2017-02-01 21:57:07 -05:00
Jack Conradson 3d2626c4c6 Change Namespace for Stored Script to Only Use Id (#22206)
Currently, stored scripts use a namespace of (lang, id) to be put, get, deleted, and executed. This is not necessary since the lang is stored with the stored script. A user should only have to specify an id to use a stored script. This change makes that possible while keeping backwards compatibility with the previous namespace of (lang, id). Anywhere the previous namespace is used will log deprecation warnings.

The new behavior is the following:

When a user specifies a stored script, that script will be stored under both the new namespace and old namespace.

Take for example script 'A' with lang 'L0' and data 'D0'. If we add script 'A' to the empty set, the scripts map will be ["A" -- D0, "A#L0" -- D0]. If a script 'A' with lang 'L1' and data 'D1' is then added, the scripts map will be ["A" -- D1, "A#L1" -- D1, "A#L0" -- D0].

When a user deletes a stored script, that script will be deleted from both the new namespace (if it exists) and the old namespace.

Take for example a scripts map with {"A" -- D1, "A#L1" -- D1, "A#L0" -- D0}. If a script is removed specified by an id 'A' and lang null then the scripts map will be {"A#L0" -- D0}. To remove the final script, the deprecated namespace must be used, so an id 'A' and lang 'L0' would need to be specified.

When a user gets/executes a stored script, if the new namespace is used then the script will be retrieved/executed using only 'id', and if the old namespace is used then the script will be retrieved/executed using 'id' and 'lang'
2017-01-31 13:27:02 -08:00
alamzeeshan a1cc683cff Updated document as per code change. (#22878)
Updated document as per this change : https://github.com/elastic/elasticsearch/pull/15235
2017-01-31 13:36:09 +01:00
Nik Everett 8a2d424d68 Generate reference links for painless API (#22775)
Adds "Appending B. Painless API Reference", a reference of all classes
and methods available from Painless. Removes links to java packages
because they contain methods that we don't expose and don't contain
methods that we do expose (the ones in Augmentation). Instead this
generates a list of every class and every exposed method using the same
type information available to the
interpreter/compiler/whatever-we-call-it. From there you can jump to
the relevant docs.

Right now you build all the asciidoc files by running
```
gradle generatePainlessApi
```

These files are expected to be committed because we build the docs
without running `gradle`.

Also changes the output of `Debug.explain` so that it is easy to
search for the class in the generated reference documentation.

You can also run it in an IDE safely if you pass the path to the
directory in which to generate the docs as the first parameter. It'll
blow away the entire directory an recreate it from scratch so be careful.

And then you can build the docs by running something like:
```
../docs/build_docs.pl --out ../built_docs/ --doc docs/reference/index.asciidoc --open
```

That is, if you have checked out https://github.com/elastic/docs in
`../docs`. Wait a minute or two and your browser will pop open in with
all of Elasticsearch's reference documentation. If you go to
`http://localhost:8000/painless-api-reference.html` you can see this
list. Or you can get there by following the links to `Modules` and
`Scripting` and `Painless` and then clicking the link in the paragraphs
below titled `Appendix B. Painless API Reference`.

I like having these in asciidoc because we can deep link to them from the
rest of the guide with constructs like
`<<painless-api-reference-Object-hashCode-0>>` and
`<<painless-api-reference->>` and we get link checking. Then the only
brittle link maintenance bit is the link generation for javadoc. Which
sucks. But I think it is important that we link to the methods directly
so they are easy to find.

Relates to #22720
2017-01-26 10:39:19 -05:00
Clinton Gormley 12f5309041 Docs: Made cross-cluster search experimental 2017-01-24 12:44:01 +01:00
Clinton Gormley e415170989 Docs: Fix link syntax in cross-cluster docs
Also tidy up settings list
2017-01-18 10:13:48 +01:00
Simon Willnauer d7eee637d9 fix some docs issues 2017-01-17 11:47:29 +01:00
Simon Willnauer 00781d24ce Merge branch 'master' into feature/multi_cluster_search 2017-01-11 23:40:46 +01:00
Jason Tedor b7995fbc0d Fix default port for unicast zen ping hosts
Today when you do not specify a port for an entry in
discovery.zen.ping.unicast.hosts, the default port is the value of the
setting transport.profiles.default.port and falls back to the value of
transport.tcp.port if this is not set. For a node that is explicitly
bound to a different port than the default port, this means that the
default port will be equal to this explicitly bound port. Yet, the docs
say that we fall back to 9300 here. This commit corrects the docs.

Relates #22568
2017-01-11 17:10:56 -05:00
Simon Willnauer 46713fb9aa remove leftover 2017-01-11 15:10:36 +01:00
Simon Willnauer ea0e72f3d9 improve docs based on review commments: 2017-01-11 14:52:41 +01:00
Simon Willnauer 4c61f1d75d Cut over to use affix setting for remote cluster configuration
Instead of `search.remote.seeds.${clustername}` we now specify the seeds as:
`search.remote.${clustername}.seeds` which is a real list setting compared to an unvalidated
group setting before.
2017-01-11 12:38:46 +01:00
Simon Willnauer 6d2d878068 Merge branch 'master' into feature/multi_cluster_search 2017-01-11 09:28:00 +01:00
Martijn van Groningen cb2333dacd percolator: remove deprecated percolate and mpercolate apis 2017-01-10 11:18:27 +01:00
javanna d3d81fcab5 fix another typo in the docs 2017-01-09 18:22:11 +01:00
javanna 56c0bd659d fix couple of typos in the docs 2017-01-09 18:17:43 +01:00
Simon Willnauer 349ea0f9b6 cut over to use : instead of | for cross cluster search 2017-01-05 17:03:12 +01:00
Simon Willnauer dca54734ac add basic docs 2017-01-05 16:10:34 +01:00
Dave Richardson eaefb5f99b Descriptions for scripting enabled (#22388)
The `Script source settings` section currently states that `false` means scripting is ENABLED.
The other sections seem to indicate that `false` means scripting is DISABLED.

If the current documentation is correct, that would imply that `inline` and `stored` scripting are ENABLED by default, which seems to conflict with all the other sections in the document.
2016-12-30 12:50:17 -05:00
Paweł Bobruk 117b63ed41 Docs fix native script usage in 5.x (#22362) 2016-12-29 09:41:26 -08:00
Clinton Gormley f96769f97b Update painless-syntax.asciidoc
Fix asciidoc syntax
2016-12-19 10:09:24 +01:00
Nik Everett 61597f2c20 Send error_trace by default when testing (#22195)
Sends the `error_trace` parameter with all requests sent by the
yaml test framework, including the doc snippet tests. This can be
overridden by settings `error_trace: false`. While this drift's
core's handling of the yaml tests from the client's slightly this
should only be a problem for tests that rely on the default value,
both of which I've fixed by setting the value explicitly.

This also escapes `\n` and `\t` in the `Stash dump on failure` so
the `stack_trace` is more readable.

Also fixes `RestUpdateSettingsAction` to not think of the `error_trace`
parameter as a setting.
2016-12-15 13:35:14 -05:00
Clinton Gormley 571b20137a Document using round-robin DNS for discovery (#21810)
* Document using round-robin DNS for discovery

Closes #8427

* Improve wording around round robin DNS

* Included jason's suggestion
2016-12-14 11:29:39 +01:00
Lucas Bremgartner 0086b99797 [Docs] Correct setting name in snapshot/restore documentation (#22023)
There is no setting include_cluster_state for snapshot restore. The correct name for this setting is include_global_state.
2016-12-07 14:12:10 +01:00
Jason Tedor b9df2e2287 Improve the out-of-the-box experience
Elasticsearch can be run in a few different ways:
 - from the command line on Linux and Windows
 - as a service on Linux and Windows

on both 32-bit client and 64-bit server VMs. We strive for a great
out-of-the-box experience any of these combinations but today it is
lacking on 32-bit client JVMs and on the Windows service. There are two
deficiencies that arise:
 - on any 32-bit client JVM we fail to start out of the box because we
   force the server JVM in jvm.options
 - when installing the Windows service, the thread stack size must be
   specified in jvm.options

This commit attempts to address these deficiencies.

We should continue to force the server JVM because there are systems
where the server JVM is not active by default (e.g., the 32-bit JDK on
Windows). This does mean that if a user tries to run with a client JVM
they will see a failure message at startup but this is the best that we
can do if we want to continue to force the server JVM. Thus, this commit
at least documents this situation.

To improve the situation with installing the Windows service, this
commit adds a default setting for the thread stack size. This default is
chosen based on the default thread stack size across all 64-bit server
JVMs. This means that if a user tries to run with a 32-bit JVM they
could otherwise see significantly higher memory usage (this situation is
complicated, it's really only on Windows where the extra memory usage is
egregious, but cutting into the 32-bit address space on any system is
bad). So this commit makes it so that the out-of-the-box experience is
improved for the Windows service on 64-bit server JVMs and we document
the need to adjust this setting on 32-bit JVMs.

Again, we are focusing on the out-of-the-box experience here and this
means optimizing for the best experience on any 64-bit server JVM as
this covers the vast majority of the user base. The users that are on
32-bit JVMs will suffer a little bit but at least now any user on any
64-bit server JVM can start Elasticsearch out of the box.

Finally, we fix some references to the jvm.options documentation.

Relates #21920
2016-12-01 17:26:29 -05:00
Ali Beyad 51bfecc7cb [DOCS] fixes word usage in allocation awareness docs 2016-11-25 11:47:40 -05:00
Nik Everett 434fa4bd26 Docs and tests for painless lack of boxing for ?: and ?. (#21756)
NOTE: The result of `?.` and `?:` can't be assigned to primitives. So
`int[] someArray = null; int l = someArray?.length` and
`int s = params.size ?: 100` don't work. Do
`def someArray = null; def l = someArray?.length` and
`def s = params.size ?: 100` instead.

Relates to #21748
2016-11-23 14:33:32 -05:00
Jason Tedor 32e6fcf256 Fix markup in Zen discovery docs
This commit fixes a markup issue in the Zen discovery docs where a link
and its referring text were not on the same line tripping the renderer.
2016-11-23 10:02:13 -05:00
Ryan Ernst 6940b2b8c7 Remove groovy scripting language (#21607)
* Scripting: Remove groovy scripting language

Groovy was deprecated in 5.0. This change removes it, along with the
legacy default language infrastructure in scripting.
2016-11-22 19:24:12 -08:00
Jason Tedor 9dc65037bc Lazy resolve unicast hosts
Today we eagerly resolve unicast hosts. This means that if DNS changes,
we will never find the host at the new address. Moreover, a single host
failng to resolve causes startup to abort. This commit introduces lazy
resolution of unicast hosts. If a DNS entry changes, there is an
opportunity for the host to be discovered. Note that under the Java
security manager, there is a default positive cache of infinity for
resolved hosts; this means that if a user does want to operate in an
environment where DNS can change, they must adjust
networkaddress.cache.ttl in their security policy. And if a host fails
to resolve, we warn log the hostname but continue pinging other
configured hosts.

When doing DNS resolutions for unicast hostnames, we wait until the DNS
lookups timeout. This appears to be forty-five seconds on modern JVMs,
and it is not configurable. If we do these serially, the cluster can be
blocked during ping for a lengthy period of time. This commit introduces
doing the DNS lookups in parallel, and adds a user-configurable timeout
for these lookups.

Relates #21630
2016-11-22 14:17:04 -05:00
Nik Everett 83ea1be185 Remove a bad callout from the new debugging docs
It was a leftover and doesn't break the usual build but breaks the docs
build.
2016-11-22 13:13:16 -05:00
Nik Everett 457c2d8fb0 Add Debug.explain to painless
You can use `Debug.explain(someObject)` in painless to throw an
`Error` that can't be caught by painless code and contains an
object's class. This is useful because painless's sandbox doesn't
allow you to call `someObject.getClass()`.

Closes #20263
2016-11-22 12:46:02 -05:00
Nik Everett c79371fd5b Remove lang-python and lang-javascript (#20734)
They were deprecated in 5.0. We are concentrating on making
Painless awesome rather than supporting every language possible.

Closes #20698
2016-11-21 22:13:25 -05:00
Nik Everett ae468441dc Implement the ?: operator in painless (#21506)
Implements a null coalescing operator in painless that looks like `?:`. This form was chosen to emulate Groovy's `?:` operator. It is different in that it only coalesces null values, instead of Groovy's `?:` operator which coalesces all falsy values. I believe that makes it the same as Kotlin's `?:` operator. In other languages this operator looks like `??` (C#) and `COALESCE` (SQL) and `:-` (bash).

This operator is lazy, meaning the right hand side is only evaluated at all if the left hand side is null.
2016-11-18 13:54:26 -05:00
Ryan Ernst 914664d89a Fix leftover reference to ScriptModule in native script docs 2016-11-17 08:22:15 -08:00
Ryan Ernst 48bfb142b9 Remove (again) test uses of onModule (#21414)
This change was reverted after it caused random test failures. This was
due to a copy/paste error in the original PR which caused the mock
version of ClusterInfoService to be used whenever the mock *ZenPing* was
used, and the real ClusterInfoService to be used when MockZenPing was
not used.
2016-11-10 16:06:14 -08:00
lslxdx e560c5be43 Fix typos (#21456)
Delete repeated words " between nodes and".
2016-11-10 11:55:04 +01:00
Nik Everett d03b8e4abb Implement reading from null safe dereferences
Null safe dereferences make handling null or missing values shorter.
Compare without:
```
if (ctx._source.missing != null && ctx._source.missing.foo != null) {
  ctx._source.foo_length = ctx.source.missing.foo.length()
}
```

To with:
```
Integer length = ctx._source.missing?.foo?.length();
if (length != null) {
  ctx._source.foo_length = length
}
```

Combining this with the as of yet unimplemented elvis operator allows
for very concise defaults for nulls:
```
ctx._source.foo_length = ctx._source.missing?.foo?.length() ?: 0;
```

Since you have to start somewhere, we started with null safe dereferenes.

Anyway, this is a feature borrowed from groovy. Groovy allows writing to
null values like:
```
def v = null
v?.field = 'cat'
```
And the writes are simply ignored. Painless doesn't support this at this
point because it'd be complex to implement and maybe not all that useful.

There is no runtime cost for this feature if it is not used. When it is
used we implement it fairly efficiently, adding a jump rather than a
temporary variable.

This should also work fairly well with doc values.
2016-11-09 07:20:11 -05:00
javanna 2f32c1173b Revert "Tests: Remove a couple test uses of onModule (#21414)"
This reverts commit b326f0bc51.
2016-11-09 11:32:16 +01:00
Ryan Ernst b326f0bc51 Tests: Remove a couple test uses of onModule (#21414)
There were still a couple test use cases and examples that were using
onModule. This change cleans those cases up.
2016-11-08 13:50:13 -08:00
Clinton Gormley 9d56c1b766 Groovy is no longer the default scripting language
Closes #21208
2016-11-05 16:31:34 +01:00
Jack Conradson 185dff7346 Cleanup ScriptType (#21179)
Refactored ScriptType to clean up some of the variable and method names. Added more documentation. Deprecated the 'in' ParseField in favor of 'stored' to match the indexed scripts being replaced by stored scripts.
2016-10-31 13:48:51 -07:00
Nik Everett 3a7a218e8f Support negative array ofsets in painless
Adds support for indexing into lists and arrays with negative
indexes meaning "counting from the back". So for if
`x = ["cat", "dog", "chicken"]` then `x[-1] == "chicken"`.

This adds an extra branch to every array and list access but
some performance testing makes it look like the branch predictor
successfully predicts the branch every time so there isn't a
in execution time for this feature when the index is positive.
When the index is negative performance testing showed the runtime
is the same as writing `x[x.length - 1]`, again, presumably thanks
to the branch predictor.

Those performance metrics were calculated for lists and arrays but
`def`s get roughly the same treatment though instead of inlining
the test they need to make a invoke dynamic so we don't screw up
maps.

Closes #20870
2016-10-29 16:12:40 -04:00
Praveen Shukla 2e18f2e818 [DOCS] clarifies master nodes to be master eligible nodes 2016-10-24 08:58:44 -04:00
Joshua Rich 63f484ffa3 Docs: Cluster Allocation Filtering
Put more emphasis on the fact that multiple values can be specified
and move examples after explanation of settings.
2016-10-17 11:02:33 +11:00
Jason Tedor 370253f95a Add doc note regarding processors bound
This commit expands the thread pool docs regarding the processor
setting.

Relates #20895
2016-10-14 10:32:31 -04:00
Nik Everett 3bba7dbe07 Docs: note about snapshot version compatibility (#20896)
It is important that folks understand that snapshot/restore isn't
for archiving. It is appropriate for backup and disaster recovery
but not for archival over long periods of time because of version
incompatibility.

Closes #20866
2016-10-13 10:49:32 -04:00
Pascal Borreli fcb01deb34 Fixed typos (#20843) 2016-10-10 14:51:47 -06:00
Simon Willnauer 194a6b1df0 Remove LocalTransport in favor of MockTcpTransport (#20695)
This change proposes the removal of all non-tcp transport implementations. The
mock transport can be used by default to run tests instead of local transport that has
roughly the same performance compared to TCP or at least not noticeably slower.

This is a master only change, deprecation notice in 5.x will be committed as a
separate change.
2016-10-07 11:27:47 +02:00
Nik Everett 3ed3e5e660 Convert more docs to CONSOLE
* plugins/discovery-azure-class.asciidoc
* reference/cluster.asciidoc
* reference/modules/cluster/misc.asciidoc
* reference/modules/indices/request_cache.asciidoc

After this is merged there will be no unconvereted snippets outside
of `reference`.

Related to #18160
2016-09-21 09:36:21 -04:00
Ali Beyad f608e6c6cf Improves the documentation for the (#20531)
`cluster.routing.allocation.cluster_concurrent_rebalance` setting,
clarifying in which shard allocation situations the rebalance limit
takes effect.

Closes #20529
2016-09-16 16:06:18 -04:00
Jason Tedor 412c61c402 Update logger names in docs
In 7560101ec7, the Elasticsearch logger
names were modified to be their fully-qualified class name (with some
exceptions for special loggers like the slow logs and the transport
tracer). This commit updates the docs accordingly.

Relates #20475
2016-09-14 08:08:49 -04:00
Nik Everett 69bf08f6c6 Disable regexes by default in painless
Adds a new node level, non-dynamic setting, `script.painless.regex.enabled`
can be used to enable regexes.

Closes #20397
2016-09-12 14:09:43 -04:00
Clinton Gormley add2fbd7b2 Update painless.asciidoc
Asciidoc typo
2016-09-09 09:30:04 +02:00
Jack Conradson 3b3baa6e6c Made deprecation of Groovy, Javascript, and Python more explicit. 2016-08-31 15:56:31 -07:00
Jack Conradson 2b9c3a1a98 Fixed a doc test. 2016-08-30 16:51:37 -07:00
Jack Conradson 7930233527 Deprecate Groovy, Python, and Javascript scripts. 2016-08-30 09:06:18 -07:00
Nik Everett 52f23918c2 Use `painless` as language for painless snippets (#20185)
The syntax highlighter does a decent job when you do this. This lets
us `grep` for painless snippets in the docs.

Closes #20025
2016-08-26 15:39:44 -04:00
Dominik Stadler f0db4d9942 Add an example call of how to stop a snapshot or restore operation (#20153) 2016-08-25 13:01:04 +02:00
Jason Tedor d4dec26aa0 Update max local storage nodes docs
This commit updates the max local storage nodes docs to reflect that the
default is now one after 1f0673c9bd.

Relates #20029
2016-08-17 12:13:09 -04:00
Clinton Gormley 15c96df5b5 Replaced "true" with true in snapshot restore docs
Closes #19947
2016-08-12 16:56:56 +02:00
Colin Goodheart-Smithe 2904562b01 [DOCS] Fix shard request cache docs
Docs have been changed to reflect the fact that shard request cache is now enabled by default

Closes #19695
2016-08-11 14:25:34 +01:00
Pius 40f3b5ab76 Update snapshots.asciidoc
>However, the version of the new cluster should be the same or newer than the cluster that was

Afaik, you can't restore a snapshot to a newer cluster that is not consecutively newer (i.e. can't restore 1.x snapshot to a 5.x cluster).  This is to clarify the statement above moving forward.
2016-08-09 18:22:29 -07:00
Lee Hinman 5849c488b5 Merge remote-tracking branch 'dakrone/compliation-breaker' 2016-08-09 11:57:26 -06:00
Ali Beyad d915230a06 Clarifies the documentation for the `http.cors.enabled` setting (#19890)
Clarifies the documentation for the `http.cors.enabled` setting
2016-08-09 12:54:38 -05:00
Lee Hinman 2be52eff09 Circuit break the number of inline scripts compiled per minute
When compiling many dynamically changing scripts, parameterized
scripts (<https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-using.html#prefer-params>)
should be preferred. This enforces a limit to the number of scripts that
can be compiled within a minute. A new dynamic setting is added -
`script.max_compilations_per_minute`, which defaults to 15.

If more dynamic scripts are sent, a user will get the following
exception:

```json
{
  "error" : {
    "root_cause" : [
      {
        "type" : "circuit_breaking_exception",
        "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
        "bytes_wanted" : 0,
        "bytes_limit" : 0
      }
    ],
    "type" : "search_phase_execution_exception",
    "reason" : "all shards failed",
    "phase" : "query",
    "grouped" : true,
    "failed_shards" : [
      {
        "shard" : 0,
        "index" : "i",
        "node" : "a5V1eXcZRYiIk8lecjZ4Jw",
        "reason" : {
          "type" : "general_script_exception",
          "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]",
          "caused_by" : {
            "type" : "circuit_breaking_exception",
            "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
            "bytes_wanted" : 0,
            "bytes_limit" : 0
          }
        }
      }
    ],
    "caused_by" : {
      "type" : "general_script_exception",
      "reason" : "Failed to compile inline script [\"aaaaaaaaaaaaaaaa\"] using lang [painless]",
      "caused_by" : {
        "type" : "circuit_breaking_exception",
        "reason" : "[script] Too many dynamic script compilations within one minute, max: [15/min]; please use on-disk, indexed, or scripts with parameters instead",
        "bytes_wanted" : 0,
        "bytes_limit" : 0
      }
    }
  },
  "status" : 500
}
```

This also fixes a bug in `ScriptService` where requests being executed
concurrently on a single node could cause a script to be compiled
multiple times (many in the case of a powerful node with many shards)
due to no synchronization between checking the cache and compiling the
script. There is now synchronization so that a script being compiled
will only be compiled once regardless of the number of concurrent
searches on a node.

Relates to #19396
2016-08-09 10:26:27 -06:00
Nicholas Knize 2d590af593 Deprecate GeoDistance enumerators and remove geo distance script helpers
GeoDistance is implemented using a crazy enum that causes issues with the scripting modules. This commit moves all distance calculations to arcDistance and planeDistance static methods in GeoUtils. It also removes unnecessary distance helper methods from ScriptDocValues.GeoPoints.
2016-08-05 18:42:06 -05:00
Brandon Wulf 6b7d40929c Switch example from inclusion to exclusion.
Page is explaining allocation exclusion- example should be about exclusion as well.
2016-07-28 21:54:22 -04:00
markwalkom ebf96bbc35 Update gateway.asciidoc (#19572)
* Update gateway.asciidoc

Added a note to clarify that, in cases where nodes in a cluster have different setting, the node that is the elected master takes precedence over anything else.

* Update gateway.asciidoc

Updated as per @bleskes's comments
2016-07-28 13:09:05 +02:00
kingrhoton 1307aa7e77 clarify awkward text (#19608) 2016-07-27 20:03:20 +02:00
Clinton Gormley 8315a64a33 provide code example for processors setting
A simple example but was missing

Closes #19567
2016-07-27 17:54:52 +02:00
kingrhoton 643ccb8cc1 [docs] Switch contraction to possesive 2016-07-26 14:01:30 -04:00
Lee Hinman 1623cff6c0 Merge remote-tracking branch 'dakrone/bucket-circuit-breaker' 2016-07-25 13:37:26 -06:00
Lee Hinman 124a9fabe3 Circuit break on aggregation bucket numbers with request breaker
This adds new circuit breaking with the "request" breaker, which adds
circuit breaks based on the number of buckets created during
aggregations. It consists of incrementing during AggregatorBase creation

This also bumps the REQUEST breaker to 60% of the JVM heap now.

The output when circuit breaking an aggregation looks like:

```json
{
  "shard" : 0,
  "index" : "i",
  "node" : "a5AvjUn_TKeTNYl0FyBW2g",
  "reason" : {
    "type" : "exception",
    "reason" : "java.util.concurrent.ExecutionException: QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: CircuitBreakingException[[request] Data too large, data for [<agg [otherthings]>] would be larger than limit of [104857600/100mb]];",
    "caused_by" : {
      "type" : "execution_exception",
      "reason" : "QueryPhaseExecutionException[Query Failed [Failed to execute main query]]; nested: CircuitBreakingException[[request] Data too large, data for [<agg [myagg]>] would be larger than limit of [104857600/100mb]];",
      "caused_by" : {
        "type" : "circuit_breaking_exception",
        "reason" : "[request] Data too large, data for [<agg [otherthings]>] would be larger than limit of [104857600/100mb]",
        "bytes_wanted" : 104860781,
        "bytes_limit" : 104857600
      }
    }
  }
}
```

Relates to #14046
2016-07-25 11:33:37 -06:00
Colin Goodheart-Smithe b717ad8eb6 Enable option to use request cache for size > 0
Previously if the size of the search request was greater than zero we would not cache the request in the request cache.

This change retains the default behaviour of not caching requests with size > 0 but also allows the `request_cache=true` query parameter
to enable the cache for requests with size > 0
2016-07-18 13:33:59 +01:00
Clinton Gormley d2f25416e4 Update node.asciidoc
Typo
2016-07-17 21:31:35 +02:00
Clinton Gormley 49d0f3406c Update node.asciidoc
Master nodes must have access to a persistent data directory
2016-07-17 21:10:33 +02:00
Clinton Gormley ab7a976e49 Make Prefer Parameters admon block linkable 2016-07-13 16:02:34 +02:00
Michael Sander c493774093 Fix typo in cluster module docs
This commit fixes a simple typo in the cluster module docs.

Closes #19393
2016-07-12 16:32:23 -04:00
Clinton Gormley 982e01d463 Update network.asciidoc
`network.publish_host` defaults to `network.host`, not `network.bind_host`

Closes #19304
2016-07-08 17:13:10 +02:00
Jim Ferenczi afe99fcdcd Restore reverted change now that alpha4 is out:
Rename `fields` to `stored_fields` and add `docvalue_fields`

`stored_fields` parameter will no longer try to retrieve fields from the _source but will only return stored fields.
`fields` will throw an exception if the user uses it.
Add `docvalue_fields` as an adjunct to `fielddata_fields` which is deprecated. `docvalue_fields` will try to load the value from the docvalue and fallback to fielddata cache if docvalues are not enabled on that field.

Closes #18943
2016-07-04 10:39:49 +02:00
David Pilato 527a9c7f48 Deprecate discovery-azure and rename it to discovery-azure-classic
As discussed at https://github.com/elastic/elasticsearch-cloud-azure/issues/91#issuecomment-229113595, we know that the current `discovery-azure` plugin only works with Azure Classic VMs / Services (which is somehow Legacy now).

The proposal here is to rename `discovery-azure` to `discovery-azure-classic` in case some users are using it.
And deprecate it for 5.0.

Closes #19144.
2016-06-30 14:42:40 +02:00
Nik Everett 67bfecc070 Painless: add "".replaceAll and "".replaceFirst
These are useful methods in groovy that give you control over
the replacements used:
```
'the quick brown fox'.replaceAll(/[aeiou]/,
		m -> m.group().toUpperCase(Locale.ROOT))
```
2016-06-28 16:39:11 -04:00
Robert Muir 6d52cec2a0 Merge pull request #19092 from rmuir/more_painless_docs
cutover some docs to painless
2016-06-28 13:40:25 -04:00
Jim Ferenczi eb1e231a63 Revert "Rename `fields` to `stored_fields` and add `docvalue_fields`"
This reverts commit 2f46f53dc8.
2016-06-27 17:20:32 +02:00
Robert Muir 6fc1a22977 cutover some docs to painless 2016-06-27 09:55:16 -04:00
Robert Muir 0b2baa7f63 Merge pull request #19065 from rmuir/help_painless_docs
Bring painless docs closer to reality
2016-06-24 12:52:30 -04:00
Robert Muir 001a060c84 Bring painless docs closer to reality 2016-06-24 12:06:41 -04:00
Jim Ferenczi 2f46f53dc8 Rename `fields` to `stored_fields` and add `docvalue_fields`
`stored_fields` parameter will no longer try to retrieve fields from the _source but will only return stored fields.
`fields` will throw an exception if the user uses it.
Add `docvalue_fields` as an adjunct to `fielddata_fields` which is deprecated. `docvalue_fields` will try to load the value from the docvalue and fallback to fielddata cache if docvalues are not enabled on that field.

Closes #18943
2016-06-22 17:38:30 +02:00
Clinton Gormley c7bd1a80af Changed path.script to path.scripts in docs 2016-06-22 12:39:52 +02:00
Adrien Grand 7d63f4b8db Fix doc build. 2016-06-22 09:34:49 +02:00
Adrien Grand db9af54ec0 Remove `_timestamp` and `_ttl` on 5.x indices. #18980
This removes the ability to use `_timestamp` and `_ttl` on indices created on
or after 5.0.

Closes #18280
2016-06-22 08:35:54 +02:00
Nik Everett 6569d35094 Fail doc tests when any shard fails
ES only sends a non-200 response all shards fail but we should
fail the tests generated by docs if any of them fail.

Depending on the outcome of #18978 this might be a temporary
workaround.
2016-06-20 12:49:30 -04:00
Eric Sherman 0660de7472 Update snapshots.asciidoc (#18923) 2016-06-20 16:00:59 +02:00
Nik Everett b665d8a187 Painless: Add flag support to regexes
Painless: Add support for //m
Painless: Add support for //s
Painless: Add support for //i
Painless: Add support for //u
Painless: Add support for //U
Painless: Add support for //l
  This means "literal" and is exposed for completeness sake with
  the java api.
Painless: Add support for //c
  c enables Java's CANON_EQ (canonical equivalence) flag which makes
  unicode characters that are canonically equal match. Java's javadoc
  gives "a\u030A" being equal to "\u00E5". That is that the "a" code
  point followed by the "combining ring above" code point is equal to
  the "a with combining ring above" code point.
Update docs and add multi-flag test
Whitelist most of the Pattern class.
2016-06-16 15:00:31 -04:00
Nik Everett 8d3ef742db Painless: =~ and ==~ operators
Adds support for the find operator (=~) and the match operator (==~)
to painless's regexes. Also whitelists most of the Matcher class and
documents regex support in painless.

The find operator (=~) returns a boolean that is the result of building
a matcher on the lhs with the Pattern on the RHS and calling `find` on
it. Use it like this:

```
if (ctx._source.last =~ /b/)
```

The match operator (==~) returns boolean like find but instead of calling
`find` on the Matcher it calls `matches`.

```
if (ctx._source.last ==~ /[^aeiou].*[aeiou]/)
```

Finally, if you want the actual matcher you do:

```
Matcher m = /[aeiou]/.matcher(ctx._source.last)
```
2016-06-16 08:42:33 -04:00
Jason Tedor 8caaf9ad11 Fix thread pool docs regarding dynamic settings
Thread pool settings are no longer dynamically updatable since
da74323141. This commit removes a leftover
note from the thread pool module docs that incorrectly states that
thread pool settings are dynamically updatable.
2016-06-15 18:25:25 -04:00
Tal Levy a26260fb72 new ScriptProcessor for Ingest (#18193)
add new ScriptProcessor for executing ES Scripts within pipelines
2016-06-15 14:57:18 -07:00
Aaron Mildenstein 41810bd63c Pluralize "index" (#18811)
This doesn't just happen to "an index" unless you're restoring just one.  It reads better this way, IMO.
2016-06-13 20:05:33 +02:00
Lee Hinman c637fea84b Change the default of `include_global_state` from true to false for restores
This changes the default value to be false *only* for restore operations.

Resolves #18569
2016-06-08 10:48:36 -06:00
Jason Tedor da74323141 Register thread pool settings
This commit refactors the handling of thread pool settings so that the
individual settings can be registered rather than registering the top
level group. With this refactoring, individual plugins must now register
their own settings for custom thread pools that they need, but a
dedicated API is provided for this in the thread pool module. This
commit also renames the prefix on the thread pool settings from
"threadpool" to "thread_pool". This enables a hard break on the settings
so that:
 - some of the settings can be given more sensible names (e.g., the max
   number of threads in a scaling thread pool is now named "max" instead
   of "size")
 - change the soft limit on the number of threads in the bulk and
   indexing thread pools to a hard limit
 - the settings names for custom plugins for thread pools can be
   prefixed (e.g., "xpack.watcher.thread_pool.size")
 - remove dynamic thread pool settings

Relates #18674
2016-06-06 22:09:12 -04:00
Nicholas Knize 371c73e140 refactor matrix agg documentation from modules to main agg section 2016-06-06 07:39:00 -05:00
Nicholas Knize 90b8f5d0d8 Adding MultiValuesSource support classes and documentation to matrix stats agg module 2016-06-01 16:39:42 -05:00
Robert Muir 2d1eb89aef improve date api for expressions/painless fields 2016-05-31 09:32:33 -04:00
Jason Tedor d23db39445 Merge pull request #18594 from jasontedor/plugins-cleanup
Plugins cleanup
2016-05-26 14:46:09 -04:00
Jason Tedor d29844e597 Remove custom plugins path
This commit removes the ability to specify a custom plugins
path. Instead, the plugins path will always be a subdirectory called
"plugins" off of the home directory.
2016-05-26 10:16:25 -04:00