Commit Graph

1474 Commits

Author SHA1 Message Date
javanna 93ce36a198 separated attributes from node roles in DiscoveryNode
Node roles are now serialized as well, they are not part of the node attributes anymore. DiscoveryNodeService takes care of dividing settings into attributes and roles. DiscoveryNode always requires to pass in attributes and roles separately.
2016-03-25 20:14:27 +01:00
Jason Tedor 7f0134e725 Revert "Merge pull request #16843 from xuzha/s3-encryption"
This reverts commit 37a183d9ed, reversing
changes made to 08903f1ed8.
2016-03-24 17:11:02 -04:00
Xu Zhang 38923b89c2 Update Format, add new settings into the setting test 2016-03-24 12:16:57 -07:00
Ryan Ernst 3adaf09675 Settings: Cleanup placeholder replacement
This change moves placeholder replacement to a pkg private class for
settings. It also adds a null check when calling replacement, as
settings objects can still contain null values, because we only prohibit
nulls on file loading. Finally, this cleans up file and stream loading a
bit to not have unnecessary exception wrapping.
2016-03-24 11:54:05 -07:00
Xu Zhang 7499e3aa4a Update and rebase the init implementation.
Also removes the MD5 checks from our side, AWS S3 SDK java is doing the
check.
2016-03-24 11:21:40 -07:00
Nicolas Trésegnie ea78fd6560 Add client-side encryption
The Java Cryptography Extension (JCE) has to be installed to use this feature.
2016-03-24 11:13:37 -07:00
David Pilato 4b1ae331f0 Update after review 2016-03-23 17:32:51 +01:00
David Pilato e907b7c11e Check that S3 setting `buffer_size` is always lower than `chunk_size`
We can be better at checking `buffer_size` and `chunk_size` for S3 repositories.
For example, we know that:

* `buffer_size` should be more than `5mb`
* `chunk_size` should be no more than `5tb`
* `buffer_size` should be lower than `chunk_size`

Otherwise, setting `buffer_size` is useless.

For the record:

`chunk_size` is a Snapshot setting whatever the implementation is.
`buffer_size` is an S3 implementation setting.

Let say that you are snapshotting a 500mb file. If you set `chunk_size` to `200mb`, then Snapshot service will call S3 repository to snapshot 3 files with the following sizes:

* `200mb`
* `200mb`
* `100mb`

If you set `buffer_size` to `100mb` (AWS maximum size recommendation), the first file of `200mb` will be uploaded on S3 using the multipart feature in 2 chunks and the workflow is basically the following:

* create the multipart request and get back an `id` from AWS S3 platform
* upload part1: `100mb`
* upload part2: `100mb`
* "commit" the full upload using the `id`.

Closes #17244.
2016-03-23 10:39:54 +01:00
Simon Willnauer 1988b8b387 [TEST] Reuse EsTestCase#createAnalysisService in KuromojiAnalysisTests 2016-03-22 13:45:20 +01:00
Jun Ohtani a9a0f262af Analysis Kuromoji: Add nbest option and NumberFilter
Add nbest_cost and nbest_examples parameter to KuromojiTokenizerFactory
Add KuromojiNumberFilterFactory
2016-03-22 20:09:56 +09:00
Ryan Ernst f71f0d6010 Revert "Build: Switch to maven-publish plugin"
This reverts commit a90a2b34fc.
2016-03-18 17:22:25 -07:00
Ryan Ernst 6af4c43c4f Merge pull request #17128 from rjernst/maven_publish
Build: Switch to maven-publish plugin
2016-03-17 11:53:50 -07:00
Simon Willnauer e91a141233 Prevent index level setting from being configured on a node level
Today we allow to set all kinds of index level settings on the node level which
is error prone and difficult to get right in a consistent manner.
For instance if some analyzers are setup in a yaml config file some nodes might
not have these analyzers and then index creation fails.

Nevertheless, this change allows some selected settings to be specified on a node level
for instance:
 * `index.codec` which is used in a hot/cold node architecture and it's value is really per node or per index
 * `index.store.fs.fs_lock` which is also dependent on the filesystem a node uses

All other index level setting must be specified on the index level. For existing clusters the index must be closed
and all settings must be updated via the API on each of the indices.

Closes #16799
2016-03-17 14:42:18 +01:00
Ryan Ernst a90a2b34fc Build: Switch to maven-publish plugin
The build currently uses the old maven support in gradle. This commit
switches to use the newer maven-publish plugin. This will allow future
changes, for example, easily publishing to artifactory.

An additional part of this change makes publishing of build-tools part
of the normal publishing, instead of requiring a separate upload step
from within buildSrc. That also sets us up for a follow up to enable
precomit checks on the buildSrc code itself.
2016-03-15 19:16:37 -07:00
Jason Tedor 618441aea3 Merge pull request #17088 from jasontedor/simplify-bootstrap-settings
Bootstrap does not set system properties
2016-03-15 19:25:16 -04:00
Jason Tedor 66ba044ec5 Use setting in integration test cluster config 2016-03-15 17:45:17 -04:00
Yannick Welsch f5e6db4090 Remove System.out.println and Throwable.printStackTrace from tests 2016-03-15 15:40:37 +01:00
Yannick Welsch d14ae5f8b6 Remove Python and Javascript Benchmark classes 2016-03-15 15:02:50 +01:00
David Pilato 84c862b825 Merge remote-tracking branch 'origin/master' 2016-03-15 09:25:26 +01:00
David Pilato a3bf57d116 Upgrade azure SDK to 0.9.3
We are ATM using azure SDK 0.9.0.

Azure latest release is now 0.9.3 (released in February 2016).

<img width="1024" alt="the central repository search engine google chrome aujourd hui at 08 41 12" src="https://cloud.githubusercontent.com/assets/274222/13662836/a806ba3a-e69d-11e5-8655-4a838db2ef47.png">

Artifacts are on [maven central](http://search.maven.org/#search%7Cga%7C1%7Cg%3A%22com.microsoft.azure%22%20AND%20(a%3Aazure-serviceruntime%20OR%20a%3Aazure-servicebus%20OR%20a%3Aazure-svc-*))

Change log:

## 2016.2.18 Version 0.9.3

* Fix enum bugs in azure-svc-mgmt-websites

## 2016.1.26 Version 0.9.2

* Fix HTTP Proxy for Apache HTTP Client in Service Clients
* Key Vault: Fix KeyVaultKey to not attempt to load RSA Private Key

## 2016.1.8 Version 0.9.1

* Support HTTP Proxy
* Fix token expiration issue #557
* Service Bus: Add missing attributes: partitionKey, viaPartitionKey
* Traffic Manager: Update API version, add MinChildEndpoints for NestedEndpoints
* Media: Add support for Widevine (DRM) dynamic encryption

Closes #17042.
2016-03-15 09:18:34 +01:00
Simon Willnauer 345e988bbc Merge pull request #17072 from s1monw/add_backwards_rest_tests
Add infrastructure to run REST tests on a multi-version cluster

This change adds the infrastructure to run the rest tests on a multi-node
cluster that users 2 different minor versions of elasticsearch. It doesn't implement
any dedicated BWC tests but rather leverages the existing REST tests.

Since we don't have a real version to test against, the tests uses the current version
until the first minor / RC is released to ensure the infrastructure works.

Given the amount of problems this change already found I think it's worth having this run with our test suite by default. The structure of this infra will likely change over time but for now it's a step into the right direction. We will likely want to split it up into integTests and integBwcTests etc. so each plugin can have it's own bwc tests but that's left for future refactoring.
2016-03-15 09:17:43 +01:00
Areek Zillur c3078f4d65 adapt tests to use index uuid as folder name 2016-03-14 23:24:24 -04:00
Simon Willnauer 554bf2c282 [TEST] Test that all processors are available 2016-03-14 22:35:25 +01:00
Simon Willnauer 6f28c173e2 [TEST] Test that all processors are available 2016-03-14 21:42:37 +01:00
Adrien Grand 5596e31068 Upgrade to lucene-6.0.0-f0aa4fc. #17075 2016-03-14 07:58:52 +01:00
Jason Tedor 8a05c2a2be Bootstrap does not set system properties
Today, certain bootstrap properties are set and read via system
properties. This action-at-distance way of managing these properties is
rather confusing, and completely unnecessary. But another problem exists
with setting these as system properties. Namely, these system properties
are interpreted as Elasticsearch settings, not all of which are
registered. This leads to Elasticsearch failing to startup if any of
these special properties are set. Instead, these properties should be
kept as local as possible, and passed around as method parameters where
needed. This eliminates the action-at-distance way of handling these
properties, and eliminates the need to register these non-setting
properties. This commit does exactly that.

Additionally, today we use the "-D" command line flag to set the
properties, but this is confusing because "-D" is a special flag to the
JVM for setting system properties. This creates confusion because some
"-D" properties should be passed via arguments to the JVM (so via
ES_JAVA_OPTS), and some should be passed as arguments to
Elasticsearch. This commit changes the "-D" flag for Elasticsearch
settings to "-E".
2016-03-13 20:09:15 -04:00
David Pilato 9acb0bb28c Merge branch 'master' into pr/16598-register-filter-settings
# Conflicts:
#	core/src/main/java/org/elasticsearch/cluster/service/InternalClusterService.java
#	core/src/main/java/org/elasticsearch/common/settings/IndexScopedSettings.java
#	core/src/main/java/org/elasticsearch/common/settings/Setting.java
2016-03-13 14:52:10 +01:00
Ryan Ernst 591fb8f028 Merge branch 'master' into cli-parsing 2016-03-11 10:45:05 -08:00
Yannick Welsch 04e55ecf6b Make logging message String constant to allow static checks 2016-03-11 10:30:59 +01:00
Yannick Welsch 718876a941 Fix wrong placeholder usage in logging statements 2016-03-11 10:30:59 +01:00
Ryan Ernst 42a6869bb1 Merge pull request #17059 from elastic/fix/16864-attachment-doctypes
Fix attachments plugins with docx
2016-03-10 17:27:02 -08:00
Ryan Ernst 2f3efc3fe1 Add doc and docx rest test to mapper attachment along with
getClassLoader permission
2016-03-10 13:28:19 -08:00
Ryan Ernst 51d87d94dc Add getClassLoader perm for tika in ingest 2016-03-10 11:17:25 -08:00
thefourtheye 304cbbbf31 fix redundant stack in comments 2016-03-11 00:31:38 +05:30
David Pilato 6deabac8e8 Can not extract text from Office documents (`.docx` extension)
Add REST test for:

* `.doc`
* `.docx`

The later fails with:

```
==> Test Info: seed=DB93397128B876D4; jvm=1; suite=1
Suite: org.elasticsearch.ingest.attachment.IngestAttachmentRestIT
  2> REPRODUCE WITH: gradle :plugins:ingest-attachment:integTest -Dtests.seed=DB93397128B876D4 -Dtests.class=org.elasticsearch.ingest.attachment.IngestAttachmentRestIT -Dtests.method="test {yaml=ingest_attachment/30_files_supported/Test ingest attachment processor with .docx file}" -Des.logger.level=WARN -Dtests.security.manager=true -Dtests.locale=bg -Dtests.timezone=Europe/Athens
FAILURE 4.53s | IngestAttachmentRestIT.test {yaml=ingest_attachment/30_files_supported/Test ingest attachment processor with .docx file} <<< FAILURES!
   > Throwable #1: java.lang.AssertionError: expected [2xx] status code but api [index] returned [400 Bad Request] [{"error":{"root_cause":[{"type":"parse_exception","reason":"Error parsing document in field [field1]"}],"type":"parse_exception","reason":"Error parsing document in field [field1]","caused_by":{"type":"tika_exception","reason":"Unexpected RuntimeException from org.apache.tika.parser.microsoft.ooxml.OOXMLParser@7f85baa5","caused_by":{"type":"illegal_state_exception","reason":"access denied (\"java.lang.RuntimePermission\" \"getClassLoader\")","caused_by":{"type":"access_control_exception","reason":"access denied (\"java.lang.RuntimePermission\" \"getClassLoader\")"}}}},"status":400}]
   >    at __randomizedtesting.SeedInfo.seed([DB93397128B876D4:53C706AB86441B2C]:0)
   >    at org.elasticsearch.test.rest.section.DoSection.execute(DoSection.java:107)
   >    at org.elasticsearch.test.rest.ESRestTestCase.test(ESRestTestCase.java:395)
   >    at java.lang.Thread.run(Thread.java:745)
```

Related to #16864
2016-03-10 10:57:59 +01:00
Simon Willnauer 7a53a396e4 Remove Unneded @Inject annotations 2016-03-09 12:10:47 +01:00
Ryan Ernst 80ae2b0002 Fix more licenses 2016-03-09 00:10:59 -08:00
Ryan Ernst 1dafead2eb Fix precommit 2016-03-08 22:55:24 -08:00
Ryan Ernst fdce9d7c4d Merge branch 'master' into cli-parsing 2016-03-08 14:18:20 -08:00
Ryan Ernst e5c852f767 Convert bootstrapcli parser to jopt-simple 2016-03-08 13:39:37 -08:00
Simon Willnauer a40587b377 Merge pull request #16982 from s1monw/remove_old_version
Remove old and unsupported version constants

All version <= 2.0 are not supported anymore. This commit removes all
uses of these versions.
2016-03-07 16:03:14 +01:00
Martijn van Groningen 82d01e4315 Added ingest info to node info API, which contains a list of available processors.
Internally the put pipeline API uses this information in node info API to validate if all specified processors in a pipeline exist on all nodes in the cluster.
2016-03-07 14:44:50 +01:00
Simon Willnauer fdfb0e56f6 Remove bw compat from size mapper 2016-03-07 12:48:02 +01:00
Simon Willnauer f96900013c Remove bw compat from murmur3 mapper 2016-03-07 12:44:53 +01:00
Robert Muir 54018a5d37 upgrade to lucene 6.0.0-snapshot-bea235f
Closes #16964

Squashed commit of the following:

commit a23f9d2d29220991aa498214530753d7a5a148c6
Merge: eec9c4e 0b0a251
Author: Robert Muir <rmuir@apache.org>
Date:   Mon Mar 7 04:12:02 2016 -0500

    Merge branch 'master' into lucene6

commit eec9c4e5cd11e9c3e0b426f04894bb2a6dae4f21
Merge: bc67205 675d940
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 13:45:00 2016 -0500

    Merge branch 'master' into lucene6

commit bc67205bdfe1526eae277ab7856fc050ecbdb7b2
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 09:56:31 2016 -0500

    fix test bug

commit a60723b007ff12d97b1810cef473bd7b553a0327
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:35:35 2016 +0100

    Fix SimpleValidateQueryIT to put braces around boosted terms

commit ae3a49d7ba7ced448d2a5262e5d8ec98671a9090
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:27:25 2016 +0100

    fix multimatchquery

commit ae23fdb88a8f6d3fb7ba60fd1aaf3fd72d899aa5
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 15:20:49 2016 +0100

    Rewrite DecayFunctionScoreIT to be independent of the similarity used

    This test relied a lot on the term scoring and compared scores
    that are dependent on the similarity. This commit changes the base query
    to be a predictable constant score query.

commit 366c2d518c35d31251033f1b6f6a93f6e2ae327d
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 14:06:14 2016 +0100

    Fix scoring in tests due to changes to idf calculation.

    Lucene 6 uses a different default similarity as well as a different
    way to calculate IDF. In contrast to older version lucene 6 uses docCount per field
    to calculate the IDF not the # of docs in the index to overcome the sparse field
    cases.

commit dac99fd64ac2fa71b8d8d106fe68825e574c49f8
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 08:21:57 2016 -0500

    don't hardcoded expected termquery score

commit 6e9f340ba49ab10eed512df86d52a121aa775b0f
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 08:04:45 2016 -0500

    suppress deprecation warning until migrated to points

commit 3ac8908424b3fdad44a90a4f7bdb3eff7efd077d
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:21:43 2016 -0500

    Remove invalid test: all commits have IDs, and its illegal to do this.

commit c12976288124ad1a26467e7e848fb810548e7eab
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:06:14 2016 -0500

    don't test with unsupported back compat

commit 18bbfe76128570bc70883bf91ff4c44c82d27817
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 07:02:18 2016 -0500

    remove now invalid lucene 4 backcompat test

commit 7e730e572886f0ef2d3faba712e4256216ff01ec
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:58:52 2016 -0500

    remove now invalid lucene 4 backwards test

commit 244d2ab6868ba5ac9e0bcde3c2833743751a25ec
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:47:23 2016 -0500

    use 6.0 codec

commit 5f64d4a431a6fdaa1234adca23f154c2a1de8284
Author: Robert Muir <rmuir@apache.org>
Date:   Fri Mar 4 06:43:08 2016 -0500

    compile, javadocs, forbidden-apis, etc

commit 1f273cd62a7fe9ca8f8944acbbfc5cbdd3d81ccb
Merge: cd33921 29e3443
Author: Simon Willnauer <simonw@apache.org>
Date:   Fri Mar 4 10:45:29 2016 +0100

    Merge branch 'master' into lucene6

commit cd33921ac742ef9fb351012eff35f3c7dbda7264
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:58:37 2016 -0500

    fix hunspell dictionary loading

commit c7fdbd837b01f7defe9cb1c24e2ec65604b0dc96
Merge: 4d4190f d8948ba
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:41:53 2016 -0500

    Merge branch 'master' into lucene6

commit 4d4190fd82601aaafac6b8254ccb3edf218faa34
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:39:14 2016 -0500

    remove nocommit

commit 77ca69e288b1a41aa9595c921ed166c272a00ea8
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:38:24 2016 -0500

    clean up numericutils vs legacynumericutils

commit a466d696fbaad04b647ffbc0857a9439b583d0bf
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:32:43 2016 -0500

    upgrade spatial4j

commit 5412c747a8cfe638bacedbc8233163cb75cc3dc5
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Mar 3 23:19:28 2016 -0500

    move to 6.0.0-snapshot-8eada27

commit b32bfe924626b87e540692375ece09e7c2edb189
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:30:09 2016 +0100

    Fix some test compile errors.

commit 6ccde35e9840b03c68d1a2cd47c7923a06edf64a
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:25:51 2016 +0100

    Current Lucene version is 6.0.0.

commit f62e1015d931b4cc04c778298a8fa1ba65e97ad9
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 11:20:48 2016 +0100

    Fix compile errors in NGramTokenFilterFactory.

commit 6837c6eabf96075f743649da9b9b52dd39611c58
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:50:59 2016 +0100

    Fix the edge ngram tokenizer/filter.

commit ccd7f070de5efcdfbeb34b9555c65c4990bf1ba6
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:42:44 2016 +0100

    The missing value is now accessible through a getter.

commit bd3b77f9b28e5b05daa3d49683a9922a6baf2963
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:41:51 2016 +0100

    Remove IndexCacheableQuery.

commit 05f3091c347aeae80eeb16349ac51d2b53cf86f7
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:39:43 2016 +0100

    Fix compilation of function_score queries.

commit 81cda79a2431ac78f56b0cc5a5765387f662d801
Author: Adrien Grand <jpountz@gmail.com>
Date:   Thu Mar 3 10:35:02 2016 +0100

    Fix compile errors in BlendedTermQuery.

commit 70994ce8dd1eca0b995870974a38e20f26f96a7b
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 23:33:03 2016 -0500

    add bug ID

commit 29d4f1a71f36f646b5a6060bed3db019564a279d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 21:02:32 2016 -0500

    easy .store changes

commit 5e1a1e6fd665fa455e88d3a8987362fad5f44bb1
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 20:47:24 2016 -0500

    cleanups mostly around boosting

commit 333a669ec6c305ada5645d13ed1da0e19ec1d053
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 20:27:56 2016 -0500

    more simple fixes

commit bd5cd98a1e089c866b6b4a5e159400b110140ce6
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 19:49:38 2016 -0500

    more easy fixes and removal of ancient cruft

commit a68f419ee47da5f9c9ce5b372f01d707e902474c
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 19:35:02 2016 -0500

    cutover numerics

commit 4ca5dc1fa47dd5892db00899032133318fff3116
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:34:18 2016 -0500

    fix some constants

commit 88710a17817086e477c6c021ec346d0534b7fb88
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:14:25 2016 -0500

    Add spatial-extras jar as a core dependency

commit c8cd6726583e5ce3f546ed355d4eca037164a30d
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Mar 2 18:03:33 2016 -0500

    update to lucene 6 jars
2016-03-07 04:12:23 -05:00
David Pilato e35032950e Merge branch 'master' into pr/16598-register-filter-settings 2016-03-05 11:37:03 +01:00
David Pilato 2bb3846d1f Update after review:
* remove `ClusterScope`
* rename `ClusterSettings` to `NodeSettings`
* rename `SettingsProperty` to `Property`
2016-03-04 16:53:24 +01:00
David Pilato 76719341dc Fix after merge 2016-03-04 13:24:39 +01:00
David Pilato c11cf3bf1f Merge branch 'master' into pr/16598-register-filter-settings
# Conflicts:
#	core/src/main/java/org/elasticsearch/common/logging/ESLoggerFactory.java
#	core/src/main/java/org/elasticsearch/common/settings/Setting.java
#	core/src/test/java/org/elasticsearch/common/settings/SettingTests.java
2016-03-04 12:23:10 +01:00
David Pilato f97ce3c728 Deprecate mapper-attachments plugin
See #16910
2016-03-04 11:49:12 +01:00
Simon Willnauer 5008694ba1 Remove support for legacy checksums
Elasticsearch 5.0 doesn't support indices wiht legacy checksums anymore.
The last time we write legacy checksums was in 1.3.0 which was based
on lucene 4.9 already which means that all files have CRC32 checksums.
All indices that Elasticsearch can read today must be written with
lucene version >= 4.8 anyway so we can drop this layer of backwards
compatibility entirely.

Since we are close to upgrading to Lucene 6.0 we should get rid of this
in a more contiained change than the lucene upgrade.
2016-03-03 22:58:18 +01:00
Lee Hinman 6adbbff97c Fix organization rename in all files in project
Basically a query-replace of "https://github.com/elasticsearch/" with "https://github.com/elastic/"
2016-03-03 12:04:13 -07:00
Adrien Grand eef19be072 Deprecate string in favor of text/keyword. #16877
This commit removes the ability to use string fields on indices created on or
after 5.0. Dynamic mappings now generate text fields by default for strings
but there are plans to also add a sub keyword field (in a future PR).

Most of the changes in this commit are just about replacing string with
keyword or text. Some tests have been removed because they existed because of
corner cases of string mappings like setting ignore-above on a text field or
enabling term vectors on a keyword field which are now impossible.

The plan is to remove strings entirely in 6.0.
2016-03-03 10:20:56 +01:00
Daniel Mitterdorfer f70e5aca50 Merge remote-tracking branch 'danielmitterdorfer/simplify-azure-settings' 2016-03-03 10:02:35 +01:00
Daniel Mitterdorfer 52acf0e6e1 Use new settings infra to parse AzureStorageSettings
With this commit we simplify the parsing logic in AzureStorageSettings
by leveraging the new settings infrastructure.

Closes #16363
2016-03-03 10:01:14 +01:00
David Pilato 3f71c1d6a5 Replace `s -> s` by `Function.identity()` 2016-03-02 10:12:40 +01:00
David Pilato e4d9e46508 Fix merge with master 2016-03-02 09:55:09 +01:00
David Pilato 5fbf1b95dc Merge branch 'master' into pr/16598-register-filter-settings
# Conflicts:
#	core/src/main/java/org/elasticsearch/common/logging/ESLoggerFactory.java
#	core/src/main/java/org/elasticsearch/discovery/DiscoveryService.java
#	core/src/main/java/org/elasticsearch/discovery/DiscoverySettings.java
#	core/src/main/java/org/elasticsearch/http/HttpTransportSettings.java
#	plugins/repository-azure/src/main/java/org/elasticsearch/cloud/azure/storage/AzureStorageService.java
2016-03-02 09:43:53 +01:00
Jason Tedor aa8ee74c6c Bump Elasticsearch version to 5.0.0-SNAPSHOT
This commit bumps the Elasticsearch version to 5.0.0-SNAPSHOT in line
with the alignment of versions across the stack.

Closes #16862
2016-03-01 17:03:47 -05:00
Simon Willnauer 80e5c0acf8 Merge pull request #16860 from s1monw/issues/16485
Add setFactory permission to GceDiscoveryPlugin

This commit adds a missing permission and a simple test that
ensures we discover other nodes via a mock http endpoint.

Closes #16485
2016-03-01 08:55:43 +01:00
Nik Everett 95cc3e38fc Check test naming conventions on all modules
The big win here is catching tests that are incorrectly named and will
be skipped by gradle, providing a false sense of security.

The whole thing takes about 10 seconds on my Macbook Air, not counting
compiling the test classes, which seems worth it. Because this runs as
a gradle task with propery UP-TO-DATE handling it can be skipped if the
tests haven't been changed which should save some time.

I chose to keep this in test:framework rather than a new subproject of
buildSrc because ESIntegTestCase and doesn't inroduce any additional
dependencies.
2016-02-29 16:31:49 -05:00
Simon Willnauer 948ee3ee3f Move keystore creation to gradle - this prevents committing a keystore to the source repo 2016-02-29 22:01:39 +01:00
Boaz Leskes 195b43d66e Remove DiscoveryService and reduce guice to just Discovery #16821
DiscoveryService was a bridge into the discovery universe. This is unneeded and we can just access discovery directly or do things in a different way.

One of those different ways, is not having a dedicated discovery implementation for each our dicovery plugins but rather reuse ZenDiscovery.

UnicastHostProviders are now classified by discovery type, removing unneeded checks on plugins.

Closes #16821
2016-02-29 20:23:38 +01:00
Simon Willnauer ecca717339 Add setFactory permission to GceDiscoveryPlugin
This commit adds a missing permission and a simple test that
ensures we discover other nodes via a mock http endpoint.

Closes #16485
2016-02-29 15:48:16 +01:00
David Pilato 7a42014909 Upgrade Azure Storage client to 4.0.0
We are using `2.0.0` today but Azure team now recommends:

```xml
<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>azure-storage</artifactId>
    <version>4.0.0</version>
</dependency>
```

This new version fix the timeout issues we have seen with azure storage although #15080 adds a timeout support.
Azure storage client 2.0.0 was not passing correctly this value when it was calling Azure services.

Note that the timeout is a server side timeout and not client side timeout.
It means that it will raise only a timeout when:

* upload of blob is complete
* if azure service is not able to process the blob (and store it) within a given time range.

In which case it will raise an exception which elasticsearch can deal with:

```
java.io.IOException
    at __randomizedtesting.SeedInfo.seed([91BC11AEF16E073F:6886FA5308FCE4D8]:0)
    at com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:643)
    at com.microsoft.azure.storage.blob.BlobOutputStream.writeBlock(BlobOutputStream.java:444)
    at com.microsoft.azure.storage.blob.BlobOutputStream.access$000(BlobOutputStream.java:53)
    at com.microsoft.azure.storage.blob.BlobOutputStream$1.call(BlobOutputStream.java:388)
    at com.microsoft.azure.storage.blob.BlobOutputStream$1.call(BlobOutputStream.java:385)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.microsoft.azure.storage.StorageException: Operation could not be completed within the specified time.
    at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
    at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:305)
    at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:175)
    at com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlockInternal(CloudBlockBlob.java:1006)
    at com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlock(CloudBlockBlob.java:978)
    at com.microsoft.azure.storage.blob.BlobOutputStream.writeBlock(BlobOutputStream.java:438)
    ... 9 more
```

The following code was used to test this against Azure platform:

```java
public void testDumb() throws URISyntaxException, StorageException, IOException, InvalidKeyException {
    String connectionString = "MY-AZURE-STRING";

    CloudStorageAccount storageAccount = CloudStorageAccount.parse(connectionString);
    CloudBlobClient client = storageAccount.createCloudBlobClient();
    client.getDefaultRequestOptions().setTimeoutIntervalInMs(1000);
    CloudBlobContainer container = client.getContainerReference("dumb");
    container.createIfNotExists();
    CloudBlockBlob blob = container.getBlockBlobReference("blob");

    File sourceFile = File.createTempFile("sourceFile", ".tmp");

    try {
        int fileSize = 10000000;

        byte[] buffer = new byte[fileSize];
        Random random = new Random();
        random.nextBytes(buffer);

        logger.info("Generate local file");
        FileOutputStream fos = new FileOutputStream(sourceFile);
        fos.write(buffer);
        fos.close();
        logger.info("End generate local file");

        FileInputStream fis = new FileInputStream(sourceFile);

        logger.info("Start uploading");
        blob.upload(fis, fileSize);
        logger.info("End uploading");

    }
    finally {
        if (sourceFile.exists()) {
            sourceFile.delete();
        }
    }
}
```

With 2.0.0, the above code was not raising any exception. With 4.0.0, the exception is now thrown correctly.

The default timeout is 5 minutes. See https://github.com/Azure/azure-storage-java/blob/master/microsoft-azure-storage/src/com/microsoft/azure/storage/core/Utility.java#L352-L375

Closes #12567.

Release notes from 2.0.0:

 * Removed deprecated table AtomPub support.
 * Removed deprecated constructors which take service clients in favor of constructors which take credentials.
 * Added support for "Add" permissions on Blob SAS.
 * Added support for "Create" permissions on Blob and File SAS.
 * Added support for IP Restricted SAS and Protocol SAS.
 * Added support for Account SAS to all services.
 * Added support for Minute and Hour Metrics to FileServiceProperties and added support for File Metrics to CloudAnalyticsClient.
 * Removed deprecated startCopyFromBlob() on CloudBlob. Use startCopy() instead.
 * Removed deprecated Credentials and StorageKey classes. Please use the appropriate methods on StorageCredentialsAccountAndKey instead.

 * Fixed a bug in table where a select on a non-existent field resulted in a null reference exception if the corresponding field in the TableEntity was not nullable.
 * Fixed a bug in table where JsonParser was automatically closing the response stream before it was completely drained causing socket exhaustion.
 * Fixed a bug in StorageCredentialsAccountAndKey.updateKey(String) which prevented valid keys from being set.
 * Added CloudBlobContainer.listBlobs(final String, final boolean) method.
 * Fixed a bug in blob where using AccessConditions on block blob uploads larger than 64MB done with the upload* methods or block blob uploads done openOutputStream with would fail if the blob did not already exist.
 * Added support for setting a proxy per request. Proxy can be set on an OperationContext instance and will be used when that instance is passed to the request method.

 * Added support for SAS to the Azure File service.
 * Added support for Append Blob.
 * Added support for Access Control Lists (ACL) to File Shares.
 * Added support for getting and setting of CORS rules to File service.
 * Added support for ShareStats to File Shares.
 * Added support for copying an Azure File to another Azure File or a Block Blob asynchronously, and aborting Azure File copy operations asynchronously.
 * Added support for copying a Blob to an Azure File asynchronously.
 * Added support for setting a maximum quota property on a File Share.
 * Removed deprecated AuthenticationScheme and its getter and setter. In the future only SharedKey will be used.
 * Removed deprecated getter/setters for all request option properties on the service clients. Please use the default request options getter/setters instead.
 * Removed getSubDirectoryReference() for blob directories and file directories. Use getDirectoryReference() instead.
 * Removed getEntityClass() in TableQuery. Please use getClazzType() instead.
 * Added client-side verification for lease duration and break periods.
 * Deprecated the setters in table for timestamp as this property is only modifiable by the service.
 * Deprecated startCopyFromBlob() on CloudBlob. Use startCopy() instead.
 * Deprecated the Credentials and StorageKey classes. Please use the appropriate methods on StorageCredentialsAccountAndKey instead.
 * Deprecated constructors which take service clients in favor of constructors which take credentials.
 * Fixed a bug where the DateBackwardCompatibility flag was not applied if set on the CloudTableClient default request options.
 * Changed library behavior to retry all exceptions thrown when parsing a response object.
 * Changed behavior to stop removing query parameters passed in with the resource URI if that URI contains a SAS token. Some query parameters such as comp, restype, snapshot and api-version will still be removed.
 * Added support for logging StringToSign to SharedKey and SAS.
 * **Added a connect timeout to prevent hangs when establishing the network connection.**
 * **Made performance enhancements to the BlobOutputStream class.**

 * Fixed a bug where maximum execution time was ignored for file, queue, and table services.
 * **Changed the socket timeout to be set to the service side timeout plus 5 minutes when maximum execution time is not set.**
 * **Changed the socket timeout to default to 5 minutes rather than infinite when neither service side timeout or maximum execution time are set.**
 * Fixed a bug where MD5 was calculated for commitBlockList even though UseTransactionalMD5 was set to false.
 * Fixed a bug where selecting fields that did not exist returned an error rather than an EntityProperty with a null value.
 * Fixed a bug where table entities with a single quote in their partition or row key could be inserted but not operated on in any other way.

 * Fixed a bug for all listing API's where next() would sometimes throw an exception if hasNext() had not been called even if there were more elements to iterate on.
 * Added sequence number to the blob properties. This is populated for page blobs.
 * Creating a page blob sets its length property.
 * Added support for page blob sequence numbers and sequence number access conditions.
 * Fixed a bug in abort copy where the lease access condition was not sent to the service.
 * Fixed an issue in startCopyFromBlob where if the URI of the source blob contained certain non-ASCII characters they would not be encoded appropriately. This would result in Authorization failures.
 * Fixed a small performance issue in XML serialization.
 * Fixed a bug in BlobOutputStream and FileOutputStream where flush added data to a request pool rather than immediately committing it to the Azure service.
 * Refactored to remove the blob, queue, and file package dependency on table in the error handling code.
 * Added additional client-side logging for REST requests, responses, and errors.

Closes #15976.
2016-02-29 15:00:34 +01:00
Martijn van Groningen c7b626c615 updated SHAs 2016-02-28 13:20:39 +01:00
David Pilato d77daf3861 Use an SettingsProperty.Dynamic for dynamic properties 2016-02-28 11:06:45 +01:00
David Pilato 31b5e0888f Use an SettingsProperty enumSet
Instead of modifying methods each time we need to add a new behavior for settings, we can simply pass `SettingsProperty... properties` instead.

`SettingsProperty` could be defined then:

```
public enum SettingsProperty {
  Filtered,
  Dynamic,
  ClusterScope,
  NodeScope,
  IndexScope
 // HereGoesYours;
}
```

Then in setting code, it become much more flexible.

TODO: Note that we need to validate SettingsProperty which are added to a Setting as some of them might be mutually exclusive.
2016-02-28 00:48:04 +01:00
Sylwester Lachiewicz 3af735c69e Update MaxMind geoip2 version to 2.6
Update to align with #16801 jackson 2.7.1
2016-02-27 12:57:24 +01:00
Nik Everett ba5be0332d Remove optional logger wrappers
Removes all our logger wrappers except the wrapper for log4j1.2. If you
depend on Elasticsearch's jar in your application you'll need to declare
log4j 1.2 and/or some bridge to your favorite logger.

We did this to simplify our builds and code. No more commons-logging like
log implementation sniffing. No more optional dependency hacks in gradle.

We might one day want to use j.u.l instead of log4j. If we do want that
we can recover its wrapper by studying this commit. We didn't go directly
to j.u.l in this commit because that is a bigger change. Our logging
configuration is based on log4j1.2 and people are used to it. So it'd
be a much more fraught breaking change to do that conversion.
2016-02-26 16:41:07 -05:00
Jason Tedor d94e391e71 Use System#lineSeparator and not system property
This commit replaces a use of the system property "line.separator" and
replaces it with a dedicated method that provides the same value.

Closes #16776
2016-02-25 12:22:57 -05:00
Jack Conradson 7986770e5f Moved Painless from a plugin to a module.
Closes #16755
2016-02-21 16:50:54 -08:00
Mike McCandless 5fffede2b0 Upgrade to Lucene 5.5.0 official release 2016-02-20 17:34:16 -05:00
David Pilato 90fba97a30 Moves GCE settings to the new infra
Closes #16720.
2016-02-19 17:00:39 -08:00
Adrien Grand 4f8895eae3 Add a text field.
This new field is intended to replace analyzed string fields.
2016-02-15 10:43:44 +01:00
David Pilato aabb124209 Add filtering support within Setting class
Now we have a nice Setting infra, we can define in Setting class if a setting should be filtered or not.
So when we register a setting, setting filtering would be automatically done.

Instead of writing:

```java
Setting<String> KEY_SETTING = Setting.simpleString("cloud.aws.access_key", false, Setting.Scope.CLUSTER);
settingsModule.registerSetting(AwsEc2Service.KEY_SETTING, false);
settingsModule.registerSettingsFilterIfMissing(AwsEc2Service.KEY_SETTING.getKey());
```

We could simply write:

```java
Setting<String> KEY_SETTING = Setting.simpleString("cloud.aws.access_key", false, Setting.Scope.CLUSTER, true);
settingsModule.registerSettingsFilterIfMissing(AwsEc2Service.KEY_SETTING.getKey());
```

It also removes `settingsModule.registerSettingsFilterIfMissing` method.

The plan would be to remove as well `settingsModule.registerSettingsFilter` method but it still used with wildcards. For example in Azure Repository plugin:

```java
module.registerSettingsFilter(AzureStorageService.Storage.PREFIX + "*.account");
module.registerSettingsFilter(AzureStorageService.Storage.PREFIX + "*.key");
```

Closes #16598.
2016-02-12 10:35:54 +01:00
Nicholas Knize 52ee4c7027 upgrade to lucene 5.5.0-snapshot-850c6c2 2016-02-11 14:28:50 -06:00
David Pilato df50371c34 Merge branch 'pr/16477-aws-settings' 2016-02-11 19:47:43 +01:00
Adrien Grand a1e251af20 Remove the MapperBuilders utility class.
We can just call constructors directly.
2016-02-11 17:32:58 +01:00
David Pilato 37b0fc4f10 Migrate AWS settings to new settings infrastructure
Reintroducing commit fb7723c but now deals with setting names conflicts
Also adds java documentation for each setting

Closes #16293.
Related to https://github.com/elastic/elasticsearch/pull/16477#discussion_r52469084
2016-02-11 12:03:09 +01:00
David Pilato 7625595364 Revert Migrate AWS settings to new settings infrastructure
It breaks when you load at the same time `repository-s3` and `discovery-ec2`.
See https://github.com/elastic/elasticsearch/pull/16477#discussion_r52469084

Reopen #16293.
2016-02-10 16:14:53 +01:00
Alexander Reelsen e8d24d10dc Tests: Fix AttachmentProcessorFactoryTests to only check for existing fields 2016-02-10 15:29:16 +01:00
David Pilato fb7723c186 Migrate AWS settings to new settings infrastructure
Closes #16293.
2016-02-10 14:44:51 +01:00
javanna d5969bb33a Attachment Processor: setFieldValue only once as a map 2016-02-10 12:38:39 +01:00
javanna 4e3fb69861 [TEST] rewrite testEnglishTextDocumentWithRandomFields 2016-02-10 12:34:51 +01:00
javanna fe7469dffb Attachment processor: remove unused NAME enum 2016-02-10 12:34:21 +01:00
Yannick Welsch 848316ad0d Merge pull request #16540 from ywelsch/fix/groovy-inflation
Add permission to access sun.reflect.MethodAccessorImpl from Groovy scripts
2016-02-09 17:39:57 +01:00
Alexander Reelsen 0d4711c2fc Ingest: Add attachment processor
This is a simple port of the mapper attachment plugin to the ingest
functionality, no new features. The only option is to limit
the number of chars to prevent indexing of huge documents.

Fields can be selected in the processor as well.

Close #16303
2016-02-09 17:03:30 +01:00
Yannick Welsch 380393a5b7 Add permission to access sun.reflect.MethodAccessorImpl from Groovy scripts
Groovy uses reflection to invoke closures. These reflective calls are optimized by the JVM after "sun.reflect.inflationThreshold" number of invocations.
After inflation, access to sun.reflect.MethodAccessorImpl is required from the security manager.

Closes #16536
2016-02-09 16:47:38 +01:00
Simon Willnauer 96bcb47fc9 Detach QueryShardContext from IndexShard and remove obsolete threadlocals
IndexShard currently holds an arbitraritly used `getQueryShardContext` that comes
out of a ThreadLocal. It's usage is undefined and arbitraty since there is also
such a method with different semantics on `IndexService` This commit removes the threadLocal on
IndexShard as well as on the context itself. It's types are now a member and the QueryShardContext
lifecycle is managed byt SearchContext which passes the types on from the SearchRequest.
2016-02-08 15:05:03 +01:00
Jack Conradson f6f2d40fd5 Minor clean up.
* Minor clean up of Writer constants.
* Removed synthetic attribute from the generated constructor and method.
* Added a safeguard for maximum script length.

Closes #16457
2016-02-05 12:00:18 -08:00
Robert Muir dcec4791c4 fix eclipse to compile at all. why does the build not fail? 2016-02-04 18:57:53 -05:00
Martijn van Groningen 7a6adfd93a ingest: Added foreach processor.
This processor is useful when all elements of a json array need to be processed in the same way.
This avoids that a processor needs to be defined for each element in an array.
Also it is very likely that it is unknown how many elements are inside an json array.
2016-02-04 23:44:01 +01:00
Simon Willnauer cc520b20e1 Merge branch 'master' into fix_settings_filter 2016-02-04 09:28:41 +01:00
Nik Everett a2f07679fd Add task status
Implements a simple task status for superclasses of ReplicationRequest to
show how you can do use the status.
2016-02-03 18:21:42 -05:00
Simon Willnauer e1cf5e745d some plugins share settings - make it easy to filter them 2016-02-03 21:50:06 +01:00
Simon Willnauer baacabf9fa convert more filters 2016-02-03 20:31:16 +01:00
Simon Willnauer f770164d9a fix imports 2016-02-03 20:14:29 +01:00
Simon Willnauer e02d2e004e Rewrite SettingsFilter to be immutable
This change rewrites the entire settings filtering mechanism to be immutable.
All filters must be registered up-front in the SettingsModule. Filters that are comma-sparated are
not allowed anymore and check on registration.
This commit also adds settings filtering to the default settings recently added to ensure we don't render
filtered settings.
2016-02-03 20:05:55 +01:00
Lee Hinman 3c7f578010 Merge remote-tracking branch 'dakrone/fix-smb-plugin-master' 2016-02-03 07:50:14 -07:00
Simon Willnauer a77344d742 Merge branch 'master' into make_settings_strict 2016-02-03 13:21:49 +01:00
Simon Willnauer 4a4e523357 Merge branch 'master' into make_settings_strict 2016-02-03 11:34:12 +01:00
Robert Muir d5dc05f69e Upgrade to lucene 5.5.0-snapshot-1725675 2016-02-02 22:53:39 -05:00
Lee Hinman 3918072362 Fix calling ensureOpen() on the wrong directory
Also removes ensureCanWrite since this already passes the
TRUNCATE_EXISTING flag when opening.

Adds a REST test that fails without this fix due to the classloader
isolation.

Additionally, move SmbDirectoryWrapper into an elasticsearch package instead of a lucene one, so this would be found at compile time instead of runtime.

Forward-port of #16383
2016-02-02 16:31:16 -07:00
Tal Levy 3191fc7347 Merge pull request #16355 from talevy/fix_ingest_exception
revert PipelineFactoryError handling with throwing ElasticsearchParseException in ingest pipeline creation
2016-02-02 14:11:24 -08:00
Tal Levy 0a1580eefa revert PipelineFactoryError handling with throwing ElasticsearchParseException in ingest pipeline creation 2016-02-02 14:08:22 -08:00
Martijn van Groningen 64ac037a82 Added an ingest qa that tests processor real world like configurations.
Renamed `ingest-with-mustache` to `smoke-test-ingest-with-all-dependencies`
Also renamed `ingest-disabled` to `smoke-test-ingest-disabled` so that the name is more inline with other qa smoke test modules.
2016-02-02 22:37:24 +01:00
Jack Conradson 54011da950 Fix imports. 2016-02-02 12:26:21 -08:00
Jack Conradson 58ad172492 Merge branch 'master' into tests 2016-02-02 11:48:27 -08:00
Jack Conradson 791417ca89 Fix test file name. 2016-02-02 11:48:19 -08:00
Tanguy Leroux 865bbc2096 Remove string formatting from Terminal print methods
This can be trappy and wrong formating strings can throws format exceptions and hide the real message.
2016-02-02 19:57:16 +01:00
Simon Willnauer 818a9eefb2 Make settings validation strict
This commit enableds strict settings validation on node startup. All settings
passed to elasticsearch either through system properties, yaml files or any other
way to pass settings must be registered and valid. Settings that are unknown ie. due to
typos or due to deprecation or removal will cause the node to NOT start up. Plugins
have to declare all their settings on the `SettingsModule#registerSetting` and settings for
plugins that are not installed must be removed.

This commit also removes the ability to specify the nodes name via `-Des.name` or just `name` in the
configuration files. The node name must be prefixed with the node prexif like `node.name: Boom`. Left over
usage of `name` will also cause startup to fail.
2016-02-02 11:32:44 +01:00
Ryan Ernst a2c37c0989 CliTool: Allow unexpected exceptions to propagate
Cli tools currently catch all exceptions, and only print the exception
message, except when a special system property is set. Even with this
flag set, certain exceptions, like IOException, are captured and their
stack trace is always lost.

This change adds a UserError class, which can be used a cli tools to
specify a message to the user, as well as an exit status. All other
exceptions are propagated out of main, so java will exit with non-zero
and print the stack trace.
2016-02-01 16:35:22 -08:00
Jack Conradson e3fd6c2a00 Removed ..= token from the Lexer. Fixed related tests. 2016-02-01 16:16:04 -08:00
Daniel Mitterdorfer 3bee2d3195 Migrate Azure settings to new settings infrastructure
With this commit we migrate all Azure related settings to
the new settings infrastructure.
2016-02-01 16:34:28 +01:00
Ryan Ernst 3787f437ec Merge branch 'master' into remove_multicast 2016-02-01 07:25:45 -08:00
Simon Willnauer af0e40ec7d Merge pull request #16316 from s1monw/isseus/13685_2nd_try
Ensure all resoruces are closed on Node#close()
2016-02-01 10:17:57 +01:00
Jason Tedor 105411060c Uppercase ells ('L') in long literals
This commit removes and forbids the use of lowercase ells ('l') in long
literals because they are often hard to distinguish from the digit
representing one ('1').

Closes #16329
2016-01-30 22:16:02 -05:00
Ryan Ernst b8f08c35ec Plugin: Remove multicast plugin
closes #16310.
2016-01-29 18:41:31 -08:00
Tal Levy fca442f4d1 Introduce Pipeline Factory Error Responses in Node Ingest
When there is an exception thrown during pipeline creation within
Rest calls (in put pipeline, and simulate) We now return a structured
error response to the user with details around which processor's
configuration is the cause of the issue, or which configuration property
is misconfigured, etc.
2016-01-29 13:37:27 -08:00
Simon Willnauer 51745d7272 Call latch in a finally block 2016-01-29 17:38:33 +01:00
Simon Willnauer e24fac644a Fix AzureRepositoryF to handle exceptions on close
Fix TribeUnitTests to handle exceptions on close
2016-01-29 17:34:02 +01:00
Martijn van Groningen f5e89f7242 mappings: remove fly weight 2016-01-29 10:12:39 +01:00
Simon Willnauer fea8676a6c remove dead code 2016-01-28 17:14:52 +01:00
Simon Willnauer a149ebdb7b remove blanks 2016-01-28 17:01:01 +01:00
Simon Willnauer 687d1d83fa Convert multcast plugin settings to the new infra 2016-01-28 17:01:01 +01:00
Boaz Leskes 2a137b5548 Make index uuid available in Index, ShardRouting & ShardId
In the early days Elasticsearch used to use the index name as the index identity. Around 1.0.0 we introduced a unique index uuid which is stored in the index setting. Since then we used that uuid in a few places but it is by far not the main identifier when working with indices, partially because it's not always readily available in all places.

This PR start to make a move in the direction of using uuids instead of name by making sure that the uuid is available on the Index class (currently just a wrapper around the name) and as such also available via ShardRouting and ShardId.

Note that this is by no means an attempt to do the right thing with the uuid in all places. In almost all places it falls back to the name based comparison that was done before. It is meant as a first step towards slowly improving the situation.

Closes #16217
2016-01-28 08:40:10 +01:00
Jack Conradson 5b836dbb11 Renamed the scripting language Plan A to Painless.
Closes #16245
2016-01-27 10:37:34 -08:00
Simon Willnauer 7ff99eb89d Merge branch 'master' into trash_context_and_headers
if (name == 'expamle-fixtures') return
2016-01-27 16:33:50 +01:00
Jason Tedor 284cc3a048 Script mode settings as booleans
This commit modifies the accept values for script mode settings from
"on", "off", and "sandbox" to "true", "false", and "sandbox".
2016-01-27 06:26:58 -05:00
Jason Tedor 9944573449 Rename methods on ScriptEngineService
This commit method renames the ScriptEngineService interface methods
types, extensions, and sandboxed to getTypes, getExtensions, and
isSandboxed, respectively.
2016-01-27 06:26:04 -05:00
Jason Tedor 087e55cc51 Script mode settings
This commit converts the script mode settings to the new settings
infrastructure. This is a major refactoring of the handling of script
mode settings. This refactoring is necessary because these settings are
determined at runtime based on the registered script engines and the
registered script contexts.
2016-01-27 06:26:04 -05:00
Simon Willnauer 71c3e57aee Merge branch 'master' into trash_context_and_headers 2016-01-27 11:42:42 +01:00
Adrien Grand 35709f62b6 Be stricter about parsing boolean values in mappings.
Parsing is currently very lenient, which has the bad side-effect that if you
have a typo and pass eg. `store: fasle` this will actually be interpreted as
`store: true`. Since mappings can't be changed after the fact, it is quite bad
if it happens on an index that already contains data.

Note that this does not cover all settings that accept a boolean, but since the
PR was quite hard to build and already covers some main settirgs like `store`
or `doc_values` this would already be a good incremental improvement.
2016-01-27 09:06:00 +01:00
Martijn van Groningen 2b2552c301 test: cleanup static test resources 2016-01-26 21:17:05 +01:00
javanna 61630c2b27 migrate node.local and node.mode to new Setting infra 2016-01-26 14:40:46 +01:00
javanna 36d98478bf Merge branch 'master' into feature/ingest 2016-01-25 18:01:09 +01:00
David Pilato b24dde88de Merge branch 'malpani-aws-discovery-seoul'
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.
2016-01-25 08:25:31 +01:00
javanna 720c531364 [TEST] remove check for jvm true, site plugins have been removed 2016-01-22 16:51:51 +01:00
Yannick Welsch 296b48b9d1 Move discovery.* settings to new setting infrastructure
Closes #16182
2016-01-22 15:35:00 +01:00
Daniel Mitterdorfer e9bb3d31a3 Convert "path.*" and "pidfile" to new settings infra 2016-01-22 15:14:13 +01:00
javanna 4ad5e67433 Geoip processor: remove redundant latitude and longitude fields and make location an object with lat and lon subfields 2016-01-22 12:26:39 +01:00
Ryan Ernst df24019261 Merge pull request #16038 from rjernst/remove_site_plugin
Plugins: Remove site plugins
2016-01-21 12:32:22 -08:00
Tal Levy 3a6c2d008e rename processor_tag to tag 2016-01-21 09:05:42 -08:00
Tal Levy c55f9bbd9b Merge pull request #16132 from talevy/holdmytag
[Ingest] add an AbstractProcessor to help hold the re-used processorTag variable
2016-01-21 07:30:29 -08:00
Simon Willnauer 142547271e Merge branch 'master' into trash_context_and_headers 2016-01-21 14:42:15 +01:00
Martijn van Groningen 44465c94f8 Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-21 11:46:27 +01:00
Tal Levy f1204cb1ab add an AbstractProcessor to help hold the re-used processorTag variable 2016-01-20 14:34:26 -08:00
Robert Muir 6e7e3a2274 Update lucene to r1725675
Adds DFI (divergence from independence) provider.
Fixes test bugs passing invalid values for BM25 parameters.
2016-01-20 03:32:51 -05:00
Martijn van Groningen e2e207687d Fixes due to changes in master branch. 2016-01-19 22:03:26 +01:00
Martijn van Groningen 602a0f183e Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-19 22:01:38 +01:00
Simon Willnauer 3d0cedbabb Merge branch 'master' into trash_context_and_headers 2016-01-19 20:47:43 +01:00
Nik Everett e1e73d9914 Create default for ExecutableScript#unwrap
Tons of scripts just return the variable they are passed and that is the
most intuitive behavior so that may as well be the default implementation.
2016-01-19 13:17:11 -05:00
Jack Conradson 2249a640bb Plan A Update...
New Features:

Exceptions (throw, try/catch)
Infinite Loop Checking
Iterators added to the API

Fixes:

Improved loop path analysis
Fixed set/add issue with shortcuts in lists
Fixed minor promotion issue with incorrect type
Fixed score issue related to score extending Number

Documentation:

Added JavaDocs for some files
2016-01-19 09:49:03 -08:00
David Pilato 725d7f968b Merge branch 'fix/12567-azure-timeout' 2016-01-19 18:02:33 +01:00
David Pilato 6db5b5033c Change exception message for wrong timeout in azure repository settings
Backport in master this change: https://github.com/elastic/elasticsearch/pull/15950#discussion-diff-50128378

Related to #15080
Related to #15950
2016-01-19 18:01:54 +01:00
David Pilato d6e6ef4ec4 Merge branch 'pr/tests-azure-repo' 2016-01-19 17:07:59 +01:00
Nik Everett 6b6d01c4fe Merge pull request #16072 from nik9000/xlint_plugins
Remove remaining xlints from plugins
2016-01-19 10:54:15 -05:00
Nik Everett 4efa8c4ff5 Remove remaining xlints from plugins 2016-01-19 10:53:48 -05:00
David Pilato 8d35bca4db Add more tests for Azure Repository client selection
One test we forgot in #14843 and #13779 is the default client selection.

Most of the time, users won't define explicitly which client they want to use because they are providing only one connection to Azure storage:

```yml
cloud:
    azure:
        storage:
            my_account:
                account: your_azure_storage_account
                key: your_azure_storage_key
```

Then using the default client like this:

```sh
# This one will use the default account (my_account1)
curl -XPUT localhost:9200/_snapshot/my_backup1?pretty -d '{
  "type": "azure"
}'
```

This commit adds tests to check that the right client is still selected when no client is explicitly set when creating the snapshot.
2016-01-19 14:23:08 +01:00
David Pilato 11e7a746ae Replace server side timeout by client side timeout
This commit replaces server side timeout (which is BTW not correctly implemented in azure client 2.0.0 but fixed later #16084) with a client side timeout.

As a consequence, for each request sent to azure, azure client will raise an exception after a given amount of time (timeout).

Closes #12567
2016-01-19 13:56:08 +01:00
Simon Willnauer fbfa9f4925 Merge branch 'master' into new_index_settings 2016-01-19 10:13:48 +01:00
Simon Willnauer 8e0390b09e Register index.version_created for several analysis plugin tests 2016-01-19 09:35:32 +01:00
Adrien Grand d6cbd6f2f0 Merge pull request #16059 from jpountz/enhancement/mapping_merge_reason
Expose the reason why a mapping merge is issued.
2016-01-19 09:27:32 +01:00
Simon Willnauer 20fe8ac854 Register index.version.created in Kuromoji unittest since it's a node level setting here 2016-01-19 09:14:41 +01:00
Nik Everett 53cce3c399 Merge pull request #16071 from nik9000/xlint_lang_groovy
Remove Xlint from lang-groovy and discovery-azure
2016-01-18 21:06:11 -05:00
Ryan Ernst ef4f0a8699 Test: Make rest test framework accept http directly for the test cluster
The rest test framework, because it used to be tightly integrated with
ESIntegTestCase, currently expects the addresses for the test cluster to
be passed using the transport protocol port. However, it only uses this
to then find the http address.

This change makes ESRestTestCase extend from ESTestCase instead of
ESIntegTestCase, and changes the sysprop used to tests.rest.cluster,
which now takes the http address.

closes #15459
2016-01-18 16:44:14 -08:00
Nik Everett bd1324ccc9 Remove Xlint from lang-groovy and discovery-azure
discovery-azure didn't actually need it.

This removes all non-default Xlints from modules.
2016-01-18 18:24:44 -05:00
Nik Everett 08ca9d7645 Fix warnings in lang-plan-a
Removes all Xlint skips in lang-plan-a. Warnings were just some extra
casts which were removed, some raw types which we just needed to <?>, and
a couple of unchecked casts in data that we know to be Map<String, Object>
because that structure is super-ultra-common in scripts.
2016-01-18 15:27:51 -05:00
Adrien Grand 055953d6b3 Expose the reason why a mapping merge is issued.
This would be useful in order to only perform some validations in the case of
a mapping update and in cases when a mapping is restored eg. after a restart,
such as discussed in #15989.

This replaces the current `applyDefault` parameter which can be derived from
the mapping merge reason: the default mapping should be applied only in case of
a mapping update, if the mapping does not exist yet and if this is not the
default mapping.
2016-01-18 17:41:23 +01:00
Simon Willnauer a61723b538 convert index.store.fs.fs_lock 2016-01-18 10:17:11 +01:00
Simon Willnauer 7925e2ef84 convert IndexModule settings 2016-01-18 09:23:35 +01:00
Simon Willnauer 79f4697f3e Register MockFSDirectoryService settings 2016-01-18 09:23:34 +01:00
Simon Willnauer a8eedd0457 fix all kinds of crazy test failures 2016-01-18 09:23:34 +01:00
Simon Willnauer ceef6d7a42 fix Murmur3FieldMapperTests 2016-01-18 09:23:34 +01:00
Ryan Ernst 3b78267c71 Plugins: Remove site plugins
Site plugins used to be used for things like kibana and marvel, but
there is no longer a need since kibana (and marvel as a kibana plugin)
uses node.js. This change removes site plugins, as well as the flag for
jvm plugins. Now all plugins are jvm plugins.
2016-01-16 22:45:37 -08:00
Tal Levy 9f48df9736 Add on_failure support for verbose _simulate execution and introduce optional processor_tag to Processors 2016-01-15 14:56:20 -08:00
javanna af36e5f01e updated ingest-geoip plugin description 2016-01-15 15:16:09 +01:00
javanna 050585e89f remove BiFunction<Environment, TemplateService, Processor.Factory> in favour of Function<TemplateService, Processor.Factory>
the environment is now available through NodeModule#getNode#getEnvironment and can be retrieved during onModule(NodeModule), no need for this indirection anymore using the BiFunction
2016-01-15 15:12:53 +01:00
javanna dd7cae7c19 move DatabaseReaders initialization to IngestGeoIpPlugin#onModule 2016-01-15 15:02:35 +01:00
Martijn van Groningen 21cc0b2316 Cleanup ingest initialization code.
* Folded IngestModule into NodeModule
* Renamed IngestBootstrapper to IngestService
* Let NodeService construct IngestService and removed the Guice annotations
* Let IngestService implement Closable
2016-01-15 13:35:06 +01:00
javanna 9c06736dbd Merge branch 'master' into feature/ingest 2016-01-15 10:11:56 +01:00
David Pilato ed45ad6327 Fix Azure repository with only one primary account
Using a single azure account is now rejected.
This commit fixes this issue and adds a test for it.

This regression was introduced with #13779. Hopefully no elasticsearch version has been released since then.

Needs to be merged in 2.2, 2.x and master branches.
2016-01-14 13:50:02 +01:00
Martijn van Groningen f3883343cb Move the pipeline configuration from the dedicated index to the cluster state.
Closes #15842
2016-01-13 22:59:36 +01:00
Simon Willnauer ce42ae4cf3 Merge pull request #15922 from s1monw/feature/ingest
Review feedback and several cleanups
2016-01-13 12:04:04 +01:00
Simon Willnauer 574d1b35b3 Replace ContextAndHeaders with a ThreadPool based ThreadLocal implementation
ContextAndHeaders has a massive impact on the core infrastructure since it has to
be manually passed on to all relevant places across threads/network calls etc. For the same reason
it's also very error prone and easily forgotten on potentially relevant APIs.

The new ThreadContext is associated with a ThreadPool (node or transport client) and ensures that
headers and context registered on a current thread are inherited to new threads spawned, send across
the network to be deserialized on the receiver end as well as restored on the response handling thread
once the response is received.
2016-01-13 11:53:32 +01:00
javanna ea8065aa3d Merge branch 'master' into feature/ingest 2016-01-12 18:28:42 +01:00
Simon Willnauer 4d38a47eb5 Review feedback and several cleanups 2016-01-12 12:06:14 +01:00
Martijn van Groningen 7bdd2583aa Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-12 01:01:30 +01:00
Nik Everett 01ce49e94e Ban Serializable
1. Uses forbidden patterns to prevent things from referencing
java.io.Serializable or from mentioning serialVersionUID.
2. Uses -Xlint:-serial so we don't have to hear from javac that we aren't
declaring serialVersionUID on any classes that we make that happen to extend
Serializable.
3. Remove Serializable and serialVersionUID declarations.

I didn't use forbidden apis because it doesn't look like it has a way to ban
explicitly implementing Serializable. If you try to ban Serializable with
forbidden apis you end up banning all Exceptions and all Strings.

Closes #15847
2016-01-11 16:57:31 -05:00
Nik Everett b29c416b7c Merge pull request #15848 from nik9000/xlint3
Remove -Xlint:-deprecation from all but core
2016-01-11 16:55:13 -05:00
Martijn van Groningen cd2155311f renamed ingest plugin to ingest-geoip plugin, since it only contains the geoip processor 2016-01-08 22:50:54 +01:00
Martijn van Groningen 1637fe9e0b Moved the grok processor to its own module, so that it will available out-of-the-box, while its dependencies are isolated 2016-01-08 21:59:23 +01:00
Robert Muir 46a8a48d23 Merge pull request #15851 from rmuir/geoip-nodns
[ingest] Don't do DNS lookups from GeoIpProcessor
2016-01-08 13:17:20 -05:00
Nik Everett 6250f4dbaa Remove deprecated azure settings 2016-01-08 13:13:14 -05:00
Jason Tedor 871d1b4885 Remove and forbid use of j.u.c.ThreadLocalRandom
This commit removes and now forbids all uses of
java.util.concurrent.ThreadLocalRandom across the codebase. The
underlying issue with ThreadLocalRandom is that it can not be
seeded. This means that if ThreadLocalRandom is used in production code,
then tests that cover any code path containing ThreadLocalRandom will be
prevented from being reproducible by use of ThreadLocalRandom. Instead,
using org.elasticsearch.common.random.Randomness#get will give
reproducible sources of random when running under tests and otherwise
still give an instance of ThreadLocalRandom when running as production
code.
2016-01-08 12:23:48 -05:00
Nik Everett 98fdb39d3d Remove deprecated settings 2016-01-08 11:17:56 -05:00
javanna de2eac4c49 removed leftover jodatime dependency 2016-01-08 14:31:35 +01:00
javanna ae69d46f92 move processors that have no deps to core, also move to core rest spec and tests and set node.inget to true by default 2016-01-08 10:39:39 +01:00
Robert Muir 9c3ebb83a7 Don't do DNS lookups from GeoIpProcessor
There is no need to involve DNS in this!
2016-01-07 23:12:44 -05:00
Nik Everett 81a7607256 Remove -Xlint:-deprecation from plugins
Instead we suppress warnings about using deprecated stuff near the usage
site with a comment about why its ok.
2016-01-07 20:44:46 -05:00
javanna adac314328 revert move of IngestPlugin class
This was moved accidentally as part of a previous refactoring.
2016-01-07 18:32:46 +01:00
Nik Everett 42cb2a5f02 Merge pull request #15811 from nik9000/def_cleanup
Cleanups for Def
2016-01-07 11:03:01 -05:00
javanna 03fe38681e renamed qa package o.e.plugin.ingest to o.e.ingest
This way InternalTemplateService constructor can be set back to package private visibility
2016-01-07 15:51:52 +01:00
javanna 1ea690e814 Merge branch 'feature/ingest' into enhancement/move_to_core 2016-01-07 15:13:31 +01:00
javanna 7d7e0db91d Merge branch 'master' into feature/ingest 2016-01-07 15:09:13 +01:00
Nik Everett 52f28888d5 Merge pull request #15813 from nik9000/xlint1
Remove Xlint:-override,-fallthrough,-static
2016-01-07 08:34:40 -05:00
javanna eca1594969 start ingest thread pool only when node.ingest is set to true 2016-01-07 14:33:56 +01:00
javanna 2803ae09dc addProcessor -> registerProcessor 2016-01-07 13:25:25 +01:00
javanna 18aabd67c8 adapt qa tests for when ingest.node is set to false
CRUD and simulate apis work now fine, every node has the pipelines in memory, but node.ingest disables ingestion, meaning that any index or bulk request with a pipeline id is going to fail
2016-01-07 13:21:06 +01:00
Martijn van Groningen 9ec2e140b8 Merge branch 'master' into feature/ingest 2016-01-07 10:44:21 +01:00
Nik Everett 244120a065 Remove more Xlint skips 2016-01-06 23:53:05 -05:00
Nik Everett 0786c506dc Remove a few more Xlint skips 2016-01-06 23:28:13 -05:00
Nik Everett 20e7fa97db Remove Xlint:-override,-fallthrough,-static
Adds `@SuppressWarnings("fallthrough")` in two places where the fallthrough
is used to implement well known hashing algorithms.
2016-01-06 22:27:14 -05:00
Nik Everett 74c132afc6 Standardize some methods on varargs
Right now we define the same sort of methods as taking String arrays and
string varargs. We should standardize on one and varargs is easier to
call so lets use varargs!
2016-01-06 21:01:58 -05:00
Nik Everett 32605ecb4f Cleanups for Def
Manually I:
1. Added some missing raw types warnings suppressions.
2. Removed some unused unchecked cast warning suppressions.
3. Added <?> to Class.

I let my IDE:
1. Remove unneeded casts.
2. Reorder imports (just ignore these, everyone does).
2016-01-06 20:28:49 -05:00
javanna 9079a7e891 wip: move all the ingest infra to core 2016-01-06 19:10:44 +01:00
javanna da3f460bd1 remove ProcessorFactoryProvider 2016-01-06 17:31:40 +01:00
javanna f651f5a531 remove MapBinder guice binding for processors, use ProcessorsRegistry instead 2016-01-06 17:31:40 +01:00
javanna 94469d75f9 revert rename InternalTemplateService -> MustacheTemplateService 2016-01-06 17:31:39 +01:00
javanna 8251a50667 make ProcessorFactoryProvider extend BiFunction 2016-01-06 17:31:39 +01:00
javanna 635b9b5a46 clarified TemplateService comments
We will keep this abstractions as it's convenient, otherwise IngestDocument would depend on ScriptService directly, and would explicitly rely on mustache which is not even part of core. better to have the interface in core, and the impl as part of the ingest plugin, which relies on mustache, shipped with core by default.
2016-01-06 17:31:39 +01:00
javanna fa4dbdaea1 create ProcessorsModule during Node creation rather than as part of IngestPlugin initialization
If we createProcessorsModule as part of the plugin, other plugins will not be able to register their own processors.
2016-01-06 17:31:39 +01:00
javanna 2478aafa46 move ingest api to core 2016-01-06 17:31:39 +01:00
Martijn van Groningen 702f712204 replaced thirdPartyAudit.missingClasses with specific excludes 2016-01-06 17:28:46 +01:00
Martijn van Groningen 1eb5ae1dce fix compile errors due to changes in master 2016-01-06 17:28:35 +01:00
Martijn van Groningen e275af8a58 Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-06 17:28:08 +01:00
Martijn van Groningen 2d6adf6428 Percolator refactoring:
* Added percolator field mapper that extracts the query terms and indexes these terms with the percolator query.
* At percolate time these extracted terms are used to query percolator queries that are like to be evaluated. This can significantly cut down the time it takes to percolate. Whereas before all percolator queries were evaluated if they matches with the document being percolated.
* Changes made to percolator queries are no longer immediately visible, a refresh needs to happen before the changes are visible.
* By default the percolate api only returns upto 10 matches instead of returning all matching percolator queries.
* Made percolate more modular, so that it is easier to add unit tests.
* Added unit tests for the percolator.

Closes #12664
Closes #13646
2016-01-06 16:08:10 +01:00
Luca Cavanna eabd5e2714 Merge pull request #15787 from javanna/enhancement/processor_package
move all the processors under the same package org.elasticsearch.ingest.processor
2016-01-06 13:28:35 +01:00
javanna 1e8995d984 move all the processors under the same package org.elasticsearch.ingest.processor 2016-01-06 12:36:28 +01:00
Igor Motov a89dba27c2 Task Management: Add framework for registering and communicating with tasks
Adds task manager class and enables all activities to register with the task manager. Currently, the immutable Transport*Activity class represents activity itself shared across all requests. This PR adds and an additional structure Task that keeps track of currently running requests and can be used to communicate with these requests using TransportTaskAction.

Related to #15117
2016-01-05 12:24:43 -05:00
Tal Levy 183386173c cleanup simulate test and add docs 2016-01-04 16:10:42 -08:00
Tal Levy 82d87e7149 Merge pull request #15647 from talevy/ingest_fail_processor
[Ingest] Fail Processor
2016-01-04 11:35:51 -08:00
Tal Levy 9dd54f2b4f Introduce a fail processor 2016-01-04 11:33:50 -08:00
Tal Levy f34ce9ddf4 add on_failure context to ingest metadata during executeOnFailure 2016-01-04 11:24:27 -08:00
javanna 8de31b3c64 make index type and id optional in simulate api
Default values are _index, _type and _id.

Closes #15711
2016-01-04 13:11:44 +01:00
Adrien Grand 1a47226d9a Merge pull request #15663 from jpountz/remove/mapping_backcompat
Remove mapping backward compatibilit with pre-2.0.
2016-01-04 10:05:39 +01:00
Robert Muir 25914ae879 Merge pull request #15688 from rmuir/thirdPartyAudit3
Improve thirdPartyAudit check, round 3
2015-12-29 09:24:51 -05:00
Martijn van Groningen f2fce0edca Removed ingest runner main class, `gradle run --debug-jvm` should be used instead. 2015-12-29 13:07:23 +01:00
Martijn van Groningen 9fa3f469e9 fix test after merging in master branch 2015-12-29 13:00:27 +01:00
Martijn van Groningen 79161d77df Merge remote-tracking branch 'es/master' into feature/ingest 2015-12-29 12:56:09 +01:00
David Pilato 96b3166c6d Add timeout settings (default to 5 minutes)
By default, azure does not timeout. This commit adds support for a timeout settings which defaults to 5 minutes.
It's a timeout **per request** not a global timeout for a snapshot request.

It can be defined globally, per account or both. Defaults to `5m`.

```yml
cloud:
    azure:
        storage:
            timeout: 10s
            my_account1:
                account: your_azure_storage_account1
                key: your_azure_storage_key1
                default: true
            my_account2:
                account: your_azure_storage_account2
                key: your_azure_storage_key2
                timeout: 30s
```

In this example, timeout will be 10s for `my_account1` and 30s for `my_account2`.

Closes #14277.
2015-12-29 11:40:48 +01:00
David Pilato a49fe189b0 Support global `repositories.azure.` settings
All those repository settings can also be defined globally in `elasticsearch.yml` file using prefix `repositories.azure.`. For example:

```yml
repositories.azure:
    container: backup-container
    base_path: backups
    chunk_size: 32m
    compress": true
```

Closes #13776.
2015-12-29 10:43:01 +01:00
Robert Muir 180ab2493e Improve thirdPartyAudit check, round 3 2015-12-28 22:38:55 -05:00
Adrien Grand 5eb7555ffb Make text parsing less lenient.
It now requires that the parser is on a value.
2015-12-28 16:06:42 +01:00
Martijn van Groningen 4a0ec0da26 Merge remote-tracking branch 'es/master' into feature/ingest 2015-12-24 15:34:20 +01:00
Martijn van Groningen 1936d6118d Added index template for the '.ingest' index and added logic that ensure the index template is installed.
An index template for the '.ingest' index is required because:
* We don't want arbitrary fields in pipeline documents, because that can turn into upgrade problems if we add more properties to the pipeline dsl.
* We know what are the usages are of the '.ingest' index, so we can optimize for that and prevent that this index is used for different purposes.

Closes #15001
2015-12-24 15:18:07 +01:00
Adrien Grand af122f4151 Remove mapping backward compatibilit with pre-2.0.
This removes the backward compatibility layer with pre-2.0 indices, notably
the extraction of _id, _routing or _timestamp from the source document when a
path is defined.
2015-12-24 13:47:37 +01:00
Robert Muir d144ba24a5 Merge pull request #15588 from rmuir/hdfs2-only
merge current hdfs improvements to master
2015-12-23 18:17:22 -05:00
Robert Muir f14a21639c add cleanups from simon 2015-12-23 18:15:33 -05:00