Commit Graph

478 Commits

Author SHA1 Message Date
David Pilato 80b142d218 Azure repository: Move to named configurations as we do for S3 repository
We should have the same behavior for Azure repositories as we have for S3 (see #22762).

Instead of:

```yml
cloud:
    azure:
        storage:
            my_account1:
                account: your_azure_storage_account1
                key: your_azure_storage_key1
                default: true
            my_account2:
                account: your_azure_storage_account2
                key: your_azure_storage_key2
```

Support something like:

```
azure.client:
            default:
                account: your_azure_storage_account1
                key: your_azure_storage_key1
            my_account2:
                account: your_azure_storage_account2
                key: your_azure_storage_key2
```

Then instead of:

```
PUT _snapshot/my_backup3
{
    "type": "azure",
    "settings": {
        "account": "my_account2"
    }
}
```

Use:

```
PUT _snapshot/my_backup3
{
    "type": "azure",
    "settings": {
        "config": "my_account2"
    }
}
```

If someone uses:

```
PUT _snapshot/my_backup3
{
    "type": "azure"
}
```

It will use the `default` azure repository settings.

And mark as deprecated old settings.

Closes #22763.
2017-08-08 15:14:47 +02:00
Ryan Ernst 0266479cab Docs: Update s3 repository docs with client settings (#26033)
This commit updates the s3 repository docs to clearly mark settings as
part of the s3 client settings, as well as those that are secure and
must be stored in the elasticsearch keystore.

relates #25619
2017-08-04 11:22:49 -07:00
Ryan Ernst e23919856e Docs: Remove docs for aws region and signer type (#26006)
These settings are removed in 6.0. This commit removes the corresponding
documentation.

relates #22872
relates #23984
2017-08-01 19:16:15 -04:00
Ryan Ernst f978974bc6 Docs: Remove s3 repository integ test documentation (#26005)
The s3 repository plugin has "third party" integ tests which rely
on external service and configuration setup. These tests are really
internal verification of the plugin (and should be moved to real integ
tests). Running them is not something a user should do, and the
documentation has been out of date for all of 5.x. This commit removes
the docs, removing potential confusion for users.
2017-08-01 19:13:45 -04:00
Ryan Ernst 8ab0d10387 Add compatibility versions to main action response (#25799)
This commit adds the min wire/index compat versions to the main action
output. Not only will this make the compatility expected more
transparent, but it also allows to test which version others think the
compat versions are, similar to how we test the lucene version.
2017-07-20 13:01:41 -07:00
Clinton Gormley ff4a2519f2 Update experimental labels in the docs (#25727)
Relates https://github.com/elastic/elasticsearch/issues/19798

Removed experimental label from:
* Painless
* Diversified Sampler Agg
* Sampler Agg
* Significant Terms Agg
* Terms Agg document count error and execution_hint
* Cardinality Agg precision_threshold
* Pipeline Aggregations
* index.shard.check_on_startup
* index.store.type (added warning)
* Preloading data into the file system cache
* foreach ingest processor
* Field caps API
* Profile API

Added experimental label to:
* Moving Average Agg Prediction


Changed experimental to beta for:
* Adjacency matrix agg
* Normalizers
* Tasks API
* Index sorting

Labelled experimental in Lucene:
* ICU plugin custom rules file
* Flatten graph token filter
* Synonym graph token filter
* Word delimiter graph token filter
* Simple pattern tokenizer
* Simple pattern split tokenizer

Replaced experimental label with warning that details may change in the future:
* Analysis explain output format
* Segments verbose output format
* Percentile Agg compression and HDR Histogram
* Percentile Rank Agg HDR Histogram
2017-07-18 14:06:22 +02:00
Clinton Gormley 091b1b0765 Fixed page breaks for ICU Collation Keyword Fields 2017-07-03 17:49:28 +02:00
Lisa Cawley 03f952a838 [DOCS] Update docs to use shared attribute file (#25403)
* [DOCS] Update docs to use shared attribute file

* [DOCS] Add shared attributes to Versions.asciidoc
2017-06-27 08:33:28 -07:00
James Baiera 9c65073852 [DOCS] Clarify expected availability of HDFS for the HDFS Repository (#25220)
If a cluster is configured with an HDFS repository and a node is started, that node must be able 
to reach HDFS, or else when it attempts to add the repository from the cluster state at start up 
it will fail to connect and the repository will be left in an inconsistent state. Adding a blurb in the 
docs to outline the expected availability for HDFS when using the repository plugin.
2017-06-16 09:47:44 -04:00
David Causse ff9edb627e [analysis-icu] Allow setting unicodeSetFilter (#20814)
UnicodeSetFilter was only allowed in the icu_folding token filter.
It seems useful to expose this setting in icu_normalizer token filter
and char filter.
2017-06-16 11:08:39 +02:00
Russ Cam a0f50e8aa4 Supported Azure Storage account types (#25167)
* Supported Azure Storage account types

Add important note for Azure Storage account types

Relates #20844
2017-06-12 17:03:18 -07:00
Sanne Grinovero a2d9b0edcd List Hibernate Search (#25145)
among the community framework integrations
2017-06-09 12:05:53 +02:00
Jason Tedor 9b4a189147 Add purge option to remove plugin CLI
By default, the remove plugin CLI command preserves configuration
files. This is so that if a user is upgrading the plugin (which is done
by first removing the old version and then installing the new version)
they do not lose their configuration file. Yet, there are circumstances
where preserving the configuration file is not desired. This commit adds
a purge option to the remove plugin CLI command.

Relates #24981
2017-06-01 08:53:39 -04:00
Jason Tedor bb63577f14 Fix plugin docs for using custom config dir
This commit fixes an issue with the plugin docs incorrectly specifying
how to set a custom configuration directory. The correct way is to use
the environment variable CONF_DIR.
2017-05-26 13:40:35 -04:00
debadair bb3a59fa70 [DOCS] Fixed cross doc xref in plugin docs. 2017-05-16 17:55:47 -07:00
Ryan Ernst d74760c306 GCS Repository: Add secure storage of credentials (#24697)
This commit adds gcs credential settings to the elasticsearch keystore.
The setting name follows the same pattern as the s3 client settings,
beginning with `gcs.client.`, followed by the client name, and then the
setting name, in this case, `credentials_file`. Using the legacy service
file setting is also deprecated.
2017-05-16 17:17:37 -07:00
propulkit 77feabb3d5 Machine Permission update on Google Compute
Machine permission can be updated without deleting instances.

Backport of #24607 in master branch
2017-05-11 07:52:34 +02:00
Matt Weber b24326271e Add ICUCollationFieldMapper (#24126)
Adds a new "icu_collation" field type that exposes lucene's
ICUCollationDocValuesField.  ICUCollationDocValuesField is the replacement
for ICUCollationKeyFilter which has been deprecated since Lucene 5.
2017-05-10 10:35:11 +02:00
James Baiera f5edd5049a Fixing permission errors for `KERBEROS` security mode for HDFS Repository (#23439)
Added missing permissions required for authenticating with Kerberos to HDFS. Also implemented 
code to support authentication in the form of using a Kerberos keytab file. In order to support 
HDFS authentication, users must install a Kerberos keytab file on each node and transfer it to the 
configuration directory. When a user specifies a Kerberos principal in the repository settings the 
plugin automatically enables security for Hadoop and begins the login process. There will be a 
separate PR and commit for the testing infrastructure to support these changes.
2017-05-04 10:51:31 -04:00
Ali Beyad 48031a2c5a [DOCS] Fixes the documentation on leading forward slashes in the (#24478)
[DOCS] Fixes the documentation on leading forward slashes in the
base_path of S3 repositories

Closes #23435
2017-05-03 22:58:43 -04:00
Clinton Gormley ba552a0736 Added "release-state" support to plugin docs 2017-04-20 15:04:20 +02:00
Tanguy Leroux e81bbc288a Remove Ubuntu 12.04 (#24161)
Ubuntu 12.04 will be EOL on April 28, 2017.
2017-04-19 09:39:42 +02:00
debadair 7e1903469e [DOCS] Added note about Elastic Cloud to improve 'elastic aws' SERP results. 2017-04-12 17:57:03 -07:00
Nik Everett 7fad7c675d Rewrite the scripting security docs (#23930)
They needed to be updated now that Painless is the default and
the non-sandboxed scripting languages are going away or gone.

I dropped the entire section about customizing the classloader
whitelists. In master this barely does anything (exposes more
things to expressions).
2017-04-07 11:46:41 -04:00
David Pilato 17be03e85e Add Backoff policy to azure repository
With this commit, Azure repositories are now using an Exponential Backoff policy before failing the backup.
It uses Azure SDK default values for this policy:

* `30s` delta backoff base with
   * `3s` min
   * `90s` max
* `3` retries max

Users can define the number of retries they wish by setting `cloud.azure.storage.xxx.max_retries` where `xxx` is the azure named account.

Closes #22728.
2017-04-03 10:52:44 +02:00
David Pilato f5d41dfc9d Merge branch 'pr/remove-repositories-azure-settings' 2017-03-31 12:33:12 +02:00
Robin Clarke 677fd68f16 Fix docs for plugin install via proxy on Windows
This commit addresses an issue with the docs for plugin install via a
proxy on Windows where the HTTP proxy options were incorrectly
specified.

Relates #23757
2017-03-27 07:40:11 -04:00
Ryan Ernst 105bc0ee1f Docs: Add note about updating plugins requiring removal and reinstallation (#23597)
closes #20321
2017-03-21 11:59:21 -07:00
Arnaud Venturi 89cbb0fed5 Docs: Corrected path to elasticsearch-plugin (#23622) 2017-03-17 10:57:58 -04:00
Md.Abdulla-Al-Sun 8b6d521037 Remove Settings.settingsBuilder (#23575)
In this repository, `Settings.builder` is used everywhere although it does exactly same as `Settings.settingsBuilder`. With the reference of the commit 42526ac28e , I think mistakenly this `Settings.settingsBuilder` remains in.
2017-03-16 11:20:42 +01:00
David Pilato da907e7a7d Remove global `repositories.azure` settings
Today we have multiple ways to define settings when a user needs to create a repository:

* in `elasticsearch.yml` file using `repositories.azure` prefix
* when creating the repository itself with `PUT _snaphot/repo`

The plan is to:

* Deprecate `repositories.azure` settings in 5.x (done with #22856)
* Remove in 6.x (this PR)

Related to #22800
2017-02-20 12:22:54 +01:00
Clinton Gormley e181a020a9 Replaced absolute URLs in docs with attributes 2017-02-04 12:05:03 +01:00
Clinton Gormley c1be26f2e1 Centralised doc versions in docs/Versions.asciidoc 2017-02-04 11:16:19 +01:00
David Pilato 858333246d Merge branch 'pr/remove-azure-container-auto-creation'
# Conflicts:
#	docs/reference/migration/migrate_6_0/plugins.asciidoc
2017-01-31 09:05:43 +01:00
David Pilato 72e15b1c7c Merge branch 'pr/fix-hdfs-settings' 2017-01-30 17:52:34 +01:00
David Pilato 1898dc2554 Remove auto creation of container for azure repository
Follow up of #22857 where we deprecate automatic creation of azure containers.

BTW I found that the `AzureSnapshotRestoreServiceIntegTests` does not bring any value because it runs basically a Snapshot/Restore operation on local files which we already test in core.

So instead of trying to fix it to make it pass with this PR, I simply removed it.
2017-01-30 11:47:08 +01:00
David Pilato 17930930a7 Update after review 2017-01-26 17:10:37 +01:00
David Pilato 3804bfcc60 Read ec2 discovery address from aws instance tags
This PR adds a new option for `host_type`: `tag:TAGNAME` where `TAGNAME` is the tag field you defined for your ec2 instance.

For example if you defined a tag `my-elasticsearch-host` in ec2 and set it to `myhostname1.mydomain.com`, then
setting `host_type: tag:my-elasticsearch-host` will tell Discovery Ec2 plugin to read the host name from the
`my-elasticsearch-host` tag. In this case, it will be resolved to `myhostname1.mydomain.com`.

Closes #22566.
2017-01-26 17:10:37 +01:00
David Pilato 7d68779655 repositories.hdfs.path can not be set
Reported at: https://discuss.elastic.co/t/combine-elasticsearch-5-1-1-and-repository-hdfs/69659

If you define as described [in our docs](https://www.elastic.co/guide/en/elasticsearch/plugins/current/repository-hdfs-config.html) the following `elasticsearch.yml` settings:

```yml
repositories:
  hdfs:
    uri: "hdfs://es-master:9000/" # optional - Hadoop file-system URI
    path: "some/path" # required - path with the file-system where data is stored/loaded
```

It fails at startup because we don't register the global setting `repositories.hdfs.path` in `HdfsPlugin`.

This PR removes that from our docs so people must provide those settings only when registering the repository with:

```
PUT _snapshot/my_hdfs_repository
{
  "type": "hdfs",
  "settings": {
    "uri": "hdfs://namenode:8020/",
    "path": "elasticsearch/respositories/my_hdfs_repository",
    "conf.dfs.client.read.shortcircuit": "true"
  }
}
```

Based on issue #22800.

Closes #22301
2017-01-26 09:22:43 +01:00
animageofmine e3546d59c4 Add support for ca-central-1 region to EC2 and S3 plugins
Closes #22458 #22454
2017-01-06 16:27:08 -06:00
David Pilato 2511442a92 Merge pull request #22300 from dadoonet/doc/ingest-attachment
Adds more information about ingest attachment properties extraction
2016-12-23 11:30:19 +01:00
David Pilato e1b6166f21 Makes more obvious that we expect an array 2016-12-23 11:28:12 +01:00
David Pilato 08f556c133 Merge pull request #22310 from gameldar/patch-3
Add ingest-attachment-with-arrays section to ingest attachments doc
2016-12-23 10:33:23 +01:00
gameldar b100f18505 Fix the ingest attachment array examples
Fix up the ingest attachment array handling example so they are full
examples and validated by the build system correctly.
2016-12-23 13:51:14 +08:00
gameldar d404ee3533 Add ingest-attachment-with-arrays section to ingest attachments doc
Added a new section detailing how to use the attachment processor
within an array.

This reverts commit #22296 and instead links to the foreach processor.
2016-12-22 00:18:33 +08:00
Nic Palmer 3894ec9bae Fixed eu-west-2 entries for discovery-ec2 and repository-s3 also updated the asciidocs 2016-12-21 15:48:07 +00:00
David Pilato 6d96cdb87c Add // CONSOLE to the snippet 2016-12-21 12:46:01 +01:00
David Pilato 80843afb19 Adds more information about ingest attachment properties extraction
This is coming from thsi thread on discuss: https://discuss.elastic.co/t/ingest-attachment-plugin-exception/69167/10
2016-12-21 12:13:16 +01:00
Gameldar 71c223c383 Add link to foreach processor to ingest-attachment.asciidoc
Add a Note to the foreach processor for processing an array of attachments, as this is required to be able to process an array of attachments.
2016-12-21 15:15:19 +08:00
Tal Levy 5a90d9d7e6 add `ignore_missing` flag to ingest plugins (#22273)
added `ignore_missing` flag to:

- Attachment Processor
- GeoIP Processor
- User-Agent Processor
2016-12-20 10:53:28 -08:00
Daniel Mitterdorfer 655a95a2bb Cache results of geoip lookups (#22231)
With this commit, we introduce a cache to the geoip ingest processor.
The cache is enabled by default and caches the 1000 most recent items.
The cache size is controlled by the setting `ingest.geoip.cache_size`.

Closes #22074
2016-12-19 10:06:12 +01:00
David Pilato 14976555f2 Fix Link to virtual hosting of buckets
Was using markdown style instead of asciidoc
2016-12-16 15:25:41 +01:00
David Pilato 18a3d6b4f3 S3/Azure snapshot repo documentation wrong for "read_only"
We used to write that people should use `read_only` although it should be `readonly`.

Closes #22007.
2016-12-08 16:57:50 +01:00
David Pilato 8923b36780 Merge pull request #21956 from alexshadow007/aws_read_timeout
Add setting to set read timeout for EC2 discovery and S3 repository plugins
2016-12-07 16:00:48 +01:00
Alexander Kazakov 0a03a62ab6 Using ClientConfiguration.DEFAULT_SOCKET_TIMEOUT as default value for read timeout 2016-12-06 21:13:28 +03:00
Alexander Kazakov 1491e2dec9 Remove default value for read_timeout setting
Fix tests and docs
2016-12-05 21:29:17 +03:00
Alexander Kazakov 23550f277b Add us-east-2 AWS region 2016-12-04 20:02:05 +03:00
Alexander Kazakov 5695eaf19e Add setting to set read timeout for EC2 discovery and S3 repository plugins 2016-12-04 01:58:53 +03:00
Clinton Gormley 9b8e3c6878 Add docs for the batch mode of plugin installation (#21700)
* Add docs for the batch mode of plugin installation

Closes https://github.com/elastic/elasticsearch/issues/21490

* Updated to clarify that all permissions are granted automatically in batch mode
2016-11-24 16:41:05 +01:00
Nik Everett c79371fd5b Remove lang-python and lang-javascript (#20734)
They were deprecated in 5.0. We are concentrating on making
Painless awesome rather than supporting every language possible.

Closes #20698
2016-11-21 22:13:25 -05:00
Chris Earle 3f2d22bc61 [DOCS] Show EC2's auto attribute (#21474)
* [DOCS] Show EC2's auto attribute

This documents the `aws_availability_zone` node attribute as part of the `discovery-ec2` plugin. Also fixes outdated usage of "cloud aws".
2016-11-21 11:04:53 -05:00
Clinton Gormley 496afbbda5 Docs: Added offline install link for discovery-file plugin
Closes #21696
2016-11-21 11:47:48 +01:00
Ludovic Dubost 366241508e Adding XWiki Elasticsearch Macro (#21505)
Added link to Elasticsearch Macro with description
2016-11-19 16:51:05 +01:00
Clinton Gormley 474f63ebac Added note to ICU docs about the possible need to reindex (#21426)
when upgrading from a previous major version.

Relates #21352
2016-11-09 15:20:15 +01:00
Ryan Ernst edb9d8fc03 Docs: Update discovery plugins example configuration with discovery.zen.hosts_provider (#21390)
This new setting was added with deguicing unicast hosts providers, but
the docs were not updated.
2016-11-07 13:33:08 -08:00
Ryan Ernst dc6ed7b8d4 Remove pluggability of ZenPing (#21049)
Plugins: Remove pluggability of ZenPing

ZenPing is the part of zen discovery which knows how to ping nodes.
There is only one alternative implementation, which is just for testing.
This change removes the ability to add custom zen pings, and instead
hooks in the MockZenPing for tests through an overridden method in
MockNode. This also folds in the ZenPingService (which was really just a
single method) into ZenDiscovery, and removes the idea of having
multiple ZenPing instances. Finally, this was the last usage of the
ExtensionPoint classes, so that is also removed here.
2016-11-03 08:20:20 -07:00
Adrien Grand 7ec51d628d Make the default S3 buffer size depend on the available memory. (#21299)
Currently the default S3 buffer size is 100MB, which can be a lot for small
heaps. This pull request updates the default to be 100MB for heaps that are
greater than 2GB and 5% of the heap size otherwise.
2016-11-03 16:07:52 +01:00
Shane Connelly c2062494a9 Fix commit message that got into ingegrations doc 2016-11-02 07:01:21 -07:00
Shane Connelly eac04e2adc Drops any plugins that haven't been updated since 2.0. Updates 5.0 links/verbiage 2016-11-02 06:38:50 -07:00
Jack Conradson 185dff7346 Cleanup ScriptType (#21179)
Refactored ScriptType to clean up some of the variable and method names. Added more documentation. Deprecated the 'in' ParseField in favor of 'stored' to match the indexed scripts being replaced by stored scripts.
2016-10-31 13:48:51 -07:00
Christoph Büscher 1f5adaa824 Docs: Adding Ukrainian analyzer 2016-10-31 18:20:39 +01:00
Adrien Grand b3cc54cf0d Upgrade to lucene-6.3.0-snapshot-ed102d6 (#21150)
Lucene 6.3 is expected to be released in the next weeks so it'd be good to give
it some integration testing. I had to upgrade randomized-testing too so that
both Lucene and Elasticsearch are on the same version.
2016-10-28 14:47:15 +02:00
Jun Ohtani a66c76eb44 Merge pull request #20704 from johtani/remove_request_params_in_analyze_api
Removing request parameters in _analyze API
2016-10-27 17:43:18 +09:00
Jason Tedor 904dcc7127 Remove timeout parameter from plugin script docs
Support for this parameter was removed but the docs were not
updated. This commit removes this stale parameter from the docs.

Relates #21068
2016-10-21 10:56:46 -04:00
David Pilato 2e1539b879 Fix URL for plugins
Reported at https://discuss.elastic.co/t/ingest-geoip-for-beta1/62505
2016-10-07 23:33:13 +02:00
Jun Ohtani eca9894c5f Removing request parameters in _analyze API
Remove unused imports
Replace POST method by GET method in docs
 Add breaking changes explanation
 Fix small issue in Kuromoji docs

Closes #20246
2016-10-07 16:23:24 +09:00
Jun Ohtani 370f0b885e Removing request parameters in _analyze API
Remove request params in _analyze API without index param
Change rest-api-test using JSON
Change docs using JSON

Closes #20246
2016-10-07 16:23:24 +09:00
Martijn van Groningen a7ad11c2e7 docs: clearify that both ip4 and ip6 addresses are supported 2016-09-28 10:03:43 +02:00
Nik Everett 3ed3e5e660 Convert more docs to CONSOLE
* plugins/discovery-azure-class.asciidoc
* reference/cluster.asciidoc
* reference/modules/cluster/misc.asciidoc
* reference/modules/indices/request_cache.asciidoc

After this is merged there will be no unconvereted snippets outside
of `reference`.

Related to #18160
2016-09-21 09:36:21 -04:00
David Pilato 4c9a77e6c0 Add link to "offline install" and show the full URL 2016-09-19 15:04:29 +02:00
David Pilato 9e58ca835c Merge branch 'master' into doc/plugins-offline-url
# Conflicts:
#	docs/plugins/mapper-attachments.asciidoc
2016-09-19 14:51:55 +02:00
David Pilato dfd1eebdd0 Remove mapper attachments plugin
We now have in 5.0.0 `ingest-attachment` plugin.
We can remove `mapper-attachments` plugin for 6.0.

Closes #18837.
2016-09-19 09:01:16 +02:00
Areek Zillur 0556c93920 Merge pull request #20488 from areek/docs/clarify_plugins_installation
[DOC] Add note requiring full cluster restart for installing plugins with custom metadata
2016-09-15 17:05:27 -04:00
Areek Zillur 9b17242b5d [DOC] Add note for full cluster restart for installing plugins using custom metadata
Currently, we check if a node has the same set of custom metadata as the master
before joining the cluster. This implies freshly installing a plugin that has its
custom metadata requires a full cluster restart.
2016-09-15 17:04:45 -04:00
Luca Cavanna 37489c3274 Add clusterUUID to RestMainAction output (#20503)
Add clusterUUID to RestMainAction output

GET / now returns the clusterUUID as well as part of its output for monitoring purposes
2016-09-15 16:25:17 +02:00
Tal Levy 4704efaef4 [ingest-geoip] do not insert null-valued fields in geoip response
update geoip to not include null-valued results from database

Originally, the plugin would still insert all the requested fields, but
assign null to each one. This fixes that by not writing the fields at
all. Makes for a better experience when the null fields conflict with
the typical geo_point field mapping.
2016-09-13 18:12:02 -07:00
Ali Beyad 730f9058a1 Documentation for file-based discovery plugin (#20462)
Documentation for file-based discovery plugin.

Closes #20458
2016-09-13 20:53:10 -04:00
David Pilato 472c679981 Add offline install instructions for plugins
Follow up of https://github.com/elastic/elasticsearch/issues/15925#issuecomment-171250150

This commit adds offline install instructions for plugins.
2016-09-12 15:36:16 +02:00
Clinton Gormley 9974e3f3d8 Bumped doc versions to 6.0.0-alpha1 2016-09-08 18:29:18 +02:00
Lee Hinman 6666892038 Default `include_in_all` for numeric-like types to false
This includes:

- All regular numeric types such as int, long, scaled-float, double, etc
- IP addresses
- Dates
- Geopoints and Geoshapes

Relates to #19784
2016-09-08 09:09:48 -06:00
Nik Everett 5cff2a046d Remove most of the need for `// NOTCONSOLE`
and be much more stingy about what we consider a console candidate.

* Add `// CONSOLE` to check-running
* Fix version in some snippets
* Mark groovy snippets as groovy
* Fix versions in plugins
* Fix language marker errors
* Fix language parsing in snippets

  This adds support for snippets who's language is written like
  `[source, txt]` and `["source","js",subs="attributes,callouts"]`.

  This also makes language required for snippets which is nice because
  then we can be sure we can grep for snippets in a particular language.
2016-09-06 10:32:54 -04:00
Clinton Gormley 234ca6b1fd Fix bad link in plugin docs 2016-09-02 10:54:05 +02:00
Jack Conradson 71d8ee5eac Merge branch 'master' into deprecate 2016-09-01 08:51:29 -07:00
Jack Conradson 3b3baa6e6c Made deprecation of Groovy, Javascript, and Python more explicit. 2016-08-31 15:56:31 -07:00
Jason Tedor 750033dc4b Update docs for Log4j 2
This commit updates the logging docs for Elasticsearch to reflect the
migration to Log4j 2.
2016-08-31 15:51:52 -04:00
Jack Conradson 7930233527 Deprecate Groovy, Python, and Javascript scripts. 2016-08-30 09:06:18 -07:00
Nik Everett c24ad1df4e Mark lots of stuff `// NOTCONSOLE` 2016-08-29 08:28:16 -04:00
Nik Everett 5357d9d9ce Add `// TESTRESPONSE` to mapper-attachements docs 2016-08-29 08:28:16 -04:00
Nik Everett 3fe42beb64 Cleanup some docs
Mark one `// NOTCONSOLE`, mark another `[source,painless]`, and
another `// TESTRESPONSE` and fix a bug in it.
2016-08-26 16:01:07 -04:00
Clinton Gormley 9cce45d4af Add cloud to S3/EC2 plugin docs 2016-08-24 15:38:43 +02:00
Lee Hinman 3298a4ed38 Revert "Merge remote-tracking branch 'dakrone/exclude-numerics-from-all'"
This reverts commit 514585290c, reversing
changes made to 8563c8d897.
2016-08-23 09:24:33 -06:00
Lee Hinman d7e516c0b4 Default `include_in_all` for numeric-like types to false
This includes:

- All regular numeric types such as int, long, scaled-float, double, etc
- IP addresses
- Dates
- Geopoints and Geoshapes

Relates to #19784
2016-08-19 15:50:38 -06:00
Tanguy Leroux 807da19150 [Doc] Update plugins intro about categories of plugins core/community 2016-08-19 14:12:10 +02:00
Nik Everett f27a7f4c1a Test response snippets in kuromoji docs
Relates to #18160
2016-08-17 09:56:00 -04:00
Adrien Grand d894db1590 Only use `PUT` for index creation, not POST. #20001
Currently both `PUT` and `POST` can be used to create indices. This commit
removes support for `POST index_name` so that we can use it to index documents
with auto-generated ids once types are removed.

Relates #15613
2016-08-17 10:15:42 +02:00
Nik Everett 7542ef3173 [docs] Don't allow `[source,javascript]`
The syntax highlighter doesn't support it. Just use `js`.
2016-08-12 17:08:43 -04:00
Nik Everett ffd226efa0 Add `// NOTCONSOLE` to docs
We have 1074 snippets that look like they should be converted to
`// CONSOLE`. At least that is what `gradle docs:listConsoleCandidates`
says. This adds `// NOTCONSOLE` to explicitly mark snippets that
*shouldn't* be converted to `// CONSOLE`. After marking the blindingly
obvious ones this cuts the remaining snippet count to 1032.
2016-08-12 16:49:57 -04:00
Nik Everett 3793d0573e Add complete examples to ingest-geoip docs
Adds `// CONSOLE` and example responses to both make usage more clear
and to test the snippets.

Relates to #18160
2016-08-10 17:13:15 -04:00
David Pilato acc50d5817 Update documentation after merge with master 2016-08-10 19:07:22 +02:00
Nik Everett f204ad026c Add complete examples to some ingest docs
These examples should make it more clear what the plugins do and they
test that the snippets actually work.

Relates to #18160
2016-08-10 12:09:25 -04:00
Nik Everett 1e587406d8 Fail yaml tests and docs snippets that get unexpected warnings
Adds `warnings` syntax to the yaml test that allows you to expect
a `Warning` header that looks like:
```
    - do:
        warnings:
            - '[index] is deprecated'
            - quotes are not required because yaml
            - but this argument is always a list, never a single string
            - no matter how many warnings you expect
        get:
            index:    test
            type:    test
            id:        1
```

These are accessible from the docs with:
```
// TEST[warning:some warning]
```

This should help to force you to update the docs if you deprecate
something. You *must* add the warnings marker to the docs or the build
will fail. While you are there you *should* update the docs to add
deprecation warnings visible in the rendered results.
2016-08-04 15:23:05 -04:00
Tyler Langlois a449b8aca1 Document proxy flags explicitly and nuke Windows section
Closes #19503
2016-07-27 18:21:20 +02:00
Tyler Langlois 0a9089beb1 Correct Java setting to pass proxy options to plugin install
It appears that `proxyHost` and `proxyPort` are not valid Java
properties; they should be prefixed with either `http` or `https`. There
is no code that the flags pass through in `plugin` that munges the flags
in any way, so they should be passed through as indicated in the Java
docs.

See:
https://docs.oracle.com/javase/7/docs/technotes/guides/net/proxies.html
https://docs.oracle.com/javase/8/docs/technotes/guides/net/proxies.html
2016-07-27 18:17:19 +02:00
David Pilato 3bcf2653bb ping_timeout is documented in discovery-ec2 but does not exist in code
Also mentioned in discovery-gce

Actually ping timeout can be set using `discovery.zen.ping_timeout`.

Closes #16600.
2016-07-21 15:03:56 +02:00
David Pilato e22bd08e3c Clarify cloud.service.name entry
Closes #18577.
2016-07-21 14:10:43 +02:00
David Pilato c6c5a1b7c8 Merge branch 'master' into pr/s3-path-style-access 2016-07-19 12:55:25 +02:00
Nik Everett 7aeea764ba Remove wait_for_status=yellow from the docs
It is no longer required after 687e2e12b3.
2016-07-15 16:02:07 -04:00
James Baiera 6b298cb2b0 Updating HDFS repository plugin documentation (#19423) 2016-07-14 16:12:59 -04:00
David Pilato bdebaba8f5 Merge branch 'master' into pr/s3-path-style-access
# Conflicts:
#	docs/plugins/repository-s3.asciidoc
#	plugins/repository-s3/src/main/java/org/elasticsearch/cloud/aws/AwsS3Service.java
#	plugins/repository-s3/src/main/java/org/elasticsearch/cloud/aws/InternalAwsS3Service.java
#	plugins/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3Repository.java
#	plugins/repository-s3/src/test/java/org/elasticsearch/cloud/aws/TestAwsS3Service.java
2016-07-11 23:17:38 +02:00
Martijn van Groningen b4defafcb2 ingest: Renamed from `ingest-useragent` to `ingest-user-agent` and processor from `useragent` to `user_agent`
and on some other places did similar renaming. This is consistent with ES naming. Also made sure that the docs are navigable from the reference guide.
2016-07-07 09:43:43 +02:00
Jim Ferenczi e3fe5c9625 Add missing footer notes in mapper size docs 2016-07-05 15:12:18 +02:00
Jim Ferenczi dcf6a96725 Add doc values support to the _size field in the mapper-size plugin
This change activates the doc_values on the _size field for indices created after 5.0.0-alpha4.
It also adds a note in the breaking changes that explain the situation and how to get around it.

Closes #18334
2016-07-05 14:47:58 +02:00
Jim Ferenczi afe99fcdcd Restore reverted change now that alpha4 is out:
Rename `fields` to `stored_fields` and add `docvalue_fields`

`stored_fields` parameter will no longer try to retrieve fields from the _source but will only return stored fields.
`fields` will throw an exception if the user uses it.
Add `docvalue_fields` as an adjunct to `fielddata_fields` which is deprecated. `docvalue_fields` will try to load the value from the docvalue and fallback to fielddata cache if docvalues are not enabled on that field.

Closes #18943
2016-07-04 10:39:49 +02:00
Christoph Wurm 42addb5692 Add ingest-useragent plugin (#19074) 2016-07-01 15:49:43 +02:00
David Pilato 72c220b1df Add deprecation notice 2016-06-30 15:29:29 +02:00
David Pilato f3ddccad17 Fix documentation filenames 2016-06-30 15:26:54 +02:00
David Pilato 527a9c7f48 Deprecate discovery-azure and rename it to discovery-azure-classic
As discussed at https://github.com/elastic/elasticsearch-cloud-azure/issues/91#issuecomment-229113595, we know that the current `discovery-azure` plugin only works with Azure Classic VMs / Services (which is somehow Legacy now).

The proposal here is to rename `discovery-azure` to `discovery-azure-classic` in case some users are using it.
And deprecate it for 5.0.

Closes #19144.
2016-06-30 14:42:40 +02:00
Clinton Gormley 4f82b2de1a Fixed bad asciidoc in azure discovery 2016-06-29 16:57:57 +02:00
Alexander Reelsen 56fa751928 Plugins: Add status bar on download (#18695)
As some plugins are becoming big now, it is hard for the user to know, if the plugin
is being downloaded or just nothing happens.

This commit adds a progress bar during download, which can be disabled by using the `-q`
parameter.

In addition this updates to jimfs 1.1, which allows us to test the batch mode, as adding
security policies are now supported due to having jimfs:// protocol support in URL stream
handlers.
2016-06-29 16:44:12 +02:00
Tanguy Leroux c557663b90 Make discovery-azure work again
The discovery-plugin has been broken since 2.x because the code was not compliant with the security manager and because plugins have been refactored.

closes #18637, #15630
2016-06-28 10:05:44 +02:00
David Pilato af989c0780 Support new Asia Pacific (Mumbai) ap-south-1 AWS region
AWS [announced](http://www.allthingsdistributed.com/2016/06/introducing-aws-asia-pacific-mumbai-region.html) a new region: Asia Pacific (Mumbai)	`ap-south-1`.

We need to support it for:

* repository-s3: s3.ap-south-1.amazonaws.com or s3-ap-south-1.amazonaws.com
* discovery-ec2: ec2.ap-south-1.amazonaws.com

For reference: http://docs.aws.amazon.com/general/latest/gr/rande.html

Closes #19110.
2016-06-28 08:50:50 +02:00
Jim Ferenczi eb1e231a63 Revert "Rename `fields` to `stored_fields` and add `docvalue_fields`"
This reverts commit 2f46f53dc8.
2016-06-27 17:20:32 +02:00
Jim Ferenczi 2f46f53dc8 Rename `fields` to `stored_fields` and add `docvalue_fields`
`stored_fields` parameter will no longer try to retrieve fields from the _source but will only return stored fields.
`fields` will throw an exception if the user uses it.
Add `docvalue_fields` as an adjunct to `fielddata_fields` which is deprecated. `docvalue_fields` will try to load the value from the docvalue and fallback to fielddata cache if docvalues are not enabled on that field.

Closes #18943
2016-06-22 17:38:30 +02:00
Jim Ferenczi 881afcba60 Fixed tests that failed now that BM25 is the default similarity. 2016-06-21 15:42:42 +02:00
David Pilato bf7a6f5509 Merge branch 'pr/update-aws-sdk'
# Conflicts:
#	plugins/repository-s3/src/main/java/org/elasticsearch/plugin/repository/s3/S3RepositoryPlugin.java
2016-06-17 17:23:12 +02:00
Mateusz Rzepa bce1abec48 Update integrations.asciidoc (#18915) 2016-06-17 16:20:03 +02:00
David Pilato 63223928dc Merge branch 'master' into pr/update-aws-sdk 2016-06-15 14:48:41 +02:00
Nik Everett 9c85569883 Test docs for plugins
We weren't doing it before because we weren't starting the plugins.
Now we are.

The hardest part of this was handling the files the tests expect
to be on the filesystem. extraConfigFiles was broken.
2016-06-14 14:32:29 -04:00
Christoph Wurm cd0473159a Installation and removal instructions for Ingest Plugins (#18769)
* Add install and remove instructions to ingest-geoip and ingest-attachment docs
2016-06-08 21:55:59 +02:00
Martijn van Groningen 3dd3ed4905 ingest: Upgrade geoip processor's dependencies and database files
The database files have been doubled in size compared to the previous files being used.
For this reason the database files are now gzip compressed, which required using
`GZIPInputStream` when loading database files.
2016-06-08 18:41:48 +02:00
Alexander Kazakov 8ff0077b52 Update repository-gcs.asciidoc
Fixed link
2016-05-27 17:44:58 +03:00
David Pilato 623a5b7a85 Merge branch 'master' into pr/update-aws-sdk 2016-05-27 10:13:35 +02:00
David Pilato a445654123 Fix after review
* changes `throttle_retries` to `use_throttle_retries`
* removes registering of all individual repository settings when the plugin starts. Not needed
* adds more comment about deprecated method in AWS SDK we need to implement though in a Delegate class within our tests
2016-05-27 10:13:16 +02:00
Jason Tedor d29844e597 Remove custom plugins path
This commit removes the ability to specify a custom plugins
path. Instead, the plugins path will always be a subdirectory called
"plugins" off of the home directory.
2016-05-26 10:16:25 -04:00
Nik Everett 72eb621bce Docs: Replace [source,json] with [source,js]
The syntax highlighter only supports [source,js].

Also adds a check to the rest test generator that runs during
the build that'll fail the build if it sees `[source,json]`.
2016-05-24 11:17:27 -04:00
Tanguy Leroux 1f011f9dea Remove Delete-By-Query plugin
closes #18469
2016-05-24 13:28:20 +02:00
Jason Tedor c257e2c51f Remove settings and system properties entanglement
Today when parsing settings during bootstrap, we add a system property
for every Elasticsearch setting. Additionally, settings can be set via
system properties. This commit simplifies this situation.
 - settings are no longer propogated to system properties
 - system properties can not be used to set settings
 - the "es." prefix on settings is no longer required (nor permitted)
 - test logging has a dedicated system property (tests.logger.level)

Relates #18198
2016-05-19 14:08:08 -04:00
David Pilato f4cd3bd348 Merge branch 'master' into pr/update-aws-sdk 2016-05-19 16:55:21 +02:00
David Pilato e289de6e96 Move `throttle_retries` under `repositories.s3.` prefix or per repository
I initially wrongly put this setting under `cloud.aws.s3.` prefix which does not make sense. It should be placed at the same place as `max_retries`.

Also applied @tlrx comments. We should set this even if max_retries is not set (when using default values).

Also added some documentation about this setting.
2016-05-19 16:50:37 +02:00
Tanguy Leroux 35d3bdab84 Add Google Cloud Storage repository plugin
Closes #12880
2016-05-19 13:26:23 +02:00
Clinton Gormley 897fe9108a Removed docs from mapper-attachment about copy_to
Closes #18361
2016-05-15 19:32:31 +02:00
Lee Hinman efff3918d8 Remove support for mulitple languages per scripting engine 2016-05-13 09:24:31 -06:00
Jason Tedor 7d1fd17172 Remove plugin script parsing of system properties
The plugin script parses command-line options looking for Java system
properties and extracts these arguments to pass to the java command when
starting the JVM. Since elasticsearch-plugin allows arbitrary user
arguments to the JVM via ES_JAVA_OPTS, this parsing is unnecessary. This
commit removes this unnecessary 

Relates #18207
2016-05-09 13:06:18 -04:00
Clinton Gormley 3f594089c2 Renamed all AUTOSENSE snippets to CONSOLE (#18210) 2016-05-09 15:42:23 +02:00
Clinton Gormley 07d8b3eb2b Fixed a bad asciidoc link 2016-05-04 22:48:51 +02:00
debadair e49d21bdd6 Docs: Fixed broken cross doc links to script topics. 2016-05-04 13:43:08 -07:00
David Pilato 7cc8a1419b Update after rebase onto master 2016-04-29 15:39:51 +02:00
David Pilato d7eb375d24 Merge branch 'master' into pr/s3-path-style-access
# Conflicts:
#	plugins/repository-s3/src/main/java/org/elasticsearch/cloud/aws/AwsS3Service.java
#	plugins/repository-s3/src/main/java/org/elasticsearch/cloud/aws/InternalAwsS3Service.java
#	plugins/repository-s3/src/main/java/org/elasticsearch/repositories/s3/S3Repository.java
#	plugins/repository-s3/src/test/java/org/elasticsearch/cloud/aws/TestAwsS3Service.java
2016-04-29 15:21:16 +02:00
xuzha cd527c5b92 Add support for customizing the rule file in ICU tokenizer
Lucene allows to create a ICUTokenizer with a special config argument
enabling the customization of the rule based iterator by providing
custom rules files.

This commit enable this feature. Users could provide a list of RBBI rule
files to ICU tokenizer.

closes #13146
2016-04-22 12:39:20 -07:00
Martijn van Groningen dd2184ab25 ingest: Streamline option naming for several processors:
* `rename` processor, renamed `to` to `target_field`
* `date` processor, renamed `match_field` to `field` and renamed `match_formats` to `formats`
* `geoip` processor, renamed `source_field` to `field` and renamed `fields` to `properties`
* `attachment` processor, renamed `source_field` to `field` and renamed `fields` to `properties`

Closes #17835
2016-04-21 13:40:43 +02:00
Clinton Gormley 098b2e03b5 Removed all references to site plugins from plugin docs 2016-04-12 19:28:09 +02:00
Alexander Reelsen da19ddf3e6 Ingest Attachment: Allow to prevent base64 conversions by using raw bytes (#16601)
CBOR is natively supported in Elasticsearch and allows for byte arrays.
This means, that by using CBOR the user can prevent base64 conversions
for the data being sent back and forth.

This PR adds support to extract data from a byte array in addition to
a string. This also required to add a ByteArrayValueSource class.
2016-04-11 14:14:56 +02:00
Clinton Gormley 88c5dfeca4 Docs: Removed references to deprecated functionality
* search_type=count
* DFS in term vectors
* Replaced string with text/keyword as appropriate
2016-04-07 13:33:35 +02:00
Clinton Gormley 7d4ed5b19e Changed JAVA_OPTS to ES_JAVA_OPTS in plugin docs 2016-04-03 16:52:37 +02:00
Clinton Gormley 6ff947427d Fixed plugin docs links to dir layouts 2016-04-03 16:50:28 +02:00
javanna 27d4994aff Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-24 18:10:11 +01:00
javanna 030453d320 Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-23 11:25:34 +01:00
David Pilato e907b7c11e Check that S3 setting `buffer_size` is always lower than `chunk_size`
We can be better at checking `buffer_size` and `chunk_size` for S3 repositories.
For example, we know that:

* `buffer_size` should be more than `5mb`
* `chunk_size` should be no more than `5tb`
* `buffer_size` should be lower than `chunk_size`

Otherwise, setting `buffer_size` is useless.

For the record:

`chunk_size` is a Snapshot setting whatever the implementation is.
`buffer_size` is an S3 implementation setting.

Let say that you are snapshotting a 500mb file. If you set `chunk_size` to `200mb`, then Snapshot service will call S3 repository to snapshot 3 files with the following sizes:

* `200mb`
* `200mb`
* `100mb`

If you set `buffer_size` to `100mb` (AWS maximum size recommendation), the first file of `200mb` will be uploaded on S3 using the multipart feature in 2 chunks and the workflow is basically the following:

* create the multipart request and get back an `id` from AWS S3 platform
* upload part1: `100mb`
* upload part2: `100mb`
* "commit" the full upload using the `id`.

Closes #17244.
2016-03-23 10:39:54 +01:00
Jun Ohtani a9a0f262af Analysis Kuromoji: Add nbest option and NumberFilter
Add nbest_cost and nbest_examples parameter to KuromojiTokenizerFactory
Add KuromojiNumberFilterFactory
2016-03-22 20:09:56 +09:00
javanna bf390a935e Merge branch 'master' into enhancement/remove_node_client_setting 2016-03-21 17:18:23 +01:00
Clinton Gormley d83e12094e Docs: Added redirect entries for multicast plugin and the cloud plugins 2016-03-16 12:31:00 +01:00
Jason Tedor 8a05c2a2be Bootstrap does not set system properties
Today, certain bootstrap properties are set and read via system
properties. This action-at-distance way of managing these properties is
rather confusing, and completely unnecessary. But another problem exists
with setting these as system properties. Namely, these system properties
are interpreted as Elasticsearch settings, not all of which are
registered. This leads to Elasticsearch failing to startup if any of
these special properties are set. Instead, these properties should be
kept as local as possible, and passed around as method parameters where
needed. This eliminates the action-at-distance way of handling these
properties, and eliminates the need to register these non-setting
properties. This commit does exactly that.

Additionally, today we use the "-D" command line flag to set the
properties, but this is confusing because "-D" is a special flag to the
JVM for setting system properties. This creates confusion because some
"-D" properties should be passed via arguments to the JVM (so via
ES_JAVA_OPTS), and some should be passed as arguments to
Elasticsearch. This commit changes the "-D" flag for Elasticsearch
settings to "-E".
2016-03-13 20:09:15 -04:00
Ryan Ernst 5f3d0067f8 Merge pull request #17024 from rjernst/cli-parsing
Cli: Switch to jopt-simple
2016-03-11 12:35:39 -08:00
Ryan Ernst 3f44e1d429 Remove old reference to site plugins example in docs 2016-03-11 11:53:20 -08:00
Ryan Ernst 591fb8f028 Merge branch 'master' into cli-parsing 2016-03-11 10:45:05 -08:00
Ed Winn c4934f5250 Current link returns 404. Updated 2016-03-10 16:52:30 -07:00
Ryan Ernst 3836f3a736 Remove reference to standalonerunner 2016-03-08 13:40:39 -08:00
javanna e5d9328a2d [DOCS] adapt docs to node.client setting removal 2016-03-05 10:55:19 +01:00
Martijn van Groningen 116acee1dd Merge pull request #16946 from dedemorton/ingest_doc_edit
Improve the ingest documentation.
2016-03-04 11:49:19 +01:00
DeDe Morton 4d0124e65c Edits to ingest plugin docs 2016-03-03 22:49:31 -08:00
Lee Hinman 6adbbff97c Fix organization rename in all files in project
Basically a query-replace of "https://github.com/elasticsearch/" with "https://github.com/elastic/"
2016-03-03 12:04:13 -07:00
Clinton Gormley 69b5b1920f Merge pull request #16907 from centic9/patch-2
Elasticsearch monitoring support for Dynatrace Application Monitoring
2016-03-02 15:25:54 +01:00
Clinton Gormley 5bb744bfde Changed v3.0.0 to v5.0.0 in plugin docs 2016-03-02 11:57:42 +01:00
David Pilato 7a42014909 Upgrade Azure Storage client to 4.0.0
We are using `2.0.0` today but Azure team now recommends:

```xml
<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>azure-storage</artifactId>
    <version>4.0.0</version>
</dependency>
```

This new version fix the timeout issues we have seen with azure storage although #15080 adds a timeout support.
Azure storage client 2.0.0 was not passing correctly this value when it was calling Azure services.

Note that the timeout is a server side timeout and not client side timeout.
It means that it will raise only a timeout when:

* upload of blob is complete
* if azure service is not able to process the blob (and store it) within a given time range.

In which case it will raise an exception which elasticsearch can deal with:

```
java.io.IOException
    at __randomizedtesting.SeedInfo.seed([91BC11AEF16E073F:6886FA5308FCE4D8]:0)
    at com.microsoft.azure.storage.core.Utility.initIOException(Utility.java:643)
    at com.microsoft.azure.storage.blob.BlobOutputStream.writeBlock(BlobOutputStream.java:444)
    at com.microsoft.azure.storage.blob.BlobOutputStream.access$000(BlobOutputStream.java:53)
    at com.microsoft.azure.storage.blob.BlobOutputStream$1.call(BlobOutputStream.java:388)
    at com.microsoft.azure.storage.blob.BlobOutputStream$1.call(BlobOutputStream.java:385)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at java.util.concurrent.FutureTask.run(FutureTask.java:266)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
    at java.lang.Thread.run(Thread.java:745)
Caused by: com.microsoft.azure.storage.StorageException: Operation could not be completed within the specified time.
    at com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
    at com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:305)
    at com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:175)
    at com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlockInternal(CloudBlockBlob.java:1006)
    at com.microsoft.azure.storage.blob.CloudBlockBlob.uploadBlock(CloudBlockBlob.java:978)
    at com.microsoft.azure.storage.blob.BlobOutputStream.writeBlock(BlobOutputStream.java:438)
    ... 9 more
```

The following code was used to test this against Azure platform:

```java
public void testDumb() throws URISyntaxException, StorageException, IOException, InvalidKeyException {
    String connectionString = "MY-AZURE-STRING";

    CloudStorageAccount storageAccount = CloudStorageAccount.parse(connectionString);
    CloudBlobClient client = storageAccount.createCloudBlobClient();
    client.getDefaultRequestOptions().setTimeoutIntervalInMs(1000);
    CloudBlobContainer container = client.getContainerReference("dumb");
    container.createIfNotExists();
    CloudBlockBlob blob = container.getBlockBlobReference("blob");

    File sourceFile = File.createTempFile("sourceFile", ".tmp");

    try {
        int fileSize = 10000000;

        byte[] buffer = new byte[fileSize];
        Random random = new Random();
        random.nextBytes(buffer);

        logger.info("Generate local file");
        FileOutputStream fos = new FileOutputStream(sourceFile);
        fos.write(buffer);
        fos.close();
        logger.info("End generate local file");

        FileInputStream fis = new FileInputStream(sourceFile);

        logger.info("Start uploading");
        blob.upload(fis, fileSize);
        logger.info("End uploading");

    }
    finally {
        if (sourceFile.exists()) {
            sourceFile.delete();
        }
    }
}
```

With 2.0.0, the above code was not raising any exception. With 4.0.0, the exception is now thrown correctly.

The default timeout is 5 minutes. See https://github.com/Azure/azure-storage-java/blob/master/microsoft-azure-storage/src/com/microsoft/azure/storage/core/Utility.java#L352-L375

Closes #12567.

Release notes from 2.0.0:

 * Removed deprecated table AtomPub support.
 * Removed deprecated constructors which take service clients in favor of constructors which take credentials.
 * Added support for "Add" permissions on Blob SAS.
 * Added support for "Create" permissions on Blob and File SAS.
 * Added support for IP Restricted SAS and Protocol SAS.
 * Added support for Account SAS to all services.
 * Added support for Minute and Hour Metrics to FileServiceProperties and added support for File Metrics to CloudAnalyticsClient.
 * Removed deprecated startCopyFromBlob() on CloudBlob. Use startCopy() instead.
 * Removed deprecated Credentials and StorageKey classes. Please use the appropriate methods on StorageCredentialsAccountAndKey instead.

 * Fixed a bug in table where a select on a non-existent field resulted in a null reference exception if the corresponding field in the TableEntity was not nullable.
 * Fixed a bug in table where JsonParser was automatically closing the response stream before it was completely drained causing socket exhaustion.
 * Fixed a bug in StorageCredentialsAccountAndKey.updateKey(String) which prevented valid keys from being set.
 * Added CloudBlobContainer.listBlobs(final String, final boolean) method.
 * Fixed a bug in blob where using AccessConditions on block blob uploads larger than 64MB done with the upload* methods or block blob uploads done openOutputStream with would fail if the blob did not already exist.
 * Added support for setting a proxy per request. Proxy can be set on an OperationContext instance and will be used when that instance is passed to the request method.

 * Added support for SAS to the Azure File service.
 * Added support for Append Blob.
 * Added support for Access Control Lists (ACL) to File Shares.
 * Added support for getting and setting of CORS rules to File service.
 * Added support for ShareStats to File Shares.
 * Added support for copying an Azure File to another Azure File or a Block Blob asynchronously, and aborting Azure File copy operations asynchronously.
 * Added support for copying a Blob to an Azure File asynchronously.
 * Added support for setting a maximum quota property on a File Share.
 * Removed deprecated AuthenticationScheme and its getter and setter. In the future only SharedKey will be used.
 * Removed deprecated getter/setters for all request option properties on the service clients. Please use the default request options getter/setters instead.
 * Removed getSubDirectoryReference() for blob directories and file directories. Use getDirectoryReference() instead.
 * Removed getEntityClass() in TableQuery. Please use getClazzType() instead.
 * Added client-side verification for lease duration and break periods.
 * Deprecated the setters in table for timestamp as this property is only modifiable by the service.
 * Deprecated startCopyFromBlob() on CloudBlob. Use startCopy() instead.
 * Deprecated the Credentials and StorageKey classes. Please use the appropriate methods on StorageCredentialsAccountAndKey instead.
 * Deprecated constructors which take service clients in favor of constructors which take credentials.
 * Fixed a bug where the DateBackwardCompatibility flag was not applied if set on the CloudTableClient default request options.
 * Changed library behavior to retry all exceptions thrown when parsing a response object.
 * Changed behavior to stop removing query parameters passed in with the resource URI if that URI contains a SAS token. Some query parameters such as comp, restype, snapshot and api-version will still be removed.
 * Added support for logging StringToSign to SharedKey and SAS.
 * **Added a connect timeout to prevent hangs when establishing the network connection.**
 * **Made performance enhancements to the BlobOutputStream class.**

 * Fixed a bug where maximum execution time was ignored for file, queue, and table services.
 * **Changed the socket timeout to be set to the service side timeout plus 5 minutes when maximum execution time is not set.**
 * **Changed the socket timeout to default to 5 minutes rather than infinite when neither service side timeout or maximum execution time are set.**
 * Fixed a bug where MD5 was calculated for commitBlockList even though UseTransactionalMD5 was set to false.
 * Fixed a bug where selecting fields that did not exist returned an error rather than an EntityProperty with a null value.
 * Fixed a bug where table entities with a single quote in their partition or row key could be inserted but not operated on in any other way.

 * Fixed a bug for all listing API's where next() would sometimes throw an exception if hasNext() had not been called even if there were more elements to iterate on.
 * Added sequence number to the blob properties. This is populated for page blobs.
 * Creating a page blob sets its length property.
 * Added support for page blob sequence numbers and sequence number access conditions.
 * Fixed a bug in abort copy where the lease access condition was not sent to the service.
 * Fixed an issue in startCopyFromBlob where if the URI of the source blob contained certain non-ASCII characters they would not be encoded appropriately. This would result in Authorization failures.
 * Fixed a small performance issue in XML serialization.
 * Fixed a bug in BlobOutputStream and FileOutputStream where flush added data to a request pool rather than immediately committing it to the Azure service.
 * Refactored to remove the blob, queue, and file package dependency on table in the error handling code.
 * Added additional client-side logging for REST requests, responses, and errors.

Closes #15976.
2016-02-29 15:00:34 +01:00
Clinton Gormley 8830817fa3 Merge pull request #16827 from ayushsangani/patch-3
Modify path of Servlet Transport
2016-02-29 00:59:44 +01:00
Clinton Gormley 2d56eed306 Merge pull request #16785 from dsem/patch-1
Fix python script filename extension
2016-02-28 23:04:41 +01:00
Itamar Syn-Hershko 8ea6264f55 Format settings in discovery-ec2 docs
Closes #16846
2016-02-28 11:15:22 -05:00
David Pilato 90fba97a30 Moves GCE settings to the new infra
Closes #16720.
2016-02-19 17:00:39 -08:00
David Pilato 55d9b6878b Deprecate Mapper Attachment Plugin
Now that we have the ingest-attachment plugin (https://github.com/elastic/elasticsearch/pull/16490)  we should deprecate the mapper-attachment plugin.

Closes #16650.
2016-02-15 16:40:12 +01:00
DeDe Morton 461f329cd8 Add ingest plugins to Elasticsearch plugin docs 2016-02-12 07:37:50 -08:00
Jim Ferenczi b146f3ecb3 Pack all the plugin files into a single folder named elasticsearch at the root of the plugin zip. 2016-02-10 10:13:05 +01:00
Nik Everett 1c741f56b9 Merge pull request #16529 from dongjoon-hyun/fix_typos_in_docs
Fix typos in docs.
2016-02-09 14:00:06 -05:00
Alexander Reelsen 0d4711c2fc Ingest: Add attachment processor
This is a simple port of the mapper attachment plugin to the ingest
functionality, no new features. The only option is to limit
the number of chars to prevent indexing of huge documents.

Fields can be selected in the processor as well.

Close #16303
2016-02-09 17:03:30 +01:00
Dongjoon Hyun 21ea552070 Fix typos in docs. 2016-02-09 02:07:32 -08:00
Jim Ferenczi 7d0181b5d4 Rename bin/plugin in bin/elasticsearch-plugin 2016-02-05 10:09:14 +01:00
Ryan Ernst 3787f437ec Merge branch 'master' into remove_multicast 2016-02-01 07:25:45 -08:00
Ryan Ernst b8f08c35ec Plugin: Remove multicast plugin
closes #16310.
2016-01-29 18:41:31 -08:00
Ofir 22b91c2322 Update analysis.asciidoc 2016-01-28 13:02:30 +02:00
Ofir ac349373f0 Update analysis.asciidoc 2016-01-28 13:02:20 +02:00
Ofir 03e04daf29 Added the Network Addresses community plugin link
Updated analysis.asciidoc with a link to the Network Addresses Analysis Plugin.
2016-01-28 12:16:28 +02:00