Commit Graph

482 Commits

Author SHA1 Message Date
Martijn van Groningen fdf4543b8e Renamed `add` processor to `set` processor.
This name makes more sense, because if a field already exists it overwrites it.
2015-11-30 15:03:20 +01:00
Martijn van Groningen 467a47670c Merge remote-tracking branch 'es/master' into feature/ingest 2015-11-30 10:24:27 +01:00
Clinton Gormley 2ab14cb21c Merge pull request from shikhar/patch-1
link to es-restlog plugin
2015-11-28 15:09:22 +01:00
Martijn van Groningen 9d1fa0d6da ingest: Add `meta` processor that allows to modify the metadata attributes of document being processed 2015-11-26 15:46:32 +01:00
Martijn van Groningen a84d35ab3f Merge remote-tracking branch 'es/master' into feature/ingest 2015-11-25 18:45:05 +01:00
Jimmi Dyson c4ee350c5e Add Kubernetes discovery community plugin 2015-11-25 12:54:29 +00:00
javanna 8f1f5d4da0 Split mutate processor into one processor per function 2015-11-24 14:31:53 +01:00
javanna eeb51ce8d0 Merge branch 'master' into feature/ingest 2015-11-24 10:23:53 +01:00
David Pilato 28109a18a2 Fix example for s3 repository bucket name
Closes .
2015-11-23 13:14:02 +01:00
David Pilato 5b0e2823b1 Merge branch 'docs/mapper-attachments' 2015-11-23 12:14:31 +01:00
javanna 36655b688c Merge branch 'master' into feature/ingest 2015-11-23 10:05:17 +01:00
Shikhar Bhushan c57e006cd5 link to es-restlog plugin 2015-11-20 12:21:50 -05:00
David Pilato 33b0e662cc Merge branch 'pr/phonetic_daitch_mokotoff' 2015-11-19 14:48:11 +01:00
David Pilato d48d8ef863 Merge branch 's3-storage-class-support' of https://github.com/socialrank/elasticsearch into socialrank-s3-storage-class-support
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.
2015-11-19 13:10:08 +01:00
craigwi 79a4d9ce36 Add support for secondary azure storage account
Follow up for .

This commit adds support for a secondary storage account:

```yml
cloud:
    azure:
        storage:
            my_account1:
                account: your_azure_storage_account1
                key: your_azure_storage_key1
                default: true
            my_account2:
                account: your_azure_storage_account2
                key: your_azure_storage_key2
```

When creating a repository, you can choose which azure account you want to use for it:

```sh
curl -XPUT localhost:9200/_snapshot/my_backup1?pretty -d '{
  "type": "azure"
}'

curl -XPUT localhost:9200/_snapshot/my_backup2?pretty -d '{
  "type": "azure",
  "settings": {
    "account" : "my_account2",
    "location_mode": "secondary_only"
  }
}'
```

`location_mode` supports `primary_only` or `secondary_only`. Defaults to `primary_only`. Note that if you set it
to `secondary_only`, it will force `read_only` to true.
2015-11-18 16:31:48 +01:00
David Pilato 52bf365013 Add support for `daitch_mokotoff`
[Daitch Mokotoff](https://en.wikipedia.org/wiki/Daitch%E2%80%93Mokotoff_Soundex) support has been added in Lucene 5.
So we can now support it as well.
2015-11-18 15:41:39 +01:00
javanna ef4d2a9426 Merge branch 'master' into feature/ingest 2015-11-17 17:42:36 +01:00
Krishna Raj f5ca9d0e10 Updated the correct doc for Kafka Consumer
Updated the correct doc for Kafka Consumer with link to latest repository.

Closes 
2015-11-17 14:11:03 +01:00
Martijn van Groningen da87cbf2b4 Merge remote-tracking branch 'es/master' into feature/ingest 2015-11-11 14:14:44 +07:00
Martijn van Groningen 4da05168f4 geoip: renamed `ip_field` option to `source_field`, because it can hold a ip or hostname.
geoip: add a `fields` option to control what fields are added by geoip processor
geoip: instead of adding all fields, only `country_code`, `city_name`, `location`, `continent_name` and `region_name` fields are added.
2015-11-10 11:50:12 +07:00
Tal Levy 8fc5a3d032 introduce mutate processor.
fix forbiddenapis

update

clean up and add rest test

update mutate factory to use configuration utilities

compile gsub pattern

cleanup, update parseBooleans, null tests
2015-11-09 18:49:55 -08:00
David Pilato e993c6a862 Migrate mapper attachements plugin to asciidoc
Followup for 
2015-11-09 15:35:06 +01:00
Clinton Gormley fe473ed24e Documented passing a local truststore to bin/plugin 2015-11-09 14:33:23 +01:00
javanna f657a7dbf3 Merge branch 'feature/ingest' into ingest/date 2015-11-09 13:42:07 +01:00
javanna 0c1fa74c3a Merge branch 'master' into feature/ingest 2015-11-09 12:36:51 +01:00
Clinton Gormley 4684be3680 Merge pull request from jurgc11/patch-2
Add community plugin
2015-11-08 22:12:26 +01:00
javanna a84ce07a90 added docs 2015-11-06 17:48:44 +01:00
Andrzej Wisłowski 7d010195f4 Update plugin-script.asciidoc
It is better not to show lang-groovy plugin in the example, as it is not used since elasticsearch 1.4.0
2015-11-05 10:27:34 -07:00
Tanguy Leroux d819930261 [Doc] Fix correct number of slashes when installing a plugin with zip file 2015-11-03 10:32:44 +01:00
Martijn van Groningen 03387266ca ingest: Added new `geoip` processor, that adds geographical information to documents based on an ip address.
The information is fetched from the Maxmind geolite2 database, that is embedded in the ingest plugin.
2015-11-03 12:06:29 +07:00
Tal Levy 464b46437f Introduce the GrokProcessor
Also moved all processor classes into a subdirectory and introduced a
ConfigException class to be a catch-all for things that can go wrong
when constructing new processors with configurations that possibly throw
exceptions. The GrokProcessor loads patterns from the resources
directory.

fix resource path issue, and add rest-api-spec test for grok

fix rest-spec tests

changes: license, remove configexception, throw IOException

add more tests and fix iso8601-hour pattern

move grok patterns from resources to config

fix tests with pom changes, updated IngestClientIT with grok processor

update gradle build script for grok deps and test configuration

move config files to src/main/packaging

move Env out of Processor, fix test for src/main/packaging change

add docs

clean up test resources task

update Grok to be immutable

- Updated the Grok class to be immutable. This means that all the
  pattern bank loading is handled by an external utility class called
  PatternUtils.
- fixed tabs in the nagios patterns file's comments
2015-11-02 20:27:49 -08:00
Tal Levy 4f5f2ff5e9 Merge remote-tracking branch 'upstream/master' into feature/ingest
Conflicts:
	plugins/pom.xml
	qa/smoke-test-plugins/pom.xml
2015-11-02 10:03:10 -08:00
David Pilato 77521560ed Merge branch 'ec2-improve-disco-nodes' of https://github.com/chaudum/elasticsearch into chaudum-ec2-improve-disco-nodes 2015-11-02 14:52:08 +01:00
David Pilato 4bbea60c40 Add FSCrawler 2015-10-30 17:36:25 +01:00
jurgc11 9550392e90 Add community plugin 2015-10-30 16:04:34 +00:00
Tal Levy 61d5ccbe7d Merge remote-tracking branch 'upstream/master' into feature/ingest 2015-10-29 16:44:37 -07:00
xuzha b3090d654a Add AWS Canned ACL support
User could set cannedACL, S3 repository would use this cannedACL to
create s3 object and bucket.
2015-10-29 11:48:02 -07:00
Christian Haudum 3334160003 improved building of disco nodes
* improved retry policy of ec2 client
* cache results for 10s
2015-10-29 18:07:52 +01:00
Martijn van Groningen 48e0a6745e Merge remote-tracking branch 'es/master' into feature/ingest 2015-10-27 16:16:45 +07:00
Hendrik Saly 58f2b6ca47 Add Kerberos/SPNEGO Shield custom realm
Closes 
2015-10-26 13:53:04 +01:00
Clinton Gormley d2bb289c64 Merge pull request from salyh/patch-1
Added IMAP/POP3 Mailimporter to documentation
2015-10-26 13:47:44 +01:00
xuzha 6a6e168b8b Adding backoff from retries on GCE errors
In case of any error while trying to get GCE instances list from GCE
API, elasticsearch will slow down its API calls.
2015-10-19 10:28:08 -07:00
Robert Muir 5d001d1578 Decentralize plugin security
* Add ability for plugins to declare additional permissions with a custom plugin-security.policy file and corresponding AccessController logic. See the plugin author's guide for more information.
* Add warning messages to users for extra plugin permissions in bin/plugin.
* When bin/plugin is run interactively (stdin is a controlling terminal and -b/--batch not supplied), require user confirmation.
* Improve unit test and IDE support for plugins with additional permissions by exposing plugin's metadata as a maven test resource.

Closes 

Squashed commit of the following:

commit cf8ace65a7397aaccd356bf55f95d6fbb8bb571c
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 13:36:05 2015 -0400

    fix new unit test from master merge

commit 9be3c5aa38f2d9ae50f3d54924a30ad9cddeeb65
Merge: 2f168b8 7368231
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 12:58:31 2015 -0400

    Merge branch 'master' into off_my_back

commit 2f168b8038e32672f01ad0279fb5db77ba902ae8
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 12:56:04 2015 -0400

    improve plugin author documentation

commit 6e6c2bfda68a418d92733ac22a58eec35508b2d0
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 12:52:14 2015 -0400

    move security confirmation after 'plugin already installed' check, to prevent user from answering unnecessary questions.

commit 08233a2972554afef2a6a7521990283102e20d92
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 05:36:42 2015 -0400

    Add documentation and pluginmanager support

commit 05dad86c51488ba43ccbd749f0164f3fbd3aee62
Author: Robert Muir <rmuir@apache.org>
Date:   Wed Oct 14 02:22:24 2015 -0400

    Decentralize plugin permissions (modulo docs and pluginmanager work)
2015-10-14 14:46:45 -04:00
Tim Schlechter 7e2f558887 Added link to Swagger for Elasticsearch 2015-10-13 21:29:53 +02:00
Martijn van Groningen 5a3c75ebac added put, get and delete pipeline APIs. 2015-10-09 18:28:55 +02:00
David Pilato becaff30a8 Replace cloud-gce with discovery-gce
Related to 
2015-10-08 08:53:54 +02:00
David Pilato 4038aed641 Fix internal link name
Related to 
2015-10-08 07:04:11 +02:00
David Pilato c73ab50df1 Rename cloud-gce plugin to discovery-gce plugin
Follow up azure and aws splits, we need to be consistent and rename `cloud-gce` to `discovery-gce`.
2015-10-08 06:53:37 +02:00
David Pilato 289cd5dcf4 [discovery-gce] add _gce_ network host setting
When running in GCE platform, an instance has access to:

http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/ip

Which gives back the private IP address, for example `10.240.0.2`.

http://metadata.google.internal/computeMetadata/v1/instance/network-interfaces/0/externalIp

Gives back the public Ip address, for example `130.211.108.21`.

As we have for `ec2`, we can support new network host settings:

* `_gce:privateIp:X_`: The private IP address of the machine for a given network interface.
* `_gce:hostname_`: The hostname of the machine.
* `_gce_`: Same as `_gce:privateIp:0_` (recommended).

Closes .
Closes .

BTW resolveIfPossible now throws IOException so code is also updated for ec2 discovery and
some basic tests have been added.
2015-10-07 22:04:34 +02:00
David Pilato c0363dd56b Update GCE discovery documentation
(cherry picked from commit d4cb81c)
(cherry picked from commit 9af04eb)
(cherry picked from commit b383344)
2015-10-07 21:43:12 +02:00
Kevin Kirsche 072f6ae138 Docs: Note that plugin removal usually requires node restart
Closes 
Closes 
2015-10-07 18:10:29 +02:00
David Pilato f0252f34e9 Enhance plugin-descriptor.properties guide
* extract doc from the descriptor
* make obvious that plugin authors will have to release a new version for each elasticsearch version
2015-10-06 22:12:45 +02:00
David Pilato 7b431ecc16 Fix cross links
Related to 
(cherry picked from commit ff4cdab)
(cherry picked from commit 8b46a4b)
2015-10-06 12:18:32 +02:00
David Pilato 685c1f4a54 [discovery-ec2] network.host must be set
With 2.0, we now bind to `localhost` by default instead of binding to the network card and use its IP address.

 When the discovery plugin gets from AWS API the list of nodes that should form the cluster, this list is pinged then. But as each node is bound to `localhost`, ping does not get an answer and the node elects itself as the master node.

`network.host` must be set.

 Closes .
2015-10-06 11:18:47 +02:00
Nik Everett 4d47015a0c Merge pull request from nik9000/run_plugin_as_elasticsearch_in_tar
Run bin/plugin as elasticsearch in tar distro
2015-09-22 11:19:19 -04:00
Nik Everett b521c606ff [test] Run bin/plugin as right user
Before this commit he tests always run bin/plugin as root which is somewhat
unrealistic and causes trouble (log files owned by root instead of
elasticsearch). After this commit `bin/plugin` runs as root when elasticsearch
is installed via the repository and as elasticsearch otherwise which is much
more realistic.

This also adds extra timeout to starting elasticsearch which is required
when all the plugins are installed. And it fixes up a problem with logging
elasticsearch's log if elasticsearch doesn't start which came up multiple
time while debugging this problem.

Also adds docs recommending running `bin/plugin` as the user that owns the
Elasticsearch files or root if installed with the packages.

Closes 
2015-09-22 11:17:57 -04:00
David Pilato 3049ef3e35 Merge pull request from mnylen/patch-1
Add docs note about cloud.aws.region ()
2015-09-22 15:15:54 +02:00
Clinton Gormley 67815eec47 Docs: Removed references to java 7 and debian wheezy
Related to 
2015-09-22 14:23:30 +02:00
Mikko Nylén a379400228 Add docs note about cloud.aws.region () 2015-09-22 14:53:54 +03:00
David Pilato f230eabc15 [cloud-azure] Split azure plugin in 3 plugins
Until now we had a cloud-azure plugin which is providing 3 distinct features:

* discovery on Azure
* snapshot/restore on Aure
* SMB store

This commit splits the plugin by feature so people can use either one or the other or both features.

Doc is updated accordingly.
2015-09-21 17:55:23 +02:00
David Pilato 71ec1f09e3 Add documentation for setting network.host with azure discovery
With 2.0, we now bind to `localhost` by default instead of binding to the network card and use its IP address.

When the discovery plugin gets from Azure API the list of nodes that should form the cluster, this list is pinged then. But as each node is bound to `localhost`, ping does not get an answer and the node elects itself as the master node.

Closes 
2015-09-17 16:29:02 +02:00
Shaunak Kashyap 516ba6c8f6 Using gcloud CLI as gcutil CLI has been deprecated 2015-09-16 06:43:47 -07:00
Paul Echeverri f8c86c7252 Adds links to Logstash plugins under the Integrations page. 2015-09-09 17:57:23 -07:00
xuzha e69551d8df This commit remove the block for user to set invalid Signer Type.
Instead of throw an exception, this change log a warning message
and accept any signer type.
2015-09-05 22:57:05 -07:00
David Pilato 30aa231f8e [plugin] split cloud-aws in repository-s3 and discovery-ec2
Until now we had a cloud-aws plugin which is providing 2 disctinct features:

* discovery on EC2
* snapshot/restore on S3

This commit splits the plugin by feature so people can use either one or the other or both features.

Doc is updated accordingly.
2015-09-03 11:12:20 +02:00
Ryan Ernst 6295f8e795 Merge branch 'master' into tell_me_your_plugins 2015-08-30 14:20:54 -07:00
Igor Motov 2b87d7d919 Add `readonly` option for repositories
Closes 
Closes 
2015-08-27 18:21:29 -04:00
Ryan Ernst d06be0707c Added link to test framework for plugin authors 2015-08-24 12:05:00 -07:00
Ryan Ernst 89e1a0fb7d Plugins: Removed plugin.types
The setting `plugin.types` is currently used to load plugins from the
classpath. This is necessary in tests, as well as the transport client.

This change removes the setting, and replaces it with the ability to
directly add plugins when building a transport client, as well as
infrastructure in the integration tests to specify which plugin classes
should be loaded on each node.
2015-08-22 01:37:43 -07:00
Adrien Grand c54a4c4f89 Docs: Fix section id for the new discovery-multicast plugin. 2015-08-21 09:36:55 +02:00
Ryan Ernst 164efaecbe Networking: Move multicast discovery to a plugin
Multicast has known issues (see  and ). This change moves
multicast into a plugin, and deprecates it in the docs.  It also allows
for plugging in multiple zen ping implementations.

closes 
2015-08-20 16:43:25 -07:00
Adrien Grand 6fa258b8fa Deprecate the `scan` search type.
This commit deprecates the `scan` search type in favour of regular scroll
requests sorted by `_doc`.

Related to 
2015-08-20 12:47:23 +02:00
Christoph Büscher 8454d49552 Plugins: Add 'name' property to plugin descriptor file to determine plugin name
At the moment, when installing from an url, a user provides the plugin name on
the command line like:

* bin/plugin install [plugin-name] --url [url]

This can lead to problems when picking an already existing name from another
plugin, and can potentially overwrite plugins already installed with that name.

This, this PR introduces a mandatory `name` property to the plugin descriptor
file which replaces the name formerly provided by the user.

With the addition of the `name` property to the plugin descriptor file, the user
does not need to specify the plugin name any longer when installing from a file
or url. Because of this, all arguments to `plugin install` command are now
either treated as a symbolic name, a URL or a file without the need to specify
this with an explicit option.

The new syntax for `plugin install` is now:

bin/plugin install [name or url]

* downloads official plugin
bin/plugin install analysis-kuromoji

* downloads github plugin
bin/plugin install lmenezes/elasticsearch-kopf

* install from URL or file
bin/plugin install http://link.to/foo.zip
bin/plugin install file:/path/to/foo.zip

If the argument does not parse to a valid URL, it is assumed to be a name and the
download location is resolved like before. Regardless of the source location of
the plugin, it is extracted to a temporary directory and the `name` property from
the descriptor file is used to determine the final install location.

Relates to 
2015-08-19 12:13:01 +02:00
David Pilato 05678bc10a [doc] Fix cloud-azure install / remove instructions 2015-08-18 13:18:39 +02:00
David Pilato 717a6dd092 [doc] Backport change in cloud-aws doc
Related to https://github.com/elastic/elasticsearch/pull/12761
2015-08-18 13:12:58 +02:00
David Pilato 44e6d1aac6 [doc] Move mapper attachment plugin to mapper page 2015-08-18 13:02:18 +02:00
David Pilato 975eb60a12 [doc] we don't use `check_lucene` anymore in plugins 2015-08-18 13:02:18 +02:00
Adrien Grand a91b3fcbb9 Move the `murmur3` field to a plugin and fix defaults.
This move the `murmur3` field to the `mapper-murmur3` plugin and fixes its
defaults so that values will not be indexed by default, as the only purpose
of this field is to speed up `cardinality` aggregations on high-cardinality
string fields, which only requires doc values.

I also removed the `rehash` option from the `cardinality` aggregation as it
doesn't bring much value (rehashing is cheap) and allowed to remove the
coupling between the `cardinality` aggregation and the `murmur3` field.

Close 
2015-08-18 11:41:52 +02:00
Clinton Gormley a0f92e101b Docs: Fixed bad link to dynamic scripting 2015-08-15 18:20:14 +02:00
Clinton Gormley e143c6e460 Docs: Prepare plugin and integration docs for 2.0
* Centralised plugin docs in docs/plugins/
* Moved integrations into same docs
* Moved community clients into the clients section of the docs
* Removed docs/community

Closes 
Closes 
Closes 
Closes 
Closes 
Closes 
Closes 
Closes 
Closes 
2015-08-15 18:02:43 +02:00
Alexander Reelsen d3e454780f PluginManager: Fix bin/plugin calls in scripts/bats test
The release and smoke test python scripts used to install
plugins in the old fashion.

Also the BATS testing suite installed/removed plugins in that
way. Here the marvel tests have been removed, as marvel currently
does not work with the master branch.

In addition documentation has been updated as well, where it was
still missing.
2015-07-21 16:17:59 +02:00
Tanguy Leroux 26e782771c Add documentation for delete by query plugin (see )
This page is placed in a /plugins directory until we figure where to place all plugins documentation.
2015-06-23 13:25:40 +02:00