Commit Graph

93 Commits

Author SHA1 Message Date
javanna 283d9c1523 [TEST] adjust processor docs. remove throws exception when the field is not there. 2015-12-09 18:24:26 +01:00
Tal Levy 45f48ac126 update all processors to only operate on one field at a time when possible 2015-12-07 08:30:00 -08:00
javanna 6c43137413 Merge branch 'master' into feature/ingest 2015-12-04 14:10:01 +01:00
Ryan 94027f1461 Made requested changes
Moved section to before testing section and made requested formatting changes.
2015-12-03 09:27:20 -08:00
Tal Levy 56da7b32ed add ability to define custom grok patterns within processor config 2015-12-03 08:24:07 -08:00
Ryan 5890c9a813 Update repository-s3.asciidoc
Documentation of AWS VPC public vs. private subnets and their affects on accessing S3.
2015-12-02 16:07:57 -08:00
Martijn van Groningen f427ad2094 docs: undo accidental rename added via: 5e07644788 2015-12-02 12:17:43 +01:00
Martijn van Groningen a9ecde041b Merge branch 'master' into feature/ingest 2015-12-02 11:21:15 +01:00
javanna 5e07644788 [DOCS] add missing comma 2015-12-01 20:07:17 +01:00
javanna 6c0510b01d Make rename processor less error prone
Rename processor now checks whether the field to rename exists and throws exception if it doesn't. It also checks that the new field to rename to doesn't exist yet, and throws exception otherwise. Also we make sure that the rename operation is atomic, otherwise things may break between the remove and the set and we'd leave the document in an inconsistent state.

Note that the requirement for the new field name to not exist simplifies the usecase for e.g. { "rename" : { "list.1": "list.2"} } as such a rename wouldn't be accepted if list is actually a list given that either list.2 already exists or the index is out of bounds for the existing list. If one really wants to replace an existing field, that field needs to be removed first through remove processor and then rename can be used.
2015-12-01 19:58:24 +01:00
David Pilato bed9bf19c6 S3 repository: fix spelling error
Reported at https://github.com/elastic/elasticsearch-cloud-aws/pull/221
2015-11-30 16:01:55 +01:00
Martijn van Groningen fdf4543b8e Renamed `add` processor to `set` processor.
This name makes more sense, because if a field already exists it overwrites it.
2015-11-30 15:03:20 +01:00
Martijn van Groningen 467a47670c Merge remote-tracking branch 'es/master' into feature/ingest 2015-11-30 10:24:27 +01:00
Clinton Gormley 2ab14cb21c Merge pull request #14900 from shikhar/patch-1
link to es-restlog plugin
2015-11-28 15:09:22 +01:00
Martijn van Groningen 9d1fa0d6da ingest: Add `meta` processor that allows to modify the metadata attributes of document being processed 2015-11-26 15:46:32 +01:00
Martijn van Groningen a84d35ab3f Merge remote-tracking branch 'es/master' into feature/ingest 2015-11-25 18:45:05 +01:00
Jimmi Dyson c4ee350c5e Add Kubernetes discovery community plugin 2015-11-25 12:54:29 +00:00
javanna 8f1f5d4da0 Split mutate processor into one processor per function 2015-11-24 14:31:53 +01:00
javanna eeb51ce8d0 Merge branch 'master' into feature/ingest 2015-11-24 10:23:53 +01:00
David Pilato 28109a18a2 Fix example for s3 repository bucket name
Closes #13588.
2015-11-23 13:14:02 +01:00
David Pilato 5b0e2823b1 Merge branch 'docs/mapper-attachments' 2015-11-23 12:14:31 +01:00
javanna 36655b688c Merge branch 'master' into feature/ingest 2015-11-23 10:05:17 +01:00
Shikhar Bhushan c57e006cd5 link to es-restlog plugin 2015-11-20 12:21:50 -05:00
David Pilato 33b0e662cc Merge branch 'pr/phonetic_daitch_mokotoff' 2015-11-19 14:48:11 +01:00
David Pilato d48d8ef863 Merge branch 's3-storage-class-support' of https://github.com/socialrank/elasticsearch into socialrank-s3-storage-class-support
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.
2015-11-19 13:10:08 +01:00
craigwi 79a4d9ce36 Add support for secondary azure storage account
Follow up for #13228.

This commit adds support for a secondary storage account:

```yml
cloud:
    azure:
        storage:
            my_account1:
                account: your_azure_storage_account1
                key: your_azure_storage_key1
                default: true
            my_account2:
                account: your_azure_storage_account2
                key: your_azure_storage_key2
```

When creating a repository, you can choose which azure account you want to use for it:

```sh
curl -XPUT localhost:9200/_snapshot/my_backup1?pretty -d '{
  "type": "azure"
}'

curl -XPUT localhost:9200/_snapshot/my_backup2?pretty -d '{
  "type": "azure",
  "settings": {
    "account" : "my_account2",
    "location_mode": "secondary_only"
  }
}'
```

`location_mode` supports `primary_only` or `secondary_only`. Defaults to `primary_only`. Note that if you set it
to `secondary_only`, it will force `read_only` to true.
2015-11-18 16:31:48 +01:00
David Pilato 52bf365013 Add support for `daitch_mokotoff`
[Daitch Mokotoff](https://en.wikipedia.org/wiki/Daitch%E2%80%93Mokotoff_Soundex) support has been added in Lucene 5.
So we can now support it as well.
2015-11-18 15:41:39 +01:00
javanna ef4d2a9426 Merge branch 'master' into feature/ingest 2015-11-17 17:42:36 +01:00
Krishna Raj f5ca9d0e10 Updated the correct doc for Kafka Consumer
Updated the correct doc for Kafka Consumer with link to latest repository.

Closes #14634
2015-11-17 14:11:03 +01:00
Martijn van Groningen da87cbf2b4 Merge remote-tracking branch 'es/master' into feature/ingest 2015-11-11 14:14:44 +07:00
Martijn van Groningen 4da05168f4 geoip: renamed `ip_field` option to `source_field`, because it can hold a ip or hostname.
geoip: add a `fields` option to control what fields are added by geoip processor
geoip: instead of adding all fields, only `country_code`, `city_name`, `location`, `continent_name` and `region_name` fields are added.
2015-11-10 11:50:12 +07:00
Tal Levy 8fc5a3d032 introduce mutate processor.
fix forbiddenapis

update

clean up and add rest test

update mutate factory to use configuration utilities

compile gsub pattern

cleanup, update parseBooleans, null tests
2015-11-09 18:49:55 -08:00
David Pilato e993c6a862 Migrate mapper attachements plugin to asciidoc
Followup for #14605
2015-11-09 15:35:06 +01:00
Clinton Gormley fe473ed24e Documented passing a local truststore to bin/plugin 2015-11-09 14:33:23 +01:00
javanna f657a7dbf3 Merge branch 'feature/ingest' into ingest/date 2015-11-09 13:42:07 +01:00
javanna 0c1fa74c3a Merge branch 'master' into feature/ingest 2015-11-09 12:36:51 +01:00
Clinton Gormley 4684be3680 Merge pull request #14391 from jurgc11/patch-2
Add community plugin
2015-11-08 22:12:26 +01:00
javanna a84ce07a90 added docs 2015-11-06 17:48:44 +01:00
Andrzej Wisłowski 7d010195f4 Update plugin-script.asciidoc
It is better not to show lang-groovy plugin in the example, as it is not used since elasticsearch 1.4.0
2015-11-05 10:27:34 -07:00
Tanguy Leroux d819930261 [Doc] Fix correct number of slashes when installing a plugin with zip file 2015-11-03 10:32:44 +01:00
Martijn van Groningen 03387266ca ingest: Added new `geoip` processor, that adds geographical information to documents based on an ip address.
The information is fetched from the Maxmind geolite2 database, that is embedded in the ingest plugin.
2015-11-03 12:06:29 +07:00
Tal Levy 464b46437f Introduce the GrokProcessor
Also moved all processor classes into a subdirectory and introduced a
ConfigException class to be a catch-all for things that can go wrong
when constructing new processors with configurations that possibly throw
exceptions. The GrokProcessor loads patterns from the resources
directory.

fix resource path issue, and add rest-api-spec test for grok

fix rest-spec tests

changes: license, remove configexception, throw IOException

add more tests and fix iso8601-hour pattern

move grok patterns from resources to config

fix tests with pom changes, updated IngestClientIT with grok processor

update gradle build script for grok deps and test configuration

move config files to src/main/packaging

move Env out of Processor, fix test for src/main/packaging change

add docs

clean up test resources task

update Grok to be immutable

- Updated the Grok class to be immutable. This means that all the
  pattern bank loading is handled by an external utility class called
  PatternUtils.
- fixed tabs in the nagios patterns file's comments
2015-11-02 20:27:49 -08:00
Tal Levy 4f5f2ff5e9 Merge remote-tracking branch 'upstream/master' into feature/ingest
Conflicts:
	plugins/pom.xml
	qa/smoke-test-plugins/pom.xml
2015-11-02 10:03:10 -08:00
David Pilato 77521560ed Merge branch 'ec2-improve-disco-nodes' of https://github.com/chaudum/elasticsearch into chaudum-ec2-improve-disco-nodes 2015-11-02 14:52:08 +01:00
David Pilato 4bbea60c40 Add FSCrawler 2015-10-30 17:36:25 +01:00
jurgc11 9550392e90 Add community plugin 2015-10-30 16:04:34 +00:00
Tal Levy 61d5ccbe7d Merge remote-tracking branch 'upstream/master' into feature/ingest 2015-10-29 16:44:37 -07:00
xuzha b3090d654a Add AWS Canned ACL support
User could set cannedACL, S3 repository would use this cannedACL to
create s3 object and bucket.
2015-10-29 11:48:02 -07:00
Christian Haudum 3334160003 improved building of disco nodes
* improved retry policy of ec2 client
* cache results for 10s
2015-10-29 18:07:52 +01:00
Martijn van Groningen 48e0a6745e Merge remote-tracking branch 'es/master' into feature/ingest 2015-10-27 16:16:45 +07:00