1290 Commits

Author SHA1 Message Date
Robert Muir
a164586796 Merge branch 'mapper_migrate' into migrate-mapper-attachments 2015-11-08 02:19:50 -05:00
Robert Muir
550f462c47 migrate mapper attachments 2015-11-08 02:09:36 -05:00
Ryan Ernst
8d78f970b0 Build: Enforce name is set for all plugins
The plugin name currently defaults to the gradle project name. But the
gradle project name for standalone repo (like an external plugin would
be) defaults to the directory name of the repo. This is trappy, since it
depends on how the repo was checked out.

This change enforces the plugin name is always set.

closes #14603
2015-11-07 16:57:11 -08:00
Robert Muir
e06cae84f3 only allow code to bind to the user's configured port numbers/ranges
Random code shouldn't be listening on sockets elsewhere.

Today its the wild west, but we only need to grant access to what the user configured.

This means e.g. multicast plugin has to declare its intentions in its security.policy

Closes #14549
2015-11-07 12:22:12 -05:00
javanna
1dfe6f6dcf make sure headers etc. are passed over to inner index and delete requests in put/delete pipeline 2015-11-06 19:21:30 +01:00
javanna
d318990339 added REST test 2015-11-06 16:10:22 +01:00
javanna
682876f7d7 added factory tests 2015-11-06 15:44:20 +01:00
javanna
dbf5c96876 Merge branch 'feature/ingest' into ingest/date 2015-11-06 13:04:51 +01:00
javanna
588de6ccff Add generic type to Processor.Factory and rename Geo*Tests to GeoIp*Tests 2015-11-06 12:53:20 +01:00
javanna
1d2e244bac Merge branch 'feature/ingest' into ingest/date 2015-11-06 12:08:14 +01:00
javanna
7798c6cd49 unified date parser tests in a single test class, added more tests for date processor 2015-11-06 12:05:30 +01:00
Christoph Büscher
40f82de4a1 Tests: make sure to load timezones in parent classloader
When GroovySecurityTests are run before any other test using
time zones, joda ZoneInfoProvider fails to load the time zones
correctly and never tries again later. This makes sure we load it
correctly on startup.

Relates to #14524
2015-11-06 11:11:28 +01:00
Martijn van Groningen
e7f0f0ed4e Enforce strict pipeline configuration
Closes #14552
2015-11-05 23:38:29 +07:00
Simon Willnauer
94bed42213 Simplify plugin API and fix IndexService internal allocation 2015-11-05 13:16:35 +01:00
Simon Willnauer
a311491c8e decouple IndexingMemoryController from IndexShard 2015-11-05 11:18:11 +01:00
Simon Willnauer
487af301ea Remove guice from the index level
This commit removes guice from the index level and adds a simple extension point
to add class instances with an index-lifecycle scope.
2015-11-05 11:18:11 +01:00
Martijn van Groningen
92452ff99a rename the ingest parameter to pipeline_id param, because it is more descriptive what the parameter should hold. 2015-11-05 14:35:49 +07:00
javanna
b45815da36 adapt to changes upstream 2015-11-04 14:56:03 +01:00
javanna
363454ce4a Merge branch 'feature/ingest' into ingest/date 2015-11-04 14:48:24 +01:00
Robert Muir
79467a138a Add property permissions so groovy scripts can serialize json
Allowing read to these properties is not really dangerous, even if
the code surrounding them is.

Closes #14488
2015-11-03 15:21:55 -05:00
Robert Muir
7e6008f0b9 refactor GroovySecurityTests into a unit test.
This was basically a resurrected form of the tests for the old sandbox.
We use it to check that groovy scripts some degree of additional containment.

The other scripting plugins (javascript, python) already have this as a unit test,
its much easier to debug any problems that way.

closes #14484
2015-11-03 13:39:17 -05:00
Robert Muir
b6a2318563 upgrade rhino for plugins/lang-javascript
the current jar is over 3 years old, we should upgrade it for bugfixes.
the current integration could be more secure: set a global policy and enforce additional (compile-time) checks

closes #14466
2015-11-03 09:53:07 -05:00
Martijn van Groningen
1a4b5bba2b Simplify processor creation from map of maps by folding the build and builder factory in one interface called Factory.
In tests processors can be created from the their constructors instead of builders.
In the IngestModule, register instances instead of class instances.
2015-11-03 13:25:58 +07:00
Tal Levy
d935a3ab81 Make Grok-specific classes package-protected. 2015-11-02 21:30:09 -08:00
Martijn van Groningen
95e6b99d2b renamed test file 2015-11-03 12:17:10 +07:00
Martijn van Groningen
03387266ca ingest: Added new geoip processor, that adds geographical information to documents based on an ip address.
The information is fetched from the Maxmind geolite2 database, that is embedded in the ingest plugin.
2015-11-03 12:06:29 +07:00
Tal Levy
464b46437f Introduce the GrokProcessor
Also moved all processor classes into a subdirectory and introduced a
ConfigException class to be a catch-all for things that can go wrong
when constructing new processors with configurations that possibly throw
exceptions. The GrokProcessor loads patterns from the resources
directory.

fix resource path issue, and add rest-api-spec test for grok

fix rest-spec tests

changes: license, remove configexception, throw IOException

add more tests and fix iso8601-hour pattern

move grok patterns from resources to config

fix tests with pom changes, updated IngestClientIT with grok processor

update gradle build script for grok deps and test configuration

move config files to src/main/packaging

move Env out of Processor, fix test for src/main/packaging change

add docs

clean up test resources task

update Grok to be immutable

- Updated the Grok class to be immutable. This means that all the
  pattern bank loading is handled by an external utility class called
  PatternUtils.
- fixed tabs in the nagios patterns file's comments
2015-11-02 20:27:49 -08:00
Tal Levy
a8d76f0117 remove pom.xml from ingest plugin 2015-11-02 10:09:36 -08:00
Tal Levy
4f5f2ff5e9 Merge remote-tracking branch 'upstream/master' into feature/ingest
Conflicts:
	plugins/pom.xml
	qa/smoke-test-plugins/pom.xml
2015-11-02 10:03:10 -08:00
javanna
14e09ad20c Add the ingest param to the context only if present 2015-11-02 17:39:30 +01:00
Martijn van Groningen
376c739fe6 added TODO 2015-11-02 23:37:09 +07:00
javanna
7b3c034caf [DOCS] fixed javadocs minor typos 2015-11-02 15:54:46 +01:00
David Pilato
7bbd2a1513 Fix test for ec2 discovery
See #14155
2015-11-02 15:33:36 +01:00
javanna
8425d26b70 use existing constant to check whether client is a transport client 2015-11-02 15:27:56 +01:00
David Pilato
77521560ed Merge branch 'ec2-improve-disco-nodes' of https://github.com/chaudum/elasticsearch into chaudum-ec2-improve-disco-nodes 2015-11-02 14:52:08 +01:00
David Pilato
1b98e1b1ad Adding US-Gov-West for S3
Follow up for #14358
2015-11-02 14:38:03 +01:00
David Pilato
bcc372a29e Merge branch 'aws-gov-west' of https://github.com/amos6224/elasticsearch-1 into amos6224-aws-gov-west 2015-11-02 14:32:23 +01:00
Robert Muir
3961c9b58b fix eclipse (again) 2015-10-31 02:39:35 -04:00
Ryan Ernst
b6dee6bd43 Merge pull request #14375 from rjernst/sweep_up_maven
Remove maven pom files and supporting ant files
2015-10-30 18:59:11 -07:00
Areek Zillur
13b60e1b92 update to lucene-5.4.x-snapshot-1711508 2015-10-30 15:42:02 -04:00
Nik Everett
28633fae21 [mapping] Remove transform
Removes the mapping transform feature which when used made debugging very
difficult. Users should transform their documents on the way into
Elasticsearch rather than having Elasticsearch do it.

Closes #12674
2015-10-30 11:46:54 -04:00
Simon Willnauer
aa38d053d7 Simplify Analysis registration and configuration
This change moves all the analysis component registration to the node level
and removes the significant API overhead to register tokenfilter, tokenizer,
charfilter and analyzer. All registration is done without guice interaction such
that real factories via functional interfaces are passed instead of class objects
that are instantiated at runtime.

This change also hides the internal analyzer caching that was done previously in the
IndicesAnalysisService entirely and decouples all analysis registration and creation
from dependency injection.
2015-10-30 11:40:18 +01:00
Ryan Ernst
542522531a Build: Remove maven pom files and supporting ant files
This change removes the leftover pom files. A couple files were left for
reference, namely in qa tests that have not yet been migrated (vagrant
and multinode). The deb and rpm assemblies also still exist for
reference when finishing their setup in gradle.

See #13930
2015-10-29 23:53:49 -07:00
Tal Levy
61d5ccbe7d Merge remote-tracking branch 'upstream/master' into feature/ingest 2015-10-29 16:44:37 -07:00
Tal Levy
c6f4cd68da add gradle build script to plugin 2015-10-29 13:40:23 -07:00
Tal Levy
5411bb1123 Merge remote-tracking branch 'upstream/master' into feature/ingest 2015-10-29 11:52:21 -07:00
xuzha
b3090d654a Add AWS Canned ACL support
User could set cannedACL, S3 repository would use this cannedACL to
create s3 object and bucket.
2015-10-29 11:48:02 -07:00
Ryan Ernst
c86100f636 Switch build system to Gradle
See #13930
2015-10-29 11:40:19 -07:00
Robert Muir
1194cd0bdc Add threadgroup isolation.
Closes #14353

Squashed commit of the following:

commit edae0729f71ea3d3f9fa9c0d27c9effc042eb5a9
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Oct 29 14:13:42 2015 -0400

    update sha1 and simplify test

commit 635c4f245d66ad353a16267c810e02b725553fad
Author: Robert Muir <rmuir@apache.org>
Date:   Thu Oct 29 07:01:26 2015 -0400

    Add threadgroup isolation.

    Code with `modifyThread` and `modifyThreadGroup` may only modify
    its own threadgroup (or an ancestor of that). This enforces
    what is intended by the ThreadGroup class.

    This has two immediate implications:
    1. Code without these permissions (scripts) may not create or mess with threads
    2. ES application threads cannot mess with Java system threads

    ES puts all application threads in one single group today, but in the future
    this can be organized better, and we will have more isolation in the system.
2015-10-29 14:14:27 -04:00
Jeff Destine
a8268ba37d Adding US-Gov-West 2015-10-29 13:09:11 -04:00