Commit Graph

763 Commits

Author SHA1 Message Date
javanna 5bd4493ea2 use ConfigurationUtils to read string value from config 2015-11-12 15:38:36 +01:00
javanna c12c9e6e29 add equals and hashcode to GsubExpression 2015-11-11 11:10:26 +01:00
javanna 2b31f4fff7 Mutate processor improvements
Remove code duplications from ConfigurationUtils
Make sure that the mutate processor doesn't use Tuple as that would require to depend on core.
Also make sure that the MutateProcessor tests don't end up testing the factory as well.
Make processor getters package private as they are only needed in tests.
Add new tests to MutateProcessorFactoryTests
2015-11-11 11:10:26 +01:00
Martijn van Groningen 2bde384825 renamed yaml tests 2015-11-11 14:41:29 +07:00
Martijn van Groningen 347b8e600e specify all the dependencies of dependencies, because transitive dependencies have been disabled 2015-11-11 14:39:48 +07:00
Martijn van Groningen da87cbf2b4 Merge remote-tracking branch 'es/master' into feature/ingest 2015-11-11 14:14:44 +07:00
Robert Muir 914cee213e Upgrade to aws 1.10.33
Security issues have been fixed. This removes our hacks.
2015-11-10 22:51:33 -05:00
Ryan Ernst 2cc89e8587 Fix concrete references to shared deps 2015-11-10 15:13:54 -08:00
Ryan Ernst 4b5f87cb7d Build: Remove transitive dependencies
Transitive dependencies can be confusing and hard to deal with when
conflicts arise between them. This change removes transitive
dependencies from elasticsearch, and forces any dependency conflicts to
be resolved manually, instead of automatically by gradle.

closes #14627
2015-11-10 15:01:41 -08:00
Tanguy Leroux 4063354dbe Fix Delete-by-Query with Shield
closes #14527
2015-11-10 20:39:38 +01:00
Ryan Ernst 7bb4e10353 Merge pull request #14645 from rjernst/version_cleanup
Consolidate dependencies specified in multiple places
2015-11-10 07:49:10 -08:00
David Pilato a61fcfd8ca Remove AbstractLegacyBlobContainer
`AbstractLegacyBlobContainer` was kept for historical reasons (see #13434).
We can migrate Azure and S3 repositories to use the new methods added in #13434 so we can remove `AbstractLegacyBlobContainer` class.
2015-11-10 16:04:44 +01:00
Ryan Ernst 7a6155e12f Build: Consolidate dependencies specified in multiple places
Some dependencies must be specified in a couple places in the build.
e.g. randomized runner is specified both in buildSrc (for the gradle
wrapper plugin), as well as in the test-framework.

This change creates buildSrc/versions.properties which acts similar to
the set of shared version properties we used to have in the maven parent
pom.
2015-11-09 23:59:56 -08:00
Martijn van Groningen 4da05168f4 geoip: renamed `ip_field` option to `source_field`, because it can hold a ip or hostname.
geoip: add a `fields` option to control what fields are added by geoip processor
geoip: instead of adding all fields, only `country_code`, `city_name`, `location`, `continent_name` and `region_name` fields are added.
2015-11-10 11:50:12 +07:00
Tal Levy e9b72f5394 remove SimpleProcessor 2015-11-09 19:14:06 -08:00
Tal Levy 8fc5a3d032 introduce mutate processor.
fix forbiddenapis

update

clean up and add rest test

update mutate factory to use configuration utilities

compile gsub pattern

cleanup, update parseBooleans, null tests
2015-11-09 18:49:55 -08:00
Adrien Grand d0853e9253 Merge pull request #14619 from jpountz/upgrade/lucene-5.4.0-snapshot-1712973
Upgrade to lucene-5.4.0-snapshot-1712973.
2015-11-09 22:38:44 +01:00
Robert Muir ac612208b7 minor attachments cleanups: IDE test support and EPUB format
The documentation says we support EPUB, but the parser is not enabled.
This parser does not require any external dependencies, so I think its ok?

Separately, test-framework drags in an ancient commons-codec (via httpclient), which gradle
"upgrades", but IDEs can't handle this case and just hit jar hell. So just wire that to 1.9,
this allows running tests in the IDE for this plugin.
2015-11-09 11:30:58 -05:00
Adrien Grand d6d7af0a6c Upgrade to lucene-5.4.0-snapshot-1712973. 2015-11-09 15:53:27 +01:00
javanna f657a7dbf3 Merge branch 'feature/ingest' into ingest/date 2015-11-09 13:42:07 +01:00
javanna 0c1fa74c3a Merge branch 'master' into feature/ingest 2015-11-09 12:36:51 +01:00
Martijn van Groningen 302621f16b geoip: don't store geoinfo if nothing can be resolved 2015-11-09 18:24:05 +07:00
Robert Muir e88896e40e apply a few post-merge cleanups 2015-11-08 02:31:26 -05:00
Robert Muir a164586796 Merge branch 'mapper_migrate' into migrate-mapper-attachments 2015-11-08 02:19:50 -05:00
Robert Muir 550f462c47 migrate mapper attachments 2015-11-08 02:09:36 -05:00
Ryan Ernst 8d78f970b0 Build: Enforce name is set for all plugins
The plugin name currently defaults to the gradle project name. But the
gradle project name for standalone repo (like an external plugin would
be) defaults to the directory name of the repo. This is trappy, since it
depends on how the repo was checked out.

This change enforces the plugin name is always set.

closes #14603
2015-11-07 16:57:11 -08:00
Robert Muir e06cae84f3 only allow code to bind to the user's configured port numbers/ranges
Random code shouldn't be listening on sockets elsewhere.

Today its the wild west, but we only need to grant access to what the user configured.

This means e.g. multicast plugin has to declare its intentions in its security.policy

Closes #14549
2015-11-07 12:22:12 -05:00
javanna 1dfe6f6dcf make sure headers etc. are passed over to inner index and delete requests in put/delete pipeline 2015-11-06 19:21:30 +01:00
javanna d318990339 added REST test 2015-11-06 16:10:22 +01:00
javanna 682876f7d7 added factory tests 2015-11-06 15:44:20 +01:00
javanna dbf5c96876 Merge branch 'feature/ingest' into ingest/date 2015-11-06 13:04:51 +01:00
javanna 588de6ccff Add generic type to Processor.Factory and rename Geo*Tests to GeoIp*Tests 2015-11-06 12:53:20 +01:00
javanna 1d2e244bac Merge branch 'feature/ingest' into ingest/date 2015-11-06 12:08:14 +01:00
javanna 7798c6cd49 unified date parser tests in a single test class, added more tests for date processor 2015-11-06 12:05:30 +01:00
Christoph Büscher 40f82de4a1 Tests: make sure to load timezones in parent classloader
When GroovySecurityTests are run before any other test using
time zones, joda ZoneInfoProvider fails to load the time zones
correctly and never tries again later. This makes sure we load it
correctly on startup.

Relates to #14524
2015-11-06 11:11:28 +01:00
Martijn van Groningen e7f0f0ed4e Enforce strict pipeline configuration
Closes #14552
2015-11-05 23:38:29 +07:00
Simon Willnauer 94bed42213 Simplify plugin API and fix IndexService internal allocation 2015-11-05 13:16:35 +01:00
Simon Willnauer a311491c8e decouple IndexingMemoryController from IndexShard 2015-11-05 11:18:11 +01:00
Simon Willnauer 487af301ea Remove guice from the index level
This commit removes guice from the index level and adds a simple extension point
to add class instances with an index-lifecycle scope.
2015-11-05 11:18:11 +01:00
Martijn van Groningen 92452ff99a rename the `ingest` parameter to `pipeline_id` param, because it is more descriptive what the parameter should hold. 2015-11-05 14:35:49 +07:00
javanna b45815da36 adapt to changes upstream 2015-11-04 14:56:03 +01:00
javanna 363454ce4a Merge branch 'feature/ingest' into ingest/date 2015-11-04 14:48:24 +01:00
Robert Muir 79467a138a Add property permissions so groovy scripts can serialize json
Allowing read to these properties is not really dangerous, even if
the code surrounding them is.

Closes #14488
2015-11-03 15:21:55 -05:00
Robert Muir 7e6008f0b9 refactor GroovySecurityTests into a unit test.
This was basically a resurrected form of the tests for the old sandbox.
We use it to check that groovy scripts some degree of additional containment.

The other scripting plugins (javascript, python) already have this as a unit test,
its much easier to debug any problems that way.

closes #14484
2015-11-03 13:39:17 -05:00
Robert Muir b6a2318563 upgrade rhino for plugins/lang-javascript
the current jar is over 3 years old, we should upgrade it for bugfixes.
the current integration could be more secure: set a global policy and enforce additional (compile-time) checks

closes #14466
2015-11-03 09:53:07 -05:00
Martijn van Groningen 1a4b5bba2b Simplify processor creation from map of maps by folding the build and builder factory in one interface called Factory.
In tests processors can be created from the their constructors instead of builders.
In the IngestModule, register instances instead of class instances.
2015-11-03 13:25:58 +07:00
Tal Levy d935a3ab81 Make Grok-specific classes package-protected. 2015-11-02 21:30:09 -08:00
Martijn van Groningen 95e6b99d2b renamed test file 2015-11-03 12:17:10 +07:00
Martijn van Groningen 03387266ca ingest: Added new `geoip` processor, that adds geographical information to documents based on an ip address.
The information is fetched from the Maxmind geolite2 database, that is embedded in the ingest plugin.
2015-11-03 12:06:29 +07:00
Tal Levy 464b46437f Introduce the GrokProcessor
Also moved all processor classes into a subdirectory and introduced a
ConfigException class to be a catch-all for things that can go wrong
when constructing new processors with configurations that possibly throw
exceptions. The GrokProcessor loads patterns from the resources
directory.

fix resource path issue, and add rest-api-spec test for grok

fix rest-spec tests

changes: license, remove configexception, throw IOException

add more tests and fix iso8601-hour pattern

move grok patterns from resources to config

fix tests with pom changes, updated IngestClientIT with grok processor

update gradle build script for grok deps and test configuration

move config files to src/main/packaging

move Env out of Processor, fix test for src/main/packaging change

add docs

clean up test resources task

update Grok to be immutable

- Updated the Grok class to be immutable. This means that all the
  pattern bank loading is handled by an external utility class called
  PatternUtils.
- fixed tabs in the nagios patterns file's comments
2015-11-02 20:27:49 -08:00