Commit Graph

190 Commits

Author SHA1 Message Date
Simon Willnauer 4a4e523357 Merge branch 'master' into make_settings_strict 2016-02-03 11:34:12 +01:00
Martijn van Groningen 64ac037a82 Added an ingest qa that tests processor real world like configurations.
Renamed `ingest-with-mustache` to `smoke-test-ingest-with-all-dependencies`
Also renamed `ingest-disabled` to `smoke-test-ingest-disabled` so that the name is more inline with other qa smoke test modules.
2016-02-02 22:37:24 +01:00
Ryan Ernst 9f47b376da Plugin cli: Improve maven coordinates detection
Identifying when a plugin id is maven coordinates is currently done by
checking if the plugin id contains 2 colons. However, a valid url could
have 2 colons, for example when a port is specified. This change adds
another check, ensuring the plugin id with maven coordinates does not
contain a slash, which only a url would have.

closes #16376
2016-02-02 12:17:56 -08:00
Simon Willnauer 818a9eefb2 Make settings validation strict
This commit enableds strict settings validation on node startup. All settings
passed to elasticsearch either through system properties, yaml files or any other
way to pass settings must be registered and valid. Settings that are unknown ie. due to
typos or due to deprecation or removal will cause the node to NOT start up. Plugins
have to declare all their settings on the `SettingsModule#registerSetting` and settings for
plugins that are not installed must be removed.

This commit also removes the ability to specify the nodes name via `-Des.name` or just `name` in the
configuration files. The node name must be prefixed with the node prexif like `node.name: Boom`. Left over
usage of `name` will also cause startup to fail.
2016-02-02 11:32:44 +01:00
Ryan Ernst a2c37c0989 CliTool: Allow unexpected exceptions to propagate
Cli tools currently catch all exceptions, and only print the exception
message, except when a special system property is set. Even with this
flag set, certain exceptions, like IOException, are captured and their
stack trace is always lost.

This change adds a UserError class, which can be used a cli tools to
specify a message to the user, as well as an exit status. All other
exceptions are propagated out of main, so java will exit with non-zero
and print the stack trace.
2016-02-01 16:35:22 -08:00
Tal Levy dcb88909d5 Merge pull request #16324 from talevy/ingest_on_failure_tag
Add processor tags to on_failure metadata in ingest pipeline
2016-02-01 10:13:26 -08:00
Ryan Ernst 3787f437ec Merge branch 'master' into remove_multicast 2016-02-01 07:25:45 -08:00
Ryan Ernst 661e3c99f7 Merge pull request #16336 from rjernst/demangle
Reduce complexity of plugin cli
2016-02-01 07:20:04 -08:00
Ryan Ernst a052dfeb38 Plugins: Reduce complexity of plugin cli
The plugin cli currently is extremely lenient, allowing most errors to
simply be logged. This can lead to either corrupt installations (eg
partially installed plugins), or confused users.

This change rewrites the plugin cli to have almost no leniency.
Unfortunately it was not possible to remove all leniency, due in
particular to how config files are handled.

The following functionality was simplified:
* The format of the name argument to install a plugin is now an official
  plugin name, maven coordinates, or a URL.
* Checksum files are required, and only checked, for official plugins
  and maven plugins. Checksums are also only SHA1.
* Downloading no longer uses a separate thread, and no longer has a timeout.
* Installation, and removal, attempts to be atomic. This only truly works
  when no config or bin files exist.
* config and bin directories are verified before copying is attempted.
* Permissions and user/group are no longer set on config and bin files.
  We rely on the users umask.
* config and bin directories must only contain files, no subdirectories.
* The code is reorganized so each command is a separate class. These
  classes already existed, but were embedded in the plugin cli class, as
  an extra layer between the cli code and the code running for each command.
2016-01-31 20:30:45 -08:00
Ryan Ernst b8f08c35ec Plugin: Remove multicast plugin
closes #16310.
2016-01-29 18:41:31 -08:00
Tal Levy 61e4283a16 Add processor tags to on_failure metadata in ingest pipeline
closes #16202
2016-01-29 14:36:11 -08:00
Simon Willnauer e24fac644a Fix AzureRepositoryF to handle exceptions on close
Fix TribeUnitTests to handle exceptions on close
2016-01-29 17:34:02 +01:00
Jack Conradson 5b836dbb11 Renamed the scripting language Plan A to Painless.
Closes #16245
2016-01-27 10:37:34 -08:00
Simon Willnauer bbba1e5d7f Convert `config.ignore_system_properties` to new settings infra 2016-01-27 16:10:06 +01:00
javanna 533af17068 Tribe node: pass path.conf to inner tribe clients
If we don't do this, and some path.conf is set when starting the tribe node, that path.conf will be ignored and the inner tribe clients will try to read elsewhere, where they most likely don't have permissions to read from.

Closes #16253
Closes #16258
2016-01-27 13:19:56 +01:00
Jason Tedor 087e55cc51 Script mode settings
This commit converts the script mode settings to the new settings
infrastructure. This is a major refactoring of the handling of script
mode settings. This refactoring is necessary because these settings are
determined at runtime based on the registered script engines and the
registered script contexts.
2016-01-27 06:26:04 -05:00
javanna 61630c2b27 migrate node.local and node.mode to new Setting infra 2016-01-26 14:40:46 +01:00
javanna a306836e1a adapt to upstream changes, mustache supports now lists and arrays 2016-01-25 18:01:25 +01:00
javanna 36d98478bf Merge branch 'master' into feature/ingest 2016-01-25 18:01:09 +01:00
Yannick Welsch 296b48b9d1 Move discovery.* settings to new setting infrastructure
Closes #16182
2016-01-22 15:35:00 +01:00
Daniel Mitterdorfer e9bb3d31a3 Convert "path.*" and "pidfile" to new settings infra 2016-01-22 15:14:13 +01:00
Ryan Ernst df24019261 Merge pull request #16038 from rjernst/remove_site_plugin
Plugins: Remove site plugins
2016-01-21 12:32:22 -08:00
Martijn van Groningen 44465c94f8 Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-21 11:46:27 +01:00
Nik Everett 3178d24bea Fix tests I just broke 2016-01-19 18:27:52 -05:00
Nik Everett 2cb7e8ce76 Use our standard xlint with standalone-test
We were not changing the xlint settings there at all. Also cleans up some
generic array warnings that this found by switching them to an ArrayList.
2016-01-19 17:54:47 -05:00
Martijn van Groningen 602a0f183e Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-19 22:01:38 +01:00
Ryan Ernst ef4f0a8699 Test: Make rest test framework accept http directly for the test cluster
The rest test framework, because it used to be tightly integrated with
ESIntegTestCase, currently expects the addresses for the test cluster to
be passed using the transport protocol port. However, it only uses this
to then find the http address.

This change makes ESRestTestCase extend from ESTestCase instead of
ESIntegTestCase, and changes the sysprop used to tests.rest.cluster,
which now takes the http address.

closes #15459
2016-01-18 16:44:14 -08:00
Ryan Ernst 3b78267c71 Plugins: Remove site plugins
Site plugins used to be used for things like kibana and marvel, but
there is no longer a need since kibana (and marvel as a kibana plugin)
uses node.js. This change removes site plugins, as well as the flag for
jvm plugins. Now all plugins are jvm plugins.
2016-01-16 22:45:37 -08:00
Martijn van Groningen f3883343cb Move the pipeline configuration from the dedicated index to the cluster state.
Closes #15842
2016-01-13 22:59:36 +01:00
javanna b111c7bbd5 redirect ingest requests to an ingest node
Rather than failing the request, when a node with node.ingest set to false receives an index or bulk request with a pipeline id, it should try to redirect the request to another node with node.ingest set to true. If there are no node with ingest set to true based on the current cluster state, an exception will be returned and the request will fail. Note that in case there are no ingest nodes and bulk has a pipeline id  specified only for a subset of index requests, the whole bulk will fail.
2016-01-12 12:31:11 +01:00
javanna bac1206161 remove use of request headers/context for pipeline id in favour of instance members added to IndexRequest and BulkRequest
Now that the ingest infra is part of es core we can remove some code that was required by the plugin and have a better integration with es core. We allow to specify the pipeline id in bulk and index as a request parameter, we have a REST filter that parses it and adds it to the relevant action request. That is not required anymore, as we can add this logic to RestIndexAction and RestBulkAction directly, no need for a filter. Also, we can allow to specify a pipeline id for each index requests in a bulk request. The small downside of this is that the ingest filter has to go over each item of a bulk request, all the time, to figure out whether they have a pipeline id.
2016-01-11 18:11:15 +01:00
Martijn van Groningen cd2155311f renamed ingest plugin to ingest-geoip plugin, since it only contains the geoip processor 2016-01-08 22:50:54 +01:00
javanna 8675784e3f adapt error message when node.ingest is set to false and we fail executing a pipeline 2016-01-08 10:39:39 +01:00
javanna ae69d46f92 move processors that have no deps to core, also move to core rest spec and tests and set node.inget to true by default 2016-01-08 10:39:39 +01:00
javanna 03fe38681e renamed qa package o.e.plugin.ingest to o.e.ingest
This way InternalTemplateService constructor can be set back to package private visibility
2016-01-07 15:51:52 +01:00
javanna 18aabd67c8 adapt qa tests for when ingest.node is set to false
CRUD and simulate apis work now fine, every node has the pipelines in memory, but node.ingest disables ingestion, meaning that any index or bulk request with a pipeline id is going to fail
2016-01-07 13:21:06 +01:00
javanna 9079a7e891 wip: move all the ingest infra to core 2016-01-06 19:10:44 +01:00
javanna 94469d75f9 revert rename InternalTemplateService -> MustacheTemplateService 2016-01-06 17:31:39 +01:00
javanna 635b9b5a46 clarified TemplateService comments
We will keep this abstractions as it's convenient, otherwise IngestDocument would depend on ScriptService directly, and would explicitly rely on mustache which is not even part of core. better to have the interface in core, and the impl as part of the ingest plugin, which relies on mustache, shipped with core by default.
2016-01-06 17:31:39 +01:00
javanna 2478aafa46 move ingest api to core 2016-01-06 17:31:39 +01:00
Martijn van Groningen e275af8a58 Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-06 17:28:08 +01:00
javanna 1e8995d984 move all the processors under the same package org.elasticsearch.ingest.processor 2016-01-06 12:36:28 +01:00
Tal Levy f34ce9ddf4 add on_failure context to ingest metadata during executeOnFailure 2016-01-04 11:24:27 -08:00
Lee Hinman 55e6cd50e8 [TEST] Use a longer timeout for plugin manager in test 2015-12-29 10:07:31 -07:00
Robert Muir 180ab2493e Improve thirdPartyAudit check, round 3 2015-12-28 22:38:55 -05:00
Martijn van Groningen dbbb296322 added a `node.ingest` setting that controls whether ingest is active or not. Defaults to `false`.
If `node.ingest` isn't active then ingest related API calls fail and if the `pipeline_id` parameter is set then index and bulk requests fail.
2015-12-22 22:38:49 +01:00
Tal Levy 44d64c8a45 rename pipeline_id param to pipeline 2015-12-22 12:30:04 -08:00
javanna 46f99a11a0 Add append processor
The append processor allows to append one or more values to an existing list; add a new list with the provided values if the field doesn't exist yet, or convert an existing scalar into a list and add the provided values to the newly created  list.

This required adapting of IngestDocument#appendFieldValue behaviour, also added support for templating to it.

Closes #14324
2015-12-22 16:11:43 +01:00
javanna 1b7dc45c28 adapt to upstream changes: remove wildcard imports from qa/ingest-with-mustache 2015-12-22 13:42:15 +01:00
javanna f214271d89 Merge branch 'master' into feature/ingest 2015-12-22 11:14:55 +01:00