Commit Graph

1184 Commits

Author SHA1 Message Date
Yannick Welsch 296b48b9d1 Move discovery.* settings to new setting infrastructure
Closes #16182
2016-01-22 15:35:00 +01:00
Daniel Mitterdorfer e9bb3d31a3 Convert "path.*" and "pidfile" to new settings infra 2016-01-22 15:14:13 +01:00
javanna 4ad5e67433 Geoip processor: remove redundant latitude and longitude fields and make location an object with lat and lon subfields 2016-01-22 12:26:39 +01:00
Ryan Ernst df24019261 Merge pull request #16038 from rjernst/remove_site_plugin
Plugins: Remove site plugins
2016-01-21 12:32:22 -08:00
Tal Levy 3a6c2d008e rename processor_tag to tag 2016-01-21 09:05:42 -08:00
Tal Levy c55f9bbd9b Merge pull request #16132 from talevy/holdmytag
[Ingest] add an AbstractProcessor to help hold the re-used processorTag variable
2016-01-21 07:30:29 -08:00
Simon Willnauer 142547271e Merge branch 'master' into trash_context_and_headers 2016-01-21 14:42:15 +01:00
Martijn van Groningen 44465c94f8 Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-21 11:46:27 +01:00
Tal Levy f1204cb1ab add an AbstractProcessor to help hold the re-used processorTag variable 2016-01-20 14:34:26 -08:00
Robert Muir 6e7e3a2274 Update lucene to r1725675
Adds DFI (divergence from independence) provider.
Fixes test bugs passing invalid values for BM25 parameters.
2016-01-20 03:32:51 -05:00
Martijn van Groningen e2e207687d Fixes due to changes in master branch. 2016-01-19 22:03:26 +01:00
Martijn van Groningen 602a0f183e Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-19 22:01:38 +01:00
Simon Willnauer 3d0cedbabb Merge branch 'master' into trash_context_and_headers 2016-01-19 20:47:43 +01:00
Nik Everett e1e73d9914 Create default for ExecutableScript#unwrap
Tons of scripts just return the variable they are passed and that is the
most intuitive behavior so that may as well be the default implementation.
2016-01-19 13:17:11 -05:00
Jack Conradson 2249a640bb Plan A Update...
New Features:

Exceptions (throw, try/catch)
Infinite Loop Checking
Iterators added to the API

Fixes:

Improved loop path analysis
Fixed set/add issue with shortcuts in lists
Fixed minor promotion issue with incorrect type
Fixed score issue related to score extending Number

Documentation:

Added JavaDocs for some files
2016-01-19 09:49:03 -08:00
David Pilato 725d7f968b Merge branch 'fix/12567-azure-timeout' 2016-01-19 18:02:33 +01:00
David Pilato 6db5b5033c Change exception message for wrong timeout in azure repository settings
Backport in master this change: https://github.com/elastic/elasticsearch/pull/15950#discussion-diff-50128378

Related to #15080
Related to #15950
2016-01-19 18:01:54 +01:00
David Pilato d6e6ef4ec4 Merge branch 'pr/tests-azure-repo' 2016-01-19 17:07:59 +01:00
Nik Everett 6b6d01c4fe Merge pull request #16072 from nik9000/xlint_plugins
Remove remaining xlints from plugins
2016-01-19 10:54:15 -05:00
Nik Everett 4efa8c4ff5 Remove remaining xlints from plugins 2016-01-19 10:53:48 -05:00
David Pilato 8d35bca4db Add more tests for Azure Repository client selection
One test we forgot in #14843 and #13779 is the default client selection.

Most of the time, users won't define explicitly which client they want to use because they are providing only one connection to Azure storage:

```yml
cloud:
    azure:
        storage:
            my_account:
                account: your_azure_storage_account
                key: your_azure_storage_key
```

Then using the default client like this:

```sh
# This one will use the default account (my_account1)
curl -XPUT localhost:9200/_snapshot/my_backup1?pretty -d '{
  "type": "azure"
}'
```

This commit adds tests to check that the right client is still selected when no client is explicitly set when creating the snapshot.
2016-01-19 14:23:08 +01:00
David Pilato 11e7a746ae Replace server side timeout by client side timeout
This commit replaces server side timeout (which is BTW not correctly implemented in azure client 2.0.0 but fixed later #16084) with a client side timeout.

As a consequence, for each request sent to azure, azure client will raise an exception after a given amount of time (timeout).

Closes #12567
2016-01-19 13:56:08 +01:00
Simon Willnauer fbfa9f4925 Merge branch 'master' into new_index_settings 2016-01-19 10:13:48 +01:00
Simon Willnauer 8e0390b09e Register index.version_created for several analysis plugin tests 2016-01-19 09:35:32 +01:00
Adrien Grand d6cbd6f2f0 Merge pull request #16059 from jpountz/enhancement/mapping_merge_reason
Expose the reason why a mapping merge is issued.
2016-01-19 09:27:32 +01:00
Simon Willnauer 20fe8ac854 Register index.version.created in Kuromoji unittest since it's a node level setting here 2016-01-19 09:14:41 +01:00
Nik Everett 53cce3c399 Merge pull request #16071 from nik9000/xlint_lang_groovy
Remove Xlint from lang-groovy and discovery-azure
2016-01-18 21:06:11 -05:00
Ryan Ernst ef4f0a8699 Test: Make rest test framework accept http directly for the test cluster
The rest test framework, because it used to be tightly integrated with
ESIntegTestCase, currently expects the addresses for the test cluster to
be passed using the transport protocol port. However, it only uses this
to then find the http address.

This change makes ESRestTestCase extend from ESTestCase instead of
ESIntegTestCase, and changes the sysprop used to tests.rest.cluster,
which now takes the http address.

closes #15459
2016-01-18 16:44:14 -08:00
Nik Everett bd1324ccc9 Remove Xlint from lang-groovy and discovery-azure
discovery-azure didn't actually need it.

This removes all non-default Xlints from modules.
2016-01-18 18:24:44 -05:00
Nik Everett 08ca9d7645 Fix warnings in lang-plan-a
Removes all Xlint skips in lang-plan-a. Warnings were just some extra
casts which were removed, some raw types which we just needed to <?>, and
a couple of unchecked casts in data that we know to be Map<String, Object>
because that structure is super-ultra-common in scripts.
2016-01-18 15:27:51 -05:00
Adrien Grand 055953d6b3 Expose the reason why a mapping merge is issued.
This would be useful in order to only perform some validations in the case of
a mapping update and in cases when a mapping is restored eg. after a restart,
such as discussed in #15989.

This replaces the current `applyDefault` parameter which can be derived from
the mapping merge reason: the default mapping should be applied only in case of
a mapping update, if the mapping does not exist yet and if this is not the
default mapping.
2016-01-18 17:41:23 +01:00
Simon Willnauer a61723b538 convert index.store.fs.fs_lock 2016-01-18 10:17:11 +01:00
Simon Willnauer 7925e2ef84 convert IndexModule settings 2016-01-18 09:23:35 +01:00
Simon Willnauer 79f4697f3e Register MockFSDirectoryService settings 2016-01-18 09:23:34 +01:00
Simon Willnauer a8eedd0457 fix all kinds of crazy test failures 2016-01-18 09:23:34 +01:00
Simon Willnauer ceef6d7a42 fix Murmur3FieldMapperTests 2016-01-18 09:23:34 +01:00
Ryan Ernst 3b78267c71 Plugins: Remove site plugins
Site plugins used to be used for things like kibana and marvel, but
there is no longer a need since kibana (and marvel as a kibana plugin)
uses node.js. This change removes site plugins, as well as the flag for
jvm plugins. Now all plugins are jvm plugins.
2016-01-16 22:45:37 -08:00
Tal Levy 9f48df9736 Add on_failure support for verbose _simulate execution and introduce optional processor_tag to Processors 2016-01-15 14:56:20 -08:00
javanna af36e5f01e updated ingest-geoip plugin description 2016-01-15 15:16:09 +01:00
javanna 050585e89f remove BiFunction<Environment, TemplateService, Processor.Factory> in favour of Function<TemplateService, Processor.Factory>
the environment is now available through NodeModule#getNode#getEnvironment and can be retrieved during onModule(NodeModule), no need for this indirection anymore using the BiFunction
2016-01-15 15:12:53 +01:00
javanna dd7cae7c19 move DatabaseReaders initialization to IngestGeoIpPlugin#onModule 2016-01-15 15:02:35 +01:00
Martijn van Groningen 21cc0b2316 Cleanup ingest initialization code.
* Folded IngestModule into NodeModule
* Renamed IngestBootstrapper to IngestService
* Let NodeService construct IngestService and removed the Guice annotations
* Let IngestService implement Closable
2016-01-15 13:35:06 +01:00
javanna 9c06736dbd Merge branch 'master' into feature/ingest 2016-01-15 10:11:56 +01:00
David Pilato ed45ad6327 Fix Azure repository with only one primary account
Using a single azure account is now rejected.
This commit fixes this issue and adds a test for it.

This regression was introduced with #13779. Hopefully no elasticsearch version has been released since then.

Needs to be merged in 2.2, 2.x and master branches.
2016-01-14 13:50:02 +01:00
Martijn van Groningen f3883343cb Move the pipeline configuration from the dedicated index to the cluster state.
Closes #15842
2016-01-13 22:59:36 +01:00
Simon Willnauer ce42ae4cf3 Merge pull request #15922 from s1monw/feature/ingest
Review feedback and several cleanups
2016-01-13 12:04:04 +01:00
Simon Willnauer 574d1b35b3 Replace ContextAndHeaders with a ThreadPool based ThreadLocal implementation
ContextAndHeaders has a massive impact on the core infrastructure since it has to
be manually passed on to all relevant places across threads/network calls etc. For the same reason
it's also very error prone and easily forgotten on potentially relevant APIs.

The new ThreadContext is associated with a ThreadPool (node or transport client) and ensures that
headers and context registered on a current thread are inherited to new threads spawned, send across
the network to be deserialized on the receiver end as well as restored on the response handling thread
once the response is received.
2016-01-13 11:53:32 +01:00
javanna ea8065aa3d Merge branch 'master' into feature/ingest 2016-01-12 18:28:42 +01:00
Simon Willnauer 4d38a47eb5 Review feedback and several cleanups 2016-01-12 12:06:14 +01:00
Martijn van Groningen 7bdd2583aa Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-12 01:01:30 +01:00
Nik Everett 01ce49e94e Ban Serializable
1. Uses forbidden patterns to prevent things from referencing
java.io.Serializable or from mentioning serialVersionUID.
2. Uses -Xlint:-serial so we don't have to hear from javac that we aren't
declaring serialVersionUID on any classes that we make that happen to extend
Serializable.
3. Remove Serializable and serialVersionUID declarations.

I didn't use forbidden apis because it doesn't look like it has a way to ban
explicitly implementing Serializable. If you try to ban Serializable with
forbidden apis you end up banning all Exceptions and all Strings.

Closes #15847
2016-01-11 16:57:31 -05:00
Nik Everett b29c416b7c Merge pull request #15848 from nik9000/xlint3
Remove -Xlint:-deprecation from all but core
2016-01-11 16:55:13 -05:00
Martijn van Groningen cd2155311f renamed ingest plugin to ingest-geoip plugin, since it only contains the geoip processor 2016-01-08 22:50:54 +01:00
Martijn van Groningen 1637fe9e0b Moved the grok processor to its own module, so that it will available out-of-the-box, while its dependencies are isolated 2016-01-08 21:59:23 +01:00
Robert Muir 46a8a48d23 Merge pull request #15851 from rmuir/geoip-nodns
[ingest] Don't do DNS lookups from GeoIpProcessor
2016-01-08 13:17:20 -05:00
Nik Everett 6250f4dbaa Remove deprecated azure settings 2016-01-08 13:13:14 -05:00
Jason Tedor 871d1b4885 Remove and forbid use of j.u.c.ThreadLocalRandom
This commit removes and now forbids all uses of
java.util.concurrent.ThreadLocalRandom across the codebase. The
underlying issue with ThreadLocalRandom is that it can not be
seeded. This means that if ThreadLocalRandom is used in production code,
then tests that cover any code path containing ThreadLocalRandom will be
prevented from being reproducible by use of ThreadLocalRandom. Instead,
using org.elasticsearch.common.random.Randomness#get will give
reproducible sources of random when running under tests and otherwise
still give an instance of ThreadLocalRandom when running as production
code.
2016-01-08 12:23:48 -05:00
Nik Everett 98fdb39d3d Remove deprecated settings 2016-01-08 11:17:56 -05:00
javanna de2eac4c49 removed leftover jodatime dependency 2016-01-08 14:31:35 +01:00
javanna ae69d46f92 move processors that have no deps to core, also move to core rest spec and tests and set node.inget to true by default 2016-01-08 10:39:39 +01:00
Robert Muir 9c3ebb83a7 Don't do DNS lookups from GeoIpProcessor
There is no need to involve DNS in this!
2016-01-07 23:12:44 -05:00
Nik Everett 81a7607256 Remove -Xlint:-deprecation from plugins
Instead we suppress warnings about using deprecated stuff near the usage
site with a comment about why its ok.
2016-01-07 20:44:46 -05:00
javanna adac314328 revert move of IngestPlugin class
This was moved accidentally as part of a previous refactoring.
2016-01-07 18:32:46 +01:00
Nik Everett 42cb2a5f02 Merge pull request #15811 from nik9000/def_cleanup
Cleanups for Def
2016-01-07 11:03:01 -05:00
javanna 03fe38681e renamed qa package o.e.plugin.ingest to o.e.ingest
This way InternalTemplateService constructor can be set back to package private visibility
2016-01-07 15:51:52 +01:00
javanna 1ea690e814 Merge branch 'feature/ingest' into enhancement/move_to_core 2016-01-07 15:13:31 +01:00
javanna 7d7e0db91d Merge branch 'master' into feature/ingest 2016-01-07 15:09:13 +01:00
Nik Everett 52f28888d5 Merge pull request #15813 from nik9000/xlint1
Remove Xlint:-override,-fallthrough,-static
2016-01-07 08:34:40 -05:00
javanna eca1594969 start ingest thread pool only when node.ingest is set to true 2016-01-07 14:33:56 +01:00
javanna 2803ae09dc addProcessor -> registerProcessor 2016-01-07 13:25:25 +01:00
javanna 18aabd67c8 adapt qa tests for when ingest.node is set to false
CRUD and simulate apis work now fine, every node has the pipelines in memory, but node.ingest disables ingestion, meaning that any index or bulk request with a pipeline id is going to fail
2016-01-07 13:21:06 +01:00
Martijn van Groningen 9ec2e140b8 Merge branch 'master' into feature/ingest 2016-01-07 10:44:21 +01:00
Nik Everett 244120a065 Remove more Xlint skips 2016-01-06 23:53:05 -05:00
Nik Everett 0786c506dc Remove a few more Xlint skips 2016-01-06 23:28:13 -05:00
Nik Everett 20e7fa97db Remove Xlint:-override,-fallthrough,-static
Adds `@SuppressWarnings("fallthrough")` in two places where the fallthrough
is used to implement well known hashing algorithms.
2016-01-06 22:27:14 -05:00
Nik Everett 74c132afc6 Standardize some methods on varargs
Right now we define the same sort of methods as taking String arrays and
string varargs. We should standardize on one and varargs is easier to
call so lets use varargs!
2016-01-06 21:01:58 -05:00
Nik Everett 32605ecb4f Cleanups for Def
Manually I:
1. Added some missing raw types warnings suppressions.
2. Removed some unused unchecked cast warning suppressions.
3. Added <?> to Class.

I let my IDE:
1. Remove unneeded casts.
2. Reorder imports (just ignore these, everyone does).
2016-01-06 20:28:49 -05:00
javanna 9079a7e891 wip: move all the ingest infra to core 2016-01-06 19:10:44 +01:00
javanna da3f460bd1 remove ProcessorFactoryProvider 2016-01-06 17:31:40 +01:00
javanna f651f5a531 remove MapBinder guice binding for processors, use ProcessorsRegistry instead 2016-01-06 17:31:40 +01:00
javanna 94469d75f9 revert rename InternalTemplateService -> MustacheTemplateService 2016-01-06 17:31:39 +01:00
javanna 8251a50667 make ProcessorFactoryProvider extend BiFunction 2016-01-06 17:31:39 +01:00
javanna 635b9b5a46 clarified TemplateService comments
We will keep this abstractions as it's convenient, otherwise IngestDocument would depend on ScriptService directly, and would explicitly rely on mustache which is not even part of core. better to have the interface in core, and the impl as part of the ingest plugin, which relies on mustache, shipped with core by default.
2016-01-06 17:31:39 +01:00
javanna fa4dbdaea1 create ProcessorsModule during Node creation rather than as part of IngestPlugin initialization
If we createProcessorsModule as part of the plugin, other plugins will not be able to register their own processors.
2016-01-06 17:31:39 +01:00
javanna 2478aafa46 move ingest api to core 2016-01-06 17:31:39 +01:00
Martijn van Groningen 702f712204 replaced thirdPartyAudit.missingClasses with specific excludes 2016-01-06 17:28:46 +01:00
Martijn van Groningen 1eb5ae1dce fix compile errors due to changes in master 2016-01-06 17:28:35 +01:00
Martijn van Groningen e275af8a58 Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-06 17:28:08 +01:00
Martijn van Groningen 2d6adf6428 Percolator refactoring:
* Added percolator field mapper that extracts the query terms and indexes these terms with the percolator query.
* At percolate time these extracted terms are used to query percolator queries that are like to be evaluated. This can significantly cut down the time it takes to percolate. Whereas before all percolator queries were evaluated if they matches with the document being percolated.
* Changes made to percolator queries are no longer immediately visible, a refresh needs to happen before the changes are visible.
* By default the percolate api only returns upto 10 matches instead of returning all matching percolator queries.
* Made percolate more modular, so that it is easier to add unit tests.
* Added unit tests for the percolator.

Closes #12664
Closes #13646
2016-01-06 16:08:10 +01:00
Luca Cavanna eabd5e2714 Merge pull request #15787 from javanna/enhancement/processor_package
move all the processors under the same package org.elasticsearch.ingest.processor
2016-01-06 13:28:35 +01:00
javanna 1e8995d984 move all the processors under the same package org.elasticsearch.ingest.processor 2016-01-06 12:36:28 +01:00
Igor Motov a89dba27c2 Task Management: Add framework for registering and communicating with tasks
Adds task manager class and enables all activities to register with the task manager. Currently, the immutable Transport*Activity class represents activity itself shared across all requests. This PR adds and an additional structure Task that keeps track of currently running requests and can be used to communicate with these requests using TransportTaskAction.

Related to #15117
2016-01-05 12:24:43 -05:00
Tal Levy 183386173c cleanup simulate test and add docs 2016-01-04 16:10:42 -08:00
Tal Levy 82d87e7149 Merge pull request #15647 from talevy/ingest_fail_processor
[Ingest] Fail Processor
2016-01-04 11:35:51 -08:00
Tal Levy 9dd54f2b4f Introduce a fail processor 2016-01-04 11:33:50 -08:00
Tal Levy f34ce9ddf4 add on_failure context to ingest metadata during executeOnFailure 2016-01-04 11:24:27 -08:00
javanna 8de31b3c64 make index type and id optional in simulate api
Default values are _index, _type and _id.

Closes #15711
2016-01-04 13:11:44 +01:00
Adrien Grand 1a47226d9a Merge pull request #15663 from jpountz/remove/mapping_backcompat
Remove mapping backward compatibilit with pre-2.0.
2016-01-04 10:05:39 +01:00
Robert Muir 25914ae879 Merge pull request #15688 from rmuir/thirdPartyAudit3
Improve thirdPartyAudit check, round 3
2015-12-29 09:24:51 -05:00
Martijn van Groningen f2fce0edca Removed ingest runner main class, `gradle run --debug-jvm` should be used instead. 2015-12-29 13:07:23 +01:00