Commit Graph

1352 Commits

Author SHA1 Message Date
Simon Willnauer 7925e2ef84 convert IndexModule settings 2016-01-18 09:23:35 +01:00
Simon Willnauer 79f4697f3e Register MockFSDirectoryService settings 2016-01-18 09:23:34 +01:00
Simon Willnauer a8eedd0457 fix all kinds of crazy test failures 2016-01-18 09:23:34 +01:00
Simon Willnauer ceef6d7a42 fix Murmur3FieldMapperTests 2016-01-18 09:23:34 +01:00
Ryan Ernst 3b78267c71 Plugins: Remove site plugins
Site plugins used to be used for things like kibana and marvel, but
there is no longer a need since kibana (and marvel as a kibana plugin)
uses node.js. This change removes site plugins, as well as the flag for
jvm plugins. Now all plugins are jvm plugins.
2016-01-16 22:45:37 -08:00
Tal Levy 9f48df9736 Add on_failure support for verbose _simulate execution and introduce optional processor_tag to Processors 2016-01-15 14:56:20 -08:00
javanna af36e5f01e updated ingest-geoip plugin description 2016-01-15 15:16:09 +01:00
javanna 050585e89f remove BiFunction<Environment, TemplateService, Processor.Factory> in favour of Function<TemplateService, Processor.Factory>
the environment is now available through NodeModule#getNode#getEnvironment and can be retrieved during onModule(NodeModule), no need for this indirection anymore using the BiFunction
2016-01-15 15:12:53 +01:00
javanna dd7cae7c19 move DatabaseReaders initialization to IngestGeoIpPlugin#onModule 2016-01-15 15:02:35 +01:00
Martijn van Groningen 21cc0b2316 Cleanup ingest initialization code.
* Folded IngestModule into NodeModule
* Renamed IngestBootstrapper to IngestService
* Let NodeService construct IngestService and removed the Guice annotations
* Let IngestService implement Closable
2016-01-15 13:35:06 +01:00
javanna 9c06736dbd Merge branch 'master' into feature/ingest 2016-01-15 10:11:56 +01:00
David Pilato ed45ad6327 Fix Azure repository with only one primary account
Using a single azure account is now rejected.
This commit fixes this issue and adds a test for it.

This regression was introduced with #13779. Hopefully no elasticsearch version has been released since then.

Needs to be merged in 2.2, 2.x and master branches.
2016-01-14 13:50:02 +01:00
Martijn van Groningen f3883343cb Move the pipeline configuration from the dedicated index to the cluster state.
Closes #15842
2016-01-13 22:59:36 +01:00
Simon Willnauer ce42ae4cf3 Merge pull request #15922 from s1monw/feature/ingest
Review feedback and several cleanups
2016-01-13 12:04:04 +01:00
Simon Willnauer 574d1b35b3 Replace ContextAndHeaders with a ThreadPool based ThreadLocal implementation
ContextAndHeaders has a massive impact on the core infrastructure since it has to
be manually passed on to all relevant places across threads/network calls etc. For the same reason
it's also very error prone and easily forgotten on potentially relevant APIs.

The new ThreadContext is associated with a ThreadPool (node or transport client) and ensures that
headers and context registered on a current thread are inherited to new threads spawned, send across
the network to be deserialized on the receiver end as well as restored on the response handling thread
once the response is received.
2016-01-13 11:53:32 +01:00
javanna ea8065aa3d Merge branch 'master' into feature/ingest 2016-01-12 18:28:42 +01:00
Simon Willnauer 4d38a47eb5 Review feedback and several cleanups 2016-01-12 12:06:14 +01:00
Martijn van Groningen 7bdd2583aa Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-12 01:01:30 +01:00
Nik Everett 01ce49e94e Ban Serializable
1. Uses forbidden patterns to prevent things from referencing
java.io.Serializable or from mentioning serialVersionUID.
2. Uses -Xlint:-serial so we don't have to hear from javac that we aren't
declaring serialVersionUID on any classes that we make that happen to extend
Serializable.
3. Remove Serializable and serialVersionUID declarations.

I didn't use forbidden apis because it doesn't look like it has a way to ban
explicitly implementing Serializable. If you try to ban Serializable with
forbidden apis you end up banning all Exceptions and all Strings.

Closes #15847
2016-01-11 16:57:31 -05:00
Nik Everett b29c416b7c Merge pull request #15848 from nik9000/xlint3
Remove -Xlint:-deprecation from all but core
2016-01-11 16:55:13 -05:00
Martijn van Groningen cd2155311f renamed ingest plugin to ingest-geoip plugin, since it only contains the geoip processor 2016-01-08 22:50:54 +01:00
Martijn van Groningen 1637fe9e0b Moved the grok processor to its own module, so that it will available out-of-the-box, while its dependencies are isolated 2016-01-08 21:59:23 +01:00
Robert Muir 46a8a48d23 Merge pull request #15851 from rmuir/geoip-nodns
[ingest] Don't do DNS lookups from GeoIpProcessor
2016-01-08 13:17:20 -05:00
Nik Everett 6250f4dbaa Remove deprecated azure settings 2016-01-08 13:13:14 -05:00
Jason Tedor 871d1b4885 Remove and forbid use of j.u.c.ThreadLocalRandom
This commit removes and now forbids all uses of
java.util.concurrent.ThreadLocalRandom across the codebase. The
underlying issue with ThreadLocalRandom is that it can not be
seeded. This means that if ThreadLocalRandom is used in production code,
then tests that cover any code path containing ThreadLocalRandom will be
prevented from being reproducible by use of ThreadLocalRandom. Instead,
using org.elasticsearch.common.random.Randomness#get will give
reproducible sources of random when running under tests and otherwise
still give an instance of ThreadLocalRandom when running as production
code.
2016-01-08 12:23:48 -05:00
Nik Everett 98fdb39d3d Remove deprecated settings 2016-01-08 11:17:56 -05:00
javanna de2eac4c49 removed leftover jodatime dependency 2016-01-08 14:31:35 +01:00
javanna ae69d46f92 move processors that have no deps to core, also move to core rest spec and tests and set node.inget to true by default 2016-01-08 10:39:39 +01:00
Robert Muir 9c3ebb83a7 Don't do DNS lookups from GeoIpProcessor
There is no need to involve DNS in this!
2016-01-07 23:12:44 -05:00
Nik Everett 81a7607256 Remove -Xlint:-deprecation from plugins
Instead we suppress warnings about using deprecated stuff near the usage
site with a comment about why its ok.
2016-01-07 20:44:46 -05:00
javanna adac314328 revert move of IngestPlugin class
This was moved accidentally as part of a previous refactoring.
2016-01-07 18:32:46 +01:00
Nik Everett 42cb2a5f02 Merge pull request #15811 from nik9000/def_cleanup
Cleanups for Def
2016-01-07 11:03:01 -05:00
javanna 03fe38681e renamed qa package o.e.plugin.ingest to o.e.ingest
This way InternalTemplateService constructor can be set back to package private visibility
2016-01-07 15:51:52 +01:00
javanna 1ea690e814 Merge branch 'feature/ingest' into enhancement/move_to_core 2016-01-07 15:13:31 +01:00
javanna 7d7e0db91d Merge branch 'master' into feature/ingest 2016-01-07 15:09:13 +01:00
Nik Everett 52f28888d5 Merge pull request #15813 from nik9000/xlint1
Remove Xlint:-override,-fallthrough,-static
2016-01-07 08:34:40 -05:00
javanna eca1594969 start ingest thread pool only when node.ingest is set to true 2016-01-07 14:33:56 +01:00
javanna 2803ae09dc addProcessor -> registerProcessor 2016-01-07 13:25:25 +01:00
javanna 18aabd67c8 adapt qa tests for when ingest.node is set to false
CRUD and simulate apis work now fine, every node has the pipelines in memory, but node.ingest disables ingestion, meaning that any index or bulk request with a pipeline id is going to fail
2016-01-07 13:21:06 +01:00
Martijn van Groningen 9ec2e140b8 Merge branch 'master' into feature/ingest 2016-01-07 10:44:21 +01:00
Nik Everett 244120a065 Remove more Xlint skips 2016-01-06 23:53:05 -05:00
Nik Everett 0786c506dc Remove a few more Xlint skips 2016-01-06 23:28:13 -05:00
Nik Everett 20e7fa97db Remove Xlint:-override,-fallthrough,-static
Adds `@SuppressWarnings("fallthrough")` in two places where the fallthrough
is used to implement well known hashing algorithms.
2016-01-06 22:27:14 -05:00
Nik Everett 74c132afc6 Standardize some methods on varargs
Right now we define the same sort of methods as taking String arrays and
string varargs. We should standardize on one and varargs is easier to
call so lets use varargs!
2016-01-06 21:01:58 -05:00
Nik Everett 32605ecb4f Cleanups for Def
Manually I:
1. Added some missing raw types warnings suppressions.
2. Removed some unused unchecked cast warning suppressions.
3. Added <?> to Class.

I let my IDE:
1. Remove unneeded casts.
2. Reorder imports (just ignore these, everyone does).
2016-01-06 20:28:49 -05:00
javanna 9079a7e891 wip: move all the ingest infra to core 2016-01-06 19:10:44 +01:00
javanna da3f460bd1 remove ProcessorFactoryProvider 2016-01-06 17:31:40 +01:00
javanna f651f5a531 remove MapBinder guice binding for processors, use ProcessorsRegistry instead 2016-01-06 17:31:40 +01:00
javanna 94469d75f9 revert rename InternalTemplateService -> MustacheTemplateService 2016-01-06 17:31:39 +01:00
javanna 8251a50667 make ProcessorFactoryProvider extend BiFunction 2016-01-06 17:31:39 +01:00
javanna 635b9b5a46 clarified TemplateService comments
We will keep this abstractions as it's convenient, otherwise IngestDocument would depend on ScriptService directly, and would explicitly rely on mustache which is not even part of core. better to have the interface in core, and the impl as part of the ingest plugin, which relies on mustache, shipped with core by default.
2016-01-06 17:31:39 +01:00
javanna fa4dbdaea1 create ProcessorsModule during Node creation rather than as part of IngestPlugin initialization
If we createProcessorsModule as part of the plugin, other plugins will not be able to register their own processors.
2016-01-06 17:31:39 +01:00
javanna 2478aafa46 move ingest api to core 2016-01-06 17:31:39 +01:00
Martijn van Groningen 702f712204 replaced thirdPartyAudit.missingClasses with specific excludes 2016-01-06 17:28:46 +01:00
Martijn van Groningen 1eb5ae1dce fix compile errors due to changes in master 2016-01-06 17:28:35 +01:00
Martijn van Groningen e275af8a58 Merge remote-tracking branch 'es/master' into feature/ingest 2016-01-06 17:28:08 +01:00
Martijn van Groningen 2d6adf6428 Percolator refactoring:
* Added percolator field mapper that extracts the query terms and indexes these terms with the percolator query.
* At percolate time these extracted terms are used to query percolator queries that are like to be evaluated. This can significantly cut down the time it takes to percolate. Whereas before all percolator queries were evaluated if they matches with the document being percolated.
* Changes made to percolator queries are no longer immediately visible, a refresh needs to happen before the changes are visible.
* By default the percolate api only returns upto 10 matches instead of returning all matching percolator queries.
* Made percolate more modular, so that it is easier to add unit tests.
* Added unit tests for the percolator.

Closes #12664
Closes #13646
2016-01-06 16:08:10 +01:00
Luca Cavanna eabd5e2714 Merge pull request #15787 from javanna/enhancement/processor_package
move all the processors under the same package org.elasticsearch.ingest.processor
2016-01-06 13:28:35 +01:00
javanna 1e8995d984 move all the processors under the same package org.elasticsearch.ingest.processor 2016-01-06 12:36:28 +01:00
Igor Motov a89dba27c2 Task Management: Add framework for registering and communicating with tasks
Adds task manager class and enables all activities to register with the task manager. Currently, the immutable Transport*Activity class represents activity itself shared across all requests. This PR adds and an additional structure Task that keeps track of currently running requests and can be used to communicate with these requests using TransportTaskAction.

Related to #15117
2016-01-05 12:24:43 -05:00
Tal Levy 183386173c cleanup simulate test and add docs 2016-01-04 16:10:42 -08:00
Tal Levy 82d87e7149 Merge pull request #15647 from talevy/ingest_fail_processor
[Ingest] Fail Processor
2016-01-04 11:35:51 -08:00
Tal Levy 9dd54f2b4f Introduce a fail processor 2016-01-04 11:33:50 -08:00
Tal Levy f34ce9ddf4 add on_failure context to ingest metadata during executeOnFailure 2016-01-04 11:24:27 -08:00
javanna 8de31b3c64 make index type and id optional in simulate api
Default values are _index, _type and _id.

Closes #15711
2016-01-04 13:11:44 +01:00
Adrien Grand 1a47226d9a Merge pull request #15663 from jpountz/remove/mapping_backcompat
Remove mapping backward compatibilit with pre-2.0.
2016-01-04 10:05:39 +01:00
Robert Muir 25914ae879 Merge pull request #15688 from rmuir/thirdPartyAudit3
Improve thirdPartyAudit check, round 3
2015-12-29 09:24:51 -05:00
Martijn van Groningen f2fce0edca Removed ingest runner main class, `gradle run --debug-jvm` should be used instead. 2015-12-29 13:07:23 +01:00
Martijn van Groningen 9fa3f469e9 fix test after merging in master branch 2015-12-29 13:00:27 +01:00
Martijn van Groningen 79161d77df Merge remote-tracking branch 'es/master' into feature/ingest 2015-12-29 12:56:09 +01:00
David Pilato 96b3166c6d Add timeout settings (default to 5 minutes)
By default, azure does not timeout. This commit adds support for a timeout settings which defaults to 5 minutes.
It's a timeout **per request** not a global timeout for a snapshot request.

It can be defined globally, per account or both. Defaults to `5m`.

```yml
cloud:
    azure:
        storage:
            timeout: 10s
            my_account1:
                account: your_azure_storage_account1
                key: your_azure_storage_key1
                default: true
            my_account2:
                account: your_azure_storage_account2
                key: your_azure_storage_key2
                timeout: 30s
```

In this example, timeout will be 10s for `my_account1` and 30s for `my_account2`.

Closes #14277.
2015-12-29 11:40:48 +01:00
David Pilato a49fe189b0 Support global `repositories.azure.` settings
All those repository settings can also be defined globally in `elasticsearch.yml` file using prefix `repositories.azure.`. For example:

```yml
repositories.azure:
    container: backup-container
    base_path: backups
    chunk_size: 32m
    compress": true
```

Closes #13776.
2015-12-29 10:43:01 +01:00
Robert Muir 180ab2493e Improve thirdPartyAudit check, round 3 2015-12-28 22:38:55 -05:00
Adrien Grand 5eb7555ffb Make text parsing less lenient.
It now requires that the parser is on a value.
2015-12-28 16:06:42 +01:00
Martijn van Groningen 4a0ec0da26 Merge remote-tracking branch 'es/master' into feature/ingest 2015-12-24 15:34:20 +01:00
Martijn van Groningen 1936d6118d Added index template for the '.ingest' index and added logic that ensure the index template is installed.
An index template for the '.ingest' index is required because:
* We don't want arbitrary fields in pipeline documents, because that can turn into upgrade problems if we add more properties to the pipeline dsl.
* We know what are the usages are of the '.ingest' index, so we can optimize for that and prevent that this index is used for different purposes.

Closes #15001
2015-12-24 15:18:07 +01:00
Adrien Grand af122f4151 Remove mapping backward compatibilit with pre-2.0.
This removes the backward compatibility layer with pre-2.0 indices, notably
the extraction of _id, _routing or _timestamp from the source document when a
path is defined.
2015-12-24 13:47:37 +01:00
Robert Muir d144ba24a5 Merge pull request #15588 from rmuir/hdfs2-only
merge current hdfs improvements to master
2015-12-23 18:17:22 -05:00
Robert Muir f14a21639c add cleanups from simon 2015-12-23 18:15:33 -05:00
Martijn van Groningen 767114adec Merge remote-tracking branch 'es/master' into feature/ingest 2015-12-23 15:16:20 +01:00
Adrien Grand d8d8666877 Remove `index_name` back compat.
Since 2.0 we enforce that fields have the same full and index names. So in 3.x
we can remove the ability to have different names on the same field.
2015-12-23 14:55:26 +01:00
Adrien Grand 56d2dd701e Fix SizeMappingTests failure. 2015-12-23 10:48:00 +01:00
Adrien Grand a2072fe927 Merge pull request #15539 from jpountz/fix/immutable_document_mapper
Make mapping updates more robust.
2015-12-23 09:55:42 +01:00
Adrien Grand f535c27024 Make mapping updates more robust.
This changes a couple of things:

Mappings are truly immutable. Before, each field mapper stored a
MappedFieldTypeReference that was shared across fields that have the same name
across types. This means that a mapping update could have the side-effect of
changing the field type in other types when updateAllTypes is true. This works
differently now: after a mapping update, a new copy of the mappings is created
in such a way that fields across different types have the same MappedFieldType.
See the new Mapper.updateFieldType API which replaces MappedFieldTypeReference.

DocumentMapper is now immutable and MapperService.merge has been refactored in
such a way that if an exception is thrown while eg. lookup structures are being
updated, then the whole mapping update will be aborted. As a consequence,
FieldTypeLookup's checkCompatibility has been folded into copyAndAddAll.

Synchronization was simplified: given that mappings are truly immutable, we
don't need the read/write lock so that no documents can be parsed while a
mapping update is being processed. Document parsing is not performed under a
lock anymore, and mapping merging uses a simple synchronized block.
2015-12-23 09:55:07 +01:00
Martijn van Groningen 8b671d7d93 added note 2015-12-22 22:52:23 +01:00
Martijn van Groningen dbbb296322 added a `node.ingest` setting that controls whether ingest is active or not. Defaults to `false`.
If `node.ingest` isn't active then ingest related API calls fail and if the `pipeline_id` parameter is set then index and bulk requests fail.
2015-12-22 22:38:49 +01:00
Tal Levy 44d64c8a45 rename pipeline_id param to pipeline 2015-12-22 12:30:04 -08:00
Lee Hinman 482843e27b Fix build to run correctly on FreeBSD
This adds the required changes/checks so that the build can run on
FreeBSD.

There are a few things that differ between FreeBSD and Linux:

- CPU probes return -1 for CPU usage
- `hot_threads` cannot be supported on FreeBSD

From OpenJDK's `os_bsd.cpp`:

```c++
bool os::is_thread_cpu_time_supported() {
  #ifdef __APPLE__
  return true;
  #else
  return false;
  #endif
}
```

So this API now returns (for each FreeBSD node):

```
curl -s localhost:9200/_nodes/hot_threads
::: {Devil Hunter Gabriel}{q8OJnKCcQS6EB9fygU4R4g}{127.0.0.1}{127.0.0.1:9300}
   hot_threads is not supported on FreeBSD
```

- multicast fails in native `join` method - known bug:
  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246

Which causes:

```
1> Caused by: java.net.SocketException: Invalid argument
1>    at java.net.PlainDatagramSocketImpl.join(Native Method)
1>    at java.net.AbstractPlainDatagramSocketImpl.join(AbstractPlainDatagramSocketImpl.java:179)
1>    at java.net.MulticastSocket.joinGroup(MulticastSocket.java:323)
1>    at org.elasticsearch.plugin.discovery.multicast.MulticastChannel$Plain.buildMulticastSocket(MulticastChannel.java:309)
```

So these tests are skipped on FreeBSD.

Resolves #15562
2015-12-22 12:36:04 -07:00
Tal Levy 0bf4c8fb82 Add on_failure field to processors and pipelines.
both processors and pipelines now have the ability to define
a separate list of processors to be executed if the original line
of execution throws an Exception.

processors without an on_failure parameter defined will throw an
exception and exit the pipeline immediately. processors with on_failure
defined will catch the exception and allow for further processors to
run. Exceptions within the on_failure block will be treated the same as
the top-level.
2015-12-22 10:30:42 -08:00
Robert Muir 7abd051734 better containing of hadoop for actual blobstore operations 2015-12-22 12:07:37 -05:00
javanna 46f99a11a0 Add append processor
The append processor allows to append one or more values to an existing list; add a new list with the provided values if the field doesn't exist yet, or convert an existing scalar into a list and add the provided values to the newly created  list.

This required adapting of IngestDocument#appendFieldValue behaviour, also added support for templating to it.

Closes #14324
2015-12-22 16:11:43 +01:00
Martijn van Groningen 72fc34731e applied feedback 2015-12-22 12:20:16 +01:00
Martijn van Groningen 7e99ee4edf ExecutionService should be able to process multiple index requests at a time. 2015-12-22 12:02:36 +01:00
Martijn van Groningen 3fcb8ecd54 Upgraded geoip2 and its dependencies 2015-12-22 11:39:43 +01:00
javanna 188953922a adapt to upstream changes
Removed wildcard imports and fixed two broken license headers
2015-12-22 11:22:47 +01:00
javanna f214271d89 Merge branch 'master' into feature/ingest 2015-12-22 11:14:55 +01:00
Robert Muir 010d1a89c5 Merge branch 'master' into hdfs2-only 2015-12-22 00:40:54 -05:00
Robert Muir 91dcc9e073 tidy up 2015-12-22 00:28:53 -05:00
Robert Muir 9573bb9f15 make sure BlobStore.close always triggers ACE on any access afterwards 2015-12-22 00:21:03 -05:00
Robert Muir a04268e42e reorder checks 2015-12-21 23:52:16 -05:00
Robert Muir a587ba110c add some safety around repository 2015-12-21 23:48:22 -05:00
Ryan Ernst d104d6d652 Refactor hdfs unit tests to be simple and check every configuration
error condition
2015-12-21 19:38:54 -08:00
Ryan Ernst af7d6b629c Change hdfs unit tests to be a single node test instead of integ test 2015-12-21 18:32:28 -08:00
Robert Muir c54d53c8d5 streamline these classes a bit 2015-12-21 21:11:06 -05:00
Robert Muir 795869c345 remove filecontextfactory 2015-12-21 20:55:17 -05:00
Robert Muir 956281f039 remove shitton of permissions 2015-12-21 20:26:13 -05:00
Ryan Ernst d0e9306413 Remove reading node settings as defaults for hdfs repository settings 2015-12-21 16:25:55 -08:00
Robert Muir 3c07a427dc fix exc handling 2015-12-21 19:06:56 -05:00
Ryan Ernst 26eaa16a89 Remove "additional config" from hdfs repositories 2015-12-21 15:55:13 -08:00
Robert Muir 2cbfc54a81 avoid too-long classpath so it works on windows 2015-12-21 18:25:08 -05:00
Robert Muir 7065639a26 add test for listing 2015-12-21 17:25:15 -05:00
Robert Muir 5ebcf183e5 tests 2015-12-21 17:02:50 -05:00
Robert Muir b8524bdb11 add tests 2015-12-21 16:16:24 -05:00
Robert Muir 3a2464b80e improve build logic on windows without native libraries 2015-12-21 15:37:34 -05:00
Robert Muir 0ed45c5bfb remove filesystem leniency 2015-12-21 14:16:53 -05:00
Robert Muir deaf8884e9 Fix exc handling 2015-12-21 13:04:22 -05:00
Robert Muir 3ffd1a5219 final 2015-12-21 12:54:33 -05:00
Robert Muir f81b12e327 minimize accessiblity, remove unused threadpool 2015-12-21 12:39:40 -05:00
Adrien Grand cf52e96c42 Upgrade to lucene-5.5.0-snapshot-1721183.
Some files that implement or use the Scorer API had to be changed because of
https://issues.apache.org/jira/browse/LUCENE-6919.
2015-12-21 17:02:08 +01:00
Adrien Grand ac393b7a31 Make mappings tests more realistic.
DocumentMapperParser has both parse and parseCompressed methods. Except that the
parse methods are ONLY used from the unit tests. This commit removes the parse
method and moves all tests to parseCompressed so that they test more
realistically how mappings are managed.

Then I renamed parseCompressed to parse given that this is the only alternative
anyway.
2015-12-21 10:44:00 +01:00
Robert Muir f67390e0c8 in the plugin: guard against HADOOP_HOME in environment on any platform.
hdfs fixture: minihdfs works on windows now, if things are properly set
but our test fixture still cannot launch this on windows.
2015-12-21 02:21:53 -05:00
Robert Muir 53530f1243 remove hacks, test fixtures are clean before each execution 2015-12-20 22:23:30 -05:00
Robert Muir 935c2c75f6 Remove slf4j hack 2015-12-20 22:08:18 -05:00
Robert Muir 04966bcc3e contain and improve hack 2015-12-20 21:02:03 -05:00
Robert Muir 03a2b6b01b Disable HDFS fixture on windows, it requires native libraries. 2015-12-20 16:30:19 -08:00
Robert Muir a37417085d blind stab at unit test issues on windows 2015-12-20 18:31:55 -05:00
Robert Muir ee546ff655 try to get windows working 2015-12-20 17:10:01 -05:00
Robert Muir 2347e3c373 Get forbidden apis passing again, this needs to be investigated 2015-12-20 16:17:17 -05:00
Robert Muir 7ac49bb278 Merge branch 'hdfs2-only' of github.com:costin/elasticsearch into hdfs2-only 2015-12-20 16:12:23 -05:00
Robert Muir 12a8428dfb Add MiniHDFS test fixture, started before integTest and shut down after.
Currently uses a hardcoded port (9999), need to apply MavenFilteringHack after it starts.
2015-12-20 16:00:37 -05:00
Martijn van Groningen 66d8b5342f s/PipelineStoreBootstrapper/IngestBootstrapper 2015-12-20 20:18:46 +01:00
Costin Leau 3204e87220 Restrict usage to HDFS only 2015-12-20 15:53:18 +02:00
Robert Muir ae89c6e51c Merge branch 'master' into hdfs2-only 2015-12-19 21:52:52 -05:00
Ryan Ernst 9cb4c82c58 Build: Add fixture capabilities to integ tests
This change adds a Fixture class for use by gradle. A Fixture is an
external process that integration tests will use. It can be added as a
dependsOn for integTest, and will automatically be shutdown upon success
or failure, as well as relevant information dumped on failure. There is
also an example fixture in this change.
2015-12-19 15:46:21 -08:00
Robert Muir 8c6f5a0c60 add failing test 2015-12-19 15:05:38 -08:00
Robert Muir 5dcccca848 add example fixture 2015-12-19 15:05:37 -08:00
Robert Muir d171773bdb remove leniency in tests 2015-12-19 04:39:01 -05:00
Robert Muir e2b2ee24fa Add licensing for dependencies 2015-12-19 03:06:40 -05:00
Robert Muir 9df447295c Fix unit tests (also works from IDE). 2015-12-19 02:43:27 -05:00
Robert Muir 3269beeb4d don't throw exceptions from ctor, guice is hell 2015-12-19 02:09:14 -05:00
Robert Muir f174e96a14 explicitly initialize some hadoop classes elevated, so we don't rely on classloading order.
maybe this allows us to do less stuff in doPriv later, we will see. at least it makes things
like unit testing easier.
2015-12-19 00:21:01 -05:00
Robert Muir 2e8c68d09b Remove no-longer needed domaincombiner stuff 2015-12-18 23:51:41 -05:00
Robert Muir 02fbd55118 enable thirdPartyAudit so you can see the crazy shit hadoop does 2015-12-18 23:45:05 -05:00
Robert Muir bc11962438 get full snapshot restore tests passing 2015-12-18 23:16:41 -05:00
Robert Muir fbe3d64ea4 add passing test that takes snapshot 2015-12-18 22:55:15 -05:00
Robert Muir 75ef9da53f get up to connectexception 2015-12-18 22:11:58 -05:00
Ryan Ernst c2c5081830 Remove uneeded class loading stuff from hdfs plugin 2015-12-18 17:01:38 -08:00
Ryan Ernst 91fe99a7f6 Make hdfs plugin not use transitive deps 2015-12-18 16:52:22 -08:00
Costin Leau 7584810ff4 * Make plugin hadoop2-only
Polish MiniDFS cluster to be Hadoop2 (instead of Hadoop1) based
2015-12-19 01:35:53 +02:00
Ryan Ernst 690fb2cd3f Rename InternalFilters.Bucket to InternalFilters.InternalBucket to avoid name collision 2015-12-18 13:22:20 -08:00
Ryan Ernst 4ea19995cf Remove wildcard imports 2015-12-18 12:43:47 -08:00
Robert Muir 447729f0e1 add missing license headers 2015-12-18 13:08:17 -05:00
Robert Muir 2e2e328879 add missing license header 2015-12-18 13:02:39 -05:00
Martijn van Groningen 6dfcee6937 Added an internal reload pipeline api that makes sure pipeline changes are visible on all ingest nodes after modifcations have been made.
* Pipeline store can now only start when there is no .ingest index or all primary shards of .ingest have been started
* IngestPlugin adds`node.ingest` setting to `true`. This is used to figure out to what nodes to send the refresh request too. This setting isn't yet configurable. This will be done in a follow up issue.
* Removed the background pipeline updater and added added logic to deal with specific scenarious to reload all pipelines.
* Ingest services are no longer be managed by Guice. Only the bootstrapper gets managed by guice and that contructs
all the services/components ingest will need.
2015-12-18 18:24:27 +01:00
Martijn van Groningen e8a8e22e09 Add template infrastructure, removed meta processor and added template support to set and remove processor.
Added ingest wide template infrastructure to IngestDocument
Added a TemplateService interface that the ingest framework uses
Added a TemplateService implementation that the ingest plugin provides that delegates to the ES' script service
Cut SetProcessor over to use the template infrastructure for the `field` and `value` settings.
Removed the MetaDataProcessor
Removed dependency on mustache library
Added qa  ingest mustache rest test so that the ingest and mustache integration can be tested.
2015-12-18 17:35:53 +01:00
Martijn van Groningen a56902567e don't register rest actions on transport clients 2015-12-18 15:49:00 +01:00
Martijn van Groningen 2c5bb84851 fix copyDefaultGeoIp2DatabaseFiles task to work again 2015-12-18 15:45:56 +01:00
javanna 3e155f7b54 adapt to upstream changes: thirdPartyAudit.missingClasses set to true
geoip depends on asm and google http client which we don't need
2015-12-18 11:14:39 +01:00
javanna f349669071 adapt to upstream changes: enableMockModules => getMockPlugins 2015-12-18 11:13:34 +01:00
javanna 8bae93eee1 adapt to upstream changes: StringText => Text 2015-12-18 11:13:18 +01:00
javanna 885b01fb49 adapt to upstream changes: RestModule -> NetworkModule 2015-12-18 10:36:29 +01:00
javanna 5f2df6b95a Merge branch 'master' into feature/ingest 2015-12-18 10:34:07 +01:00
Simon Willnauer eca2435838 Merge branch 'master' into settings_prototype 2015-12-18 09:15:58 +01:00
Adrien Grand 6ea16671f4 Simplify the Text API.
We have the Text API, which is essentially a wrapper around a String and a
BytesReference and then we have 3 implementations depending on whether the
String view should be cached, the BytesReference view should be cached, or both
should be cached.

This commit merges everything into a single Text that is essentially the old
StringAndBytesText impl.

Long term we should look into whether this API has any performance benefit or
if we could just use plain strings. This would greatly simplify all our other
APIs that currently use Text.
2015-12-17 17:22:38 +01:00
Simon Willnauer eae3da3b54 Merge branch 'master' into settings_prototype 2015-12-17 15:13:41 +01:00
Adrien Grand 6ccc759691 Merge pull request #15480 from jpountz/fix/mapping_explicit_defaults
Make mapping serialization more robust.
2015-12-17 11:23:27 +01:00
Robert Muir a7cc91e868 Merge pull request #15501 from rmuir/sheisty_classes
thirdPartyAudit round 2
2015-12-17 03:44:27 -05:00
Robert Muir 6692e42d9a thirdPartyAudit round 2
This fixes the `lenient` parameter to be `missingClasses`. I will remove this boolean and we can handle them via the normal whitelist.
It also adds a check for sheisty classes (jar hell with the jdk).
This is inspired by the lucene "sheisty" classes check, but it has false positives. This check is more evil, it validates every class file against the extension classloader as a resource, to see if it exists there. If so: jar hell.

This jar hell is a problem for several reasons:

1. causes insanely-hard-to-debug problems (like bugs in forbidden-apis)
2. hides problems (like internal api access)
3. the code you think is executing, is not really executing
4. security permissions are not what you think they are
5. brings in unnecessary dependencies
6. its jar hell

The more difficult problems are stuff like jython, where these classes are simply 'uberjared' directly in, so you cant just fix them by removing a bogus dependency. And there is a legit reason for them to do that, they want to support java 1.4.
2015-12-17 02:35:00 -05:00
Jack Conradson 4523eaec88 Added plumbing for compile time script parameters.
Closes #15464
2015-12-16 18:29:21 -08:00
Robert Muir 4f9d4103f2 Merge pull request #15491 from rmuir/forbidden_third_party
Add gradle thirdPartyAudit to precommit tasks
2015-12-16 18:56:50 -05:00
Robert Muir 42138007db add some more comments about internal api usage 2015-12-16 18:56:02 -05:00
Robert Muir ee79d46583 Add gradle thirdPartyAudit to precommit tasks 2015-12-16 16:38:16 -05:00
Ryan Ernst a2b8f4b90a Merge pull request #15434 from rjernst/http_type
Expose http.type setting, and collapse al(most all) modules relating to transport/http
2015-12-16 11:54:30 -08:00
Simon Willnauer 71b204ea49 Merge branch 'master' into settings_prototype 2015-12-16 20:29:21 +01:00
Adrien Grand 8ac8c1f547 Make mapping serialization more robust.
When creating a metadata mapper for a new type, we reuse an existing
configuration from an existing type (if any) in order to avoid introducing
conflicts. However this field type that is provided is considered as both an
initial configuration and the default configuration. So at serialization time,
we might only serialize the difference between the current configuration and
this default configuration, which might be different to what is actually
considered the default configuration.

This does not cause bugs today because metadata mappers usually override the
toXContent method and compare the current field type with Defaults.FIELD_TYPE
instead of defaultFieldType() but I would still like to do this change to
avoid future bugs.
2015-12-16 16:08:45 +01:00
Martijn van Groningen 07951fc731 added comment why 'accessDeclaredMembers' permission is needed 2015-12-16 10:11:46 +01:00
Simon Willnauer 6ea266a89c Merge branch 'master' into settings_prototype 2015-12-15 16:33:01 +01:00
Adrien Grand d94bba2d9c Remove back compat for the `path` option.
The `path` option allowed to index/store a field `a.b.c` under just `c` when
set to `just_name`. This "feature" has been removed in 2.0 in favor of `copy_to`
so we can remove the back compat in 3.x.
2015-12-15 14:55:23 +01:00
Adrien Grand 50eeafa75c Make mappings immutable.
Today mappings are mutable because of two APIs:
 - Mapper.merge, which expects changes to be performed in-place
 - IncludeInAll, which allows to change whether values should be put in the
   `_all` field in place.

This commit changes both APIs to return a modified copy instead of modifying in
place so that mappings can be immutable. For now, only the type-level object is
immutable, but in the future we can imagine making them immutable at the
index-level so that mapping updates could be completely atomic at the index
level.

Close #9365
2015-12-15 10:20:28 +01:00
Martijn van Groningen e87709f593 fix ingest runner 2015-12-15 10:18:19 +01:00
Ryan Ernst 60d35c81af Plugins: Expose http.type setting, and collapse al(most all) modules relating to transport/http
This change adds back the http.type setting. It also cleans up all the
transport related guice code to be consolidated within the
NetworkModule (as transport and http related stuff is what and how ES
exposes over the network). The setter methods previously used by some
plugins to override eg the TransportService or HttpServerTransport are
removed, and those plugins should now register a custom implementation
of the class with a name and set that using the appropriate config
setting. Note that I think ActionModule should also be moved into here,
to sit along side the rest actions, but I left that for a followup.

closes #14148
2015-12-14 22:01:04 -08:00
Costin Leau 7bca97bba6 HDFS Snapshot/Restore plugin
Migrated from ES-Hadoop. Contains several improvements regarding:

* Security
Takes advantage of the pluggable security in ES 2.2 and uses that in order
to grant the necessary permissions to the Hadoop libs. It relies on a
dedicated DomainCombiner to grant permissions only when needed only to the
libraries installed in the plugin folder
Add security checks for SpecialPermission/scripting and provides out of
the box permissions for the latest Hadoop 1.x (1.2.1) and 2.x (2.7.1)

* Testing
Uses a customized Local FS to perform actual integration testing of the
Hadoop stack (and thus to make sure the proper permissions and ACC blocks
are in place) however without requiring extra permissions for testing.
If needed, a MiniDFS cluster is provided (though it requires extra
permissions to bind ports)
Provides a RestIT test

* Build system
Picks the build system used in ES (still Gradle)
2015-12-14 21:50:09 +02:00
Martijn van Groningen aaacf096d2 Merge remote-tracking branch 'es/master' into feature/ingest 2015-12-14 11:57:10 +01:00
Jason Tedor 3383c24be0 Remove and forbid use of Collections#shuffle(List) and Random#<init>()
This commit removes and now forbids all uses of
Collections#shuffle(List) and Random#<init>() across the codebase. The
rationale for removing and forbidding these methods is to increase test
reproducibility. As these methods use non-reproducible seeds, production
code and tests that rely on these methods contribute to
non-reproducbility of tests.

Instead of Collections#shuffle(List) the method
Collections#shuffle(List, Random) can be used. All that is required then
is a reproducible source of randomness. Consequently, the utility class
Randomness has been added to assist in creating reproducible sources of
randomness.

Instead of Random#<init>(), Random#<init>(long) with a reproducible seed
or the aforementioned Randomess class can be used.

Closes #15287
2015-12-11 11:16:38 -05:00
Martijn van Groningen d38cccb8a1 Fix issues after merging in master 2015-12-11 14:51:45 +01:00
Martijn van Groningen 503a166b71 Merge remote-tracking branch 'es/master' into feature/ingest 2015-12-11 14:32:16 +01:00
Robert Muir 2741888498 Remove RuntimePermission("accessDeclaredMembers")
Upgrades lucene to 5.5.0-1719088, randomizedtesting to 2.3.2, and securemock to 1.2
2015-12-10 14:26:55 -05:00
Boaz Leskes fafeb3abdd Introduce a common base response class to all single doc write ops
IndexResponse, DeleteResponse and UpdateResponse share some logic. This can be unified to a single DocWriteResponse base class. On top, some replication actions are now not about write operations anymore. This commit renames ActionWriteResponse to ReplicationResponse

Last some toXContent is moved from the Rest layer to the actual response classes, for more code re-sharing.

Closes #15334
2015-12-10 15:14:02 +01:00
Jack Conradson da5b07ae13 Added a new scripting language (PlanA).
Closes #15136
2015-12-09 16:32:37 -08:00
David Pilato 414fccb7d1 Merge branch 'fix/15268-proxy-auth' 2015-12-09 23:21:00 +01:00
javanna a8382de09d add comment to clarifiy why metadata fields can be set all the time to IndexRequest in PipelineExecutionService 2015-12-09 18:52:43 +01:00
javanna a95f81c015 avoid stripping out _source prefix when in _ingest context 2015-12-09 18:42:20 +01:00
javanna 57d6971252 Streamline support for get/set/remove of metadata fields and ingest metadata fields
Unify metadata map and source, add also support for _ingest prefix. Depending on the prefix, either _source, nothing or _ingest, we will figure out which map to use for values retrieval, but also modifications.
2015-12-09 18:36:44 +01:00
javanna 744d2908a8 avoid null values in simulate serialization prototypes, use empty maps instead 2015-12-09 18:36:44 +01:00
javanna 6b7446beb9 Remove sourceModified flag from IngestDocument
If one is using the ingest plugin and providing a pipeline id with the request, the chance that the source is going to be modified is 99%. We shouldn't worry about keeping track of whether something changed. That seemed useful at first so we can save the resources for setting back the source (map to bytes) when not needed. Also, we are trying to unify metadata fields and source in the same map and that is going to complicate how we keep track of changes that happen in the source only. Best solution is to remove the flag.
2015-12-09 18:36:43 +01:00
javanna b0d7d604ff Add support for transient metadata to IngestDocument
IngestDocument now holds an additional map of transient metadata. The only field that gets added automatically is `timestamp`, which contains the timestamp of ingestion in ISO8601 format. In the future it will be possible to eventually add or modify these fields, which will not get indexed, but they will be available via templates to all of the processors.

Transient metadata will be visualized by the simulate api, although they will never get indexed. Moved WriteableIngestDocument to the simulate package as it's only used by simulate and it's now modelled for that specific usecase.

 Also taken the chance to remove one IngestDocument constructor used only for testing (accepting only a subset of es metadata fields). While doing that introduced some more randomizations to some existing processor tests.

Closes #15036
2015-12-09 18:36:01 +01:00
javanna 5bc1e46113 setFieldValue for list to replace when an index is specified
It used to do add instead, which is not consistent with the behaviour of set, which always replaces.
2015-12-09 18:28:07 +01:00
javanna 8240031216 Merge branch 'master' into feature/ingest 2015-12-09 18:14:32 +01:00
Simon Willnauer a49120bfc1 fix compilation 2015-12-09 12:26:28 +01:00
Simon Willnauer 85a1b54867 fix compilation 2015-12-09 11:41:14 +01:00
Martijn van Groningen 233de434a0 Merge pull request #15310 from martijnvg/ingest/stream_put_and_delete_responses
Streamline put & delete pipeline responses with index & delete responses
2015-12-09 11:30:57 +01:00
Simon Willnauer c9d7c92243 fold ClusterSettingsService into ClusterSettings 2015-12-09 09:57:39 +01:00
Britta Weber e0aa481bf5 Merge pull request #15213 from brwe/copy-to-in-multi-fields-exception
throw exception if a copy_to is within a multi field

Copy to within multi field is ignored from 2.0 on, see #10802.
Instead of just ignoring it, we should throw an exception if this
is found in the mapping when a mapping is added. For already
existing indices we should at least log a warning.
We remove the copy_to in any case.

related to #14946
2015-12-08 14:41:07 +01:00
Simon Willnauer fbbb04b87e Add infrastructure to transactionally apply and reset dynamic settings
This commit adds the infrastructure to make settings that are updateable
resetable and changes the application of updates to be transactional. This means
setting updates are either applied or not. If the application failes all values are rejected.

This initial commit converts all dynamic cluster settings to make use of the new infrastructure.
All cluster level dynamic settings are not resettable to their defaults or to the node level settings.
The infrastructure also allows to list default values and descriptions which is not fully implemented yet.

Values can be reset using a list of key or simple regular expressions. This has only been implemented on the java
layer yet. For instance to reset all recovery settings to their defaults a user can just specify `indices.recovery.*`.

This commit also adds strict settings validation, if a setting is unknown or if a setting can not be applied the entire
settings update request will fail.
2015-12-08 14:39:15 +01:00
Martijn van Groningen a2cda4e3f2 Streamline the put and delete pipelines responses with the index and delete response in core. 2015-12-08 14:01:28 +01:00
David Pilato 7dcb40bcac Add support for proxy authentication for s3 and ec2
When using S3 or EC2, it was possible to use a proxy to access EC2 or S3 API but username and password were not possible to be set.

This commit adds support for this. Also, to make all that consistent, proxy settings for both plugins have been renamed:

* from `cloud.aws.proxy_host` to `cloud.aws.proxy.host`
* from `cloud.aws.ec2.proxy_host` to `cloud.aws.ec2.proxy.host`
* from `cloud.aws.s3.proxy_host` to `cloud.aws.s3.proxy.host`
* from `cloud.aws.proxy_port` to `cloud.aws.proxy.port`
* from `cloud.aws.ec2.proxy_port` to `cloud.aws.ec2.proxy.port`
* from `cloud.aws.s3.proxy_port` to `cloud.aws.s3.proxy.port`

New settings are `proxy.username` and `proxy.password`.

```yml
cloud:
    aws:
        protocol: https
        proxy:
            host: proxy1.company.com
            port: 8083
            username: myself
            password: theBestPasswordEver!
```

You can also set different proxies for `ec2` and `s3`:

```yml
cloud:
    aws:
        s3:
            proxy:
                host: proxy1.company.com
                port: 8083
                username: myself1
                password: theBestPasswordEver1!
        ec2:
            proxy:
                host: proxy2.company.com
                port: 8083
                username: myself2
                password: theBestPasswordEver2!
```

Note that `password` is filtered with `SettingsFilter`.

We also fix a potential issue in S3 repository. We were supposed to accept key/secret either set under `cloud.aws` or `cloud.aws.s3` but the actual code never implemented that.

It was:

```java
account = settings.get("cloud.aws.access_key");
key = settings.get("cloud.aws.secret_key");
```

We replaced that by:

```java
String account = settings.get(CLOUD_S3.KEY, settings.get(CLOUD_AWS.KEY));
String key = settings.get(CLOUD_S3.SECRET, settings.get(CLOUD_AWS.SECRET));
```

Also, we extract all settings for S3 in `AwsS3Service` as it's already the case for `AwsEc2Service` class.

Closes #15268.
2015-12-07 23:10:54 +01:00
Tal Levy e6b287c083 Merge pull request #15133 from talevy/the_one_true_field
[Ingest] Have processors operate one field at time.
2015-12-07 08:31:08 -08:00
Tal Levy 45f48ac126 update all processors to only operate on one field at a time when possible 2015-12-07 08:30:00 -08:00
Martijn van Groningen 6062c4eac9 Merge branch 'master' into feature/ingest 2015-12-07 15:53:39 +01:00
Robert Muir 2169a123a5 Filter classes loaded by scripts
Since 2.2 we run all scripts with minimal privileges, similar to applets in your browser.
The problem is, they have unrestricted access to other things they can muck with (ES, JDK, whatever).
So they can still easily do tons of bad things

This PR restricts what classes scripts can load via the classloader mechanism, to make life more difficult.
The "standard" list was populated from the old list used for the groovy sandbox: though
a few more were needed for tests to pass (java.lang.String, java.util.Iterator, nothing scary there).

Additionally, each scripting engine typically needs permissions to some runtime stuff.
That is the downside of this "good old classloader" approach, but I like the transparency and simplicity,
and I don't want to waste my time with any feature provided by the engine itself for this, I don't trust them.

This is not perfect and the engines are not perfect but you gotta start somewhere. For expert users that
need to tweak the permissions, we already support that via the standard java security configuration files, the
specification is simple, supports wildcards, etc (though we do not use them ourselves).
2015-12-05 21:46:52 -05:00
Robert Muir 46377778a9 Merge branch 'master' into getClassLoader 2015-12-04 15:58:36 -05:00
Robert Muir b0c64910b0 ban RuntimePermission("getClassLoader")
this gives more isolation between modules and plugins.
2015-12-04 15:58:02 -05:00
Ryan Ernst 01d48e2062 Merge branch 'master' into jigsaw 2015-12-04 11:29:49 -08:00
David Pilato 619fb998e8 Update Azure Service Management API to 0.9.0
Azure team released new versions of their Java SDK.

According to https://github.com/Azure/azure-sdk-for-java/wiki/Azure-SDK-for-Java-Features, it comes with 2 versions.
We should at least update to `0.9.0` of V1 but also consider moving to the new APIs (V2).

This commit first updates to latest API V1.

```xml
<dependency>
    <groupId>com.microsoft.azure</groupId>
    <artifactId>azure-svc-mgmt-compute</artifactId>
    <version>0.9.0</version>
</dependency>
```

Closes #15209
2015-12-04 17:32:11 +01:00
javanna d7c3b51b9c [TEST] adapt to upstream changes 2015-12-04 16:35:53 +01:00
javanna 73986cc54f adapt to upstream changes 2015-12-04 14:17:07 +01:00
javanna 6c43137413 Merge branch 'master' into feature/ingest 2015-12-04 14:10:01 +01:00
Adrien Grand 3f86adddbf Remove MergeMappingException.
Failures to merge a mapping can either come as a MergeMappingException if they
come from Mapper.merge or as an IllegalArgumentException if they come from
FieldTypeLookup.checkCompatibility. I think we should settle on one: this pull
request replaces all usage of MergeMappingException with
IllegalArgumentException.
2015-12-04 12:56:26 +01:00
Ryan Ernst 0a4a81afaf Added modules, distributions now include them (just plugins installed in
a diff dir)
2015-12-03 14:18:26 -08:00
Tal Levy ffa8998f36 Merge pull request #15181 from martijnvg/ingest_geoip_only_read_mmdb_files
[Ingest] The geoip processor should only try to read *.mmdb files from the geoip config directory
2015-12-03 12:17:09 -08:00
Tal Levy 56da7b32ed add ability to define custom grok patterns within processor config 2015-12-03 08:24:07 -08:00
Tal Levy cf1c393d70 Merge pull request #15166 from talevy/remove_pattern_utils
move PatternUtils#loadBankFromStream into GrokProcessor.Factory
2015-12-03 08:08:00 -08:00
Martijn van Groningen 6acf8ec263 Removed pipeline tests with a simpler tests
The PipelineTests tried to test if the configured map/list in set processor wasn't modified while documents were ingested. Creating a pipeline programmatically created more noise than the test needed. The new tests in IngestDocumentTests have the same goal, but is much smaller and clearer by directly testing against IngestDocument.
2015-12-03 15:19:04 +01:00
Jason Tedor fbe736c9bb Cleaner type-inference assistance 2015-12-02 10:49:35 -05:00
Jason Tedor 05430a788a Remove and forbid use of the type-unsafe empty Collections fields
This commit removes and now forbids all uses of the type-unsafe empty
Collections fields Collections#EMPTY_LIST, Collections#EMPTY_MAP, and
Collections#EMPTY_SET. The type-safe methods Collections#emptyList,
Collections#emptyMap, and Collections#emptySet should be used instead.
2015-12-02 10:41:59 -05:00
Martijn van Groningen 9ab765b851 The geoip processor should only try to read *.mmdb files from the geoip config directory 2015-12-02 14:38:49 +01:00
Martijn van Groningen a9ecde041b Merge branch 'master' into feature/ingest 2015-12-02 11:21:15 +01:00
Martijn van Groningen 270a3977bc Removed the lazy cache in DatabaseReaderService and eagerly build all available databases. 2015-12-02 11:16:02 +01:00
David Pilato d23d8a891f Remove "empty" licenses dir
Follow up #15168

We don't need to have "fake" licenses dir anymore.
2015-12-02 10:22:52 +01:00
Ryan Ernst d68c6673a2 Build: Cleanup precommit task gradle code
This change attempts to simplify the gradle tasks for precommit. One
major part of that is using a "less groovy style", as well as being more
consistent about how tasks are created and where they are configured. It
also allows the things creating the tasks to set up inter task
dependencies, instead of assuming them (ie decoupling from tasks
eleswhere in the build).
2015-12-01 22:36:54 -08:00
Tal Levy 767bd1d4d5 move PatternUtils#loadBankFromStream into GrokProcessor.Factory 2015-12-01 15:46:02 -08:00
javanna 6c0510b01d Make rename processor less error prone
Rename processor now checks whether the field to rename exists and throws exception if it doesn't. It also checks that the new field to rename to doesn't exist yet, and throws exception otherwise. Also we make sure that the rename operation is atomic, otherwise things may break between the remove and the set and we'd leave the document in an inconsistent state.

Note that the requirement for the new field name to not exist simplifies the usecase for e.g. { "rename" : { "list.1": "list.2"} } as such a rename wouldn't be accepted if list is actually a list given that either list.2 already exists or the index is out of bounds for the existing list. If one really wants to replace an existing field, that field needs to be removed first through remove processor and then rename can be used.
2015-12-01 19:58:24 +01:00
Martijn van Groningen 15b6708a5d and now make use of the lifecycle infrastructure 2015-12-01 18:20:25 +01:00
Tal Levy 8e4c288b5c Merge pull request #15132 from talevy/no_match_for_grok
[Ingest] No match for grok
2015-12-01 09:13:11 -08:00
Tal Levy 2c1effdd41 throw exception when grok processor does not match 2015-12-01 08:58:58 -08:00
Martijn van Groningen 9dd52ad7d3 Removed pollution from the Processor.Factory interface.
1) It no longer extends from Closeable.
2) Removed the config directory setter. Implementation that relied on it, now get the location to the config dir via their constructors.
2015-12-01 17:32:37 +01:00
Martijn van Groningen fa9fcb3b11 geo processor should add a list of doubles instead of an array to the ingest document 2015-12-01 17:12:34 +01:00
Martijn van Groningen 99a4295330 If a list or map value gets set on ingest document a deep copy needs to be made.
If this is not done this can lead to processor configuration being changed by an bulk or index request.
2015-12-01 16:02:04 +01:00
javanna c67a332486 Query DSL: Enforce distance is greater than 0 in geo distance query
Validation is not done as part of the distance setter method and tested in GeoDistanceQueryBuilderTests. Fixed GeoDistanceTests to adapt to the new validation.

Closes #15135
2015-12-01 14:07:32 +01:00
Martijn van Groningen 4402da1af0 also change the tests to deal with Exception instead of IOException 2015-11-30 15:45:40 +01:00
Martijn van Groningen dde274d944 Replaced IOException with Exception on factory implementations' `Processor.Factory#create(Map)` method. 2015-11-30 15:37:16 +01:00
Britta Weber d8a1a4bd43 fix toXContent() for mapper attachments field
We must use simpleName() instead of name() because otherwise when the mapping
is generated as a string the field name will be the full path with dots
and that is illegal from es 2.0 on.

closes https://github.com/elastic/elasticsearch-mapper-attachments/issues/169
2015-11-30 15:28:12 +01:00
Martijn van Groningen fdf4543b8e Renamed `add` processor to `set` processor.
This name makes more sense, because if a field already exists it overwrites it.
2015-11-30 15:03:20 +01:00
javanna 43b861b076 IngestDocument to support accessing and modifying list items
When reading, through #getFieldValue and #hasField, and a list is encountered, the next element in the path is treated as the index of the item that the path points to (e.g. `list.0.key`). If the index is not a number or out of bounds, an exception gets thrown.

Added #appendFieldValue method that has the same behaviour as setFieldValue, but when a list is the last element in the path, instead of replacing the whole list it will simply add a new element to the existing list. This method is currently unused, we have to decide whether the set processor or a new processor should use it.

A few other changes made:
- Renamed hasFieldValue to hasField, as this method is not really about values but only keys. It will return true if a key is there but its value is null, while it returns false only when a field is not there at all.
- Changed null semantic in getFieldValue. null gets returned only when it was an actual value in the source, an exception is thrown otherwise when trying to access a non existing field, so that null != field not present.
- Made remove stricter about non existing fields. Throws error when trying to remove a non existing field. This is more consistent with the other methods in IngestDocument which are strict about fields that are not present.

Relates to #14324
2015-11-30 13:58:03 +01:00
Jim Ferenczi e182072b6f Merge pull request #15017 from jimferenczi/fields_option
Refuse to load fields from _source when using the `fields` option and support wildcards.
2015-11-30 11:01:21 +01:00
Jim Ferenczi 731833cfc6 Fixes #14489
Do not to load fields from _source when using the `fields` option.
  Non stored (non existing) fields are ignored by the fields visitor when using the `fields` option.

Fixes #10783
  Support * wildcard to retrieve stored fields when using the `fields` option.
  Supported pattern styles are "xxx*", "*xxx", "*xxx*" and "xxx*yyy".
2015-11-30 11:00:32 +01:00
Martijn van Groningen 467a47670c Merge remote-tracking branch 'es/master' into feature/ingest 2015-11-30 10:24:27 +01:00
Robert Muir a2816ec574 don't assert exact expected lengths for documents.
This will change depending on newline of the operating system
2015-11-29 11:48:54 -05:00
Robert Muir 415c37340a do not assert charset for mapper-attachments tests.
Its enough to test the content type for what we are testing.

Currently tests are flaky if charset is detected as e.g. windows-1252 vs iso-8859-1 and so on.
In fact, they fail on windows 100% of the time.

We are not trying to test charset detection heuristics (which might be different even due to newlines in tests or other things).
If we want to do test that, we should test it separately.
2015-11-29 11:40:27 -05:00
javanna b4b698f653 Merge branch 'master' into feature/ingest 2015-11-28 11:15:25 +01:00
Simon Willnauer 65b661b1f4 [TEST] Fix MapperUpgrade tests to use a dedicated master to ensure dangeling index import works predictably
When importing dangling indices on a single node that is data and master eligable the async dangling index
call can still be in-flight when the cluster is checked for green / yellow. Adding a dedicated master node
and a data only node that does the importing fixes this issus just like we do in OldIndexBackwardsCompatibilityIT
2015-11-27 10:32:21 +01:00
Martijn van Groningen 0fe1b4eab1 PipelineStore no longer is a lifecycle component
Client in PipelineStore gets provided via a guice provider
Processor and Factory throw Exception instead of IOException
Removed PipelineExecutionService.Listener with ActionListener
2015-11-26 18:12:15 +01:00
Martijn van Groningen 9aff8c6352 fix compile errors 2015-11-26 17:00:16 +01:00
Martijn van Groningen 9d1fa0d6da ingest: Add `meta` processor that allows to modify the metadata attributes of document being processed 2015-11-26 15:46:32 +01:00
Martijn van Groningen afc9069c99 * Inlined PipelineStoreClient class into the PipelineStore class
* Moved PipelineReference to a top level class and named it PipelineDefinition
* Pulled some logic from the crud transport classes to the PipelineStore
* Use IOUtils#close(...) where appropriate
2015-11-26 14:42:39 +01:00
Simon Willnauer 9f6598b18d Fix compile errors 2015-11-26 13:41:00 +01:00
javanna 1a7391070f Simulate api improvements
Move ParsedSimulateRequest to SimulatePipelineRequest and remove Parser class in favor of static parse methods.
Simplified execute methods in SimulateExecutionService.
2015-11-26 13:24:45 +01:00
Martijn van Groningen 2890432421 made updatePipelines() to not make it prone to race conditions 2015-11-25 18:32:56 +01:00
javanna 5d510b59c8 use MetaData enum for metadata field names
Also rename getName to getFieldName in MetaData to prevent confusion with name() enum method.
2015-11-25 18:08:53 +01:00
javanna ec162c458e Replace property with field in IngestDocument
getPropertyValue => getFieldValue
hasPropertyValue => hasFieldValue
setPropertyValue => setFieldValue
removeProperty => removeField
2015-11-25 18:08:53 +01:00
Luca Cavanna e15fa99ee3 Merge pull request #15019 from javanna/enhancement/java8_date_parser
date formats: use a function instead of our own interface
2015-11-25 16:17:39 +01:00
javanna c4cf55c196 [TEST] generate random timezone out of the available ones in joda 2015-11-25 15:58:01 +01:00
javanna 5daa73b350 date formats: use a function instead of our own interface
Also turn the different date formats into an enum.
2015-11-25 15:37:35 +01:00
javanna 4759a6e50f Merge branch 'master' into feature/ingest 2015-11-25 14:59:10 +01:00
Adrien Grand e8520bf519 Tests: For single data path for *FieldMapperUpgradeTests. 2015-11-25 11:46:19 +01:00
javanna e0fcee642e [TEST] fix locale comparison 2015-11-25 10:26:46 +01:00
javanna 388e637fa9 add a few more asserts to IngestActionFilterTests 2015-11-24 19:43:54 +01:00
Adrien Grand aad84395c9 Add a test that upgrades succeed even if a mapping contains fields that come from a plugin. 2015-11-24 19:14:19 +01:00
javanna 49bfe6410e Rename Data leftovers 2015-11-24 15:49:25 +01:00
javanna 8f1f5d4da0 Split mutate processor into one processor per function 2015-11-24 14:31:53 +01:00
Martijn van Groningen 1e9d5c7b22 test: also test what happens if all index requests fail to be processed by the pipeline 2015-11-24 13:41:40 +01:00
Martijn van Groningen 8b1f117e51 Instead of failing the entire bulk request if the pipeline fails, only fail a bulk item. 2015-11-24 12:40:30 +01:00
javanna eeb51ce8d0 Merge branch 'master' into feature/ingest 2015-11-24 10:23:53 +01:00
Adrien Grand 5f33fbdb75 Register field mappers at the node level.
This moves the registration of field mappers from the index level to the node
level and also ensures that mappers coming from plugins are treated no
differently from core mappers.
2015-11-24 08:59:37 +01:00
Michael McCandless e13b0d4bde upgrade lucene to 5.4.0-snapshot-1715952 2015-11-23 17:13:49 -05:00
Martijn van Groningen ecc8158b89 renamed Data to IngestDocument
moved all metadata related fields to a single metadata map
removed specific metadata getters with a generic getMetadata()
2015-11-23 18:24:42 +01:00
David Pilato 5b0e2823b1 Merge branch 'docs/mapper-attachments' 2015-11-23 12:14:31 +01:00
javanna 36655b688c Merge branch 'master' into feature/ingest 2015-11-23 10:05:17 +01:00
Robert Muir 30529c008d Ban write access to system properties
* Forbid System.setProperties & co in forbidden APIs.
* Ban property write access at runtime with security manager.

Plugins that need to modify system properties will need to request permission in their plugin-security.policy
2015-11-21 22:33:06 -05:00
Ryan Ernst 88ee2e23a6 Merge branch 'master' into javadocs 2015-11-20 12:20:59 -08:00
markharwood c3a50d7ca2 Search: shift SearchTimeoutTests into core tests minus the Groovy dependency and renamed with IT test suffix 2015-11-20 18:50:17 +00:00
markharwood 5a5f05a0e9 Aggregations: Shift significant terms score test back into core without the Groovy dependency 2015-11-20 18:32:04 +00:00
javanna bb298ed27a add some javadocs to Data#getDocument 2015-11-20 18:01:27 +01:00
javanna be3e913349 set Data modified flag when a property is removed and improve behaviour when adding a field 2015-11-20 17:59:45 +01:00
Colin Goodheart-Smithe 5a18f740ba Aggregations: Moves SumTests out of lang-groovy and back into core
Renames SumTests to SumIT and puts it in the same package as the other aggregation tests. Also updates the tests to not require Groovy
2015-11-20 12:42:08 +00:00
Daniel Mitterdorfer c95c142829 Merge remote-tracking branch 'danielmitterdorfer/fix/ValueCountTests' 2015-11-20 13:21:15 +01:00
Daniel Mitterdorfer 7d6b0fc481 Move ValueCountTests to core
This script moves ValueCountTests from plugins back to core
by using a mock script engine instead of Groovy.
2015-11-20 13:19:55 +01:00
Adrien Grand 4602b8177b Move AvgTests back to core.
This makes AvgTests use a mock plugin engine. I also removed the
textScriptExplicit* methods for the base class since they only make sense for
a groovy script, not a mock script.
2015-11-20 11:58:38 +01:00
Ryan Ernst c3cb1fd08c Merge branch 'master' into javadocs 2015-11-19 10:43:43 -08:00
David Pilato 149279fbee Filter cloud azure credentials
Bug introduced in #13779: we don't filter anymore credentials because we were filtering `cloud.azure.storage.account` and `cloud.azure.storage.key` but now credentials are like `cloud.azure.storage.XXX.account` and `cloud.azure.storage.XXX.key` where `XXX` can be a storage setting id.

Closes #14843.
2015-11-19 15:40:56 +01:00
David Pilato 33b0e662cc Merge branch 'pr/phonetic_daitch_mokotoff' 2015-11-19 14:48:11 +01:00
David Pilato d48d8ef863 Merge branch 's3-storage-class-support' of https://github.com/socialrank/elasticsearch into socialrank-s3-storage-class-support
# Please enter a commit message to explain why this merge is necessary,
# especially if it merges an updated upstream into a topic branch.
#
# Lines starting with '#' will be ignored, and an empty message aborts
# the commit.
2015-11-19 13:10:08 +01:00
Nicholas Knize 0d349854d3 Add CONTAINS relation to geo_shape query
At the time of geo_shape query conception, CONTAINS was not yet a supported spatial operation in Lucene. Since it is now available this commit adds ShapeRelation.CONTAINS to GeoShapeQuery. Randomized testing is included and documentation is updated.
2015-11-18 14:15:35 -06:00
David Pilato 052285ac88 Revert 3cacec4468
We actually want to keep the test when using deprecated setting in 3.0.
We will keep this setting deprecated as well so people will be able to update in a smoother way.

Also add the deprecating information to the migration documentation.
2015-11-18 19:14:26 +01:00
javanna 404ae395ca add javadocs for Data#removeProperty 2015-11-18 18:00:25 +01:00
javanna 59868cd02e add support for removeProperty 2015-11-18 17:28:37 +01:00
David Pilato 3cacec4468 Fix test in Azure
Related to #13656.

We need to remove deprecated settings in another PR.
2015-11-18 17:14:10 +01:00
David Pilato dc17ab31d3 Merge remote-tracking branch 'origin/master' 2015-11-18 16:37:02 +01:00
Christoph Büscher 3e483b5626 Merge pull request #14587 from cbuescher/move-datehistogramtests-core
Move DateHistogramTests back to core module
2015-11-18 16:35:59 +01:00
craigwi 79a4d9ce36 Add support for secondary azure storage account
Follow up for #13228.

This commit adds support for a secondary storage account:

```yml
cloud:
    azure:
        storage:
            my_account1:
                account: your_azure_storage_account1
                key: your_azure_storage_key1
                default: true
            my_account2:
                account: your_azure_storage_account2
                key: your_azure_storage_key2
```

When creating a repository, you can choose which azure account you want to use for it:

```sh
curl -XPUT localhost:9200/_snapshot/my_backup1?pretty -d '{
  "type": "azure"
}'

curl -XPUT localhost:9200/_snapshot/my_backup2?pretty -d '{
  "type": "azure",
  "settings": {
    "account" : "my_account2",
    "location_mode": "secondary_only"
  }
}'
```

`location_mode` supports `primary_only` or `secondary_only`. Defaults to `primary_only`. Note that if you set it
to `secondary_only`, it will force `read_only` to true.
2015-11-18 16:31:48 +01:00