Commit Graph

344 Commits

Author SHA1 Message Date
Alex Ksikes afe9c52f07 XMoreLikeThis: fix potentially unpositioned enum 2015-07-06 18:08:06 +02:00
Robert Muir d7dab0df72 fix test to allow integration tests 2015-07-06 11:20:30 -04:00
Robert Muir 3f4b8df00d Merge pull request #12026 from rmuir/integ_tests
add integration test harness to maven build
2015-07-06 10:16:54 -04:00
Alex Ksikes 8ef1708395 [TEST] fix testMoreLikeThisArtificialDocs
Ensure that the indexed doc hasn't all its fields empty and that the
artificial doc requested is always routed to the shard having that doc.
2015-07-06 15:27:38 +02:00
Colin Goodheart-Smithe b93512e000 [TEST] NettyTransportMultiPortTests now checks selected random ports are free before using them 2015-07-06 12:04:22 +01:00
markharwood f95981b977 Search enhancement: adds a special case for the “_index” field which allows queries to test the name of the index. Adding this capability means we can deprecate the specialist indices query.
IndexFieldMapper is changed to make the term query factories produce match_all or match_none queries based on tests on the index name.

Closes #3316
2015-07-06 10:55:28 +01:00
javanna d085088793 BoolQueryBuilder#hasClauses to take into account filterClauses too 2015-07-06 11:02:09 +02:00
Boaz Leskes d74eac94f2 ZenDiscovery: #11960 failed to remove eager reroute from node join
It is removed now :)

Closes #12019
2015-07-03 22:06:42 +02:00
Robert Muir 9d233aeaf0 use external test cluster for integration tests 2015-07-03 12:20:35 -04:00
Tanguy Leroux 9495816cb7 Remove sigar completely 2015-07-03 15:49:17 +02:00
Simon Willnauer ac32f3d310 Don't special-case on ElasticsearchWrapperException in toXContent
the specialization can cause stack overflows if an exception is a
ElasticsearchWrapperException as well as a ElasticsearchException.
This commit just relies on the unwrap logic now to find the cause and only
renders if we the rendering exception is the cause otherwise forwards
to the generic exception rendering.

Closes #11994
2015-07-03 13:51:11 +02:00
Ryan Ernst e83ae64ea9 Mappings: Completely move doc values and fielddata settings to field types
While MappedFieldType contains settings for doc values and fielddata,
AbstractFieldMapper had special logic in its constructor that
required setting these on the field type from there. This change
removes those settings from the AbstractFieldMapper constructor.
As a result, defaultDocValues(), defaultFieldType() and
defaultFieldDataType() are no longer needed.
2015-07-03 03:10:20 -07:00
Martijn van Groningen 13602a591f Removed forcedDateParser from hashCode() and equals() as it holds no value when it comes to equality. 2015-07-03 10:04:22 +02:00
Simon Willnauer 20d0b4f446 Promote headers to first class citizens on exceptions
This commit merges the pre-existing special exception that
allowed to associate headers with exceptions and the elasticsaerch
base class `ElasticsearchException` This allows for more generic use
of exceptions where plugins can associate meta-data with any elasticsearch
base exception to control behavior etc.

This also addds a generic SecurityException to allow plugins to pass on
information based on the RestStatus.
2015-07-03 08:58:44 +02:00
Robert Muir 80871bae2b Add simple integ testing infra 2015-07-03 02:12:01 -04:00
Boaz Leskes 3084bed194 Test: force local transport in ZenDiscoveryTests.testHandleNodeJoin_incompatibleMinVersion
Otherwise we ignore the join request due to a transport address mismatch instead of failing it.
2015-07-02 22:20:12 +02:00
Robert Muir 3171f52461 Merge pull request #12001 from rmuir/no_dont_do_that
Don't add CWD to classpath when ES_CLASSPATH isn't set.
2015-07-02 14:53:24 -04:00
Martijn van Groningen a4b99e6291 zen: Don't join master nodes or accept join requests of old and too new nodes.
If the version of a node is lower than the minimum supported version or higher than the maximum supported version, a node shouldn't be allowed to join and nodes should join that elected master node

Closes #11924
2015-07-02 20:51:21 +02:00
Ryan Ernst 0f62950856 fix compile error after bad merge 2015-07-02 10:53:30 -07:00
Ryan Ernst e12bb4a54a Merge pull request #11977 from rjernst/pr/mapper-names
Move short name access out of field type
2015-07-02 10:38:30 -07:00
Robert Muir c598c4cc97 blind stab at windows 2015-07-02 13:27:39 -04:00
Robert Muir 68be95ca1a Don't add CWD to classpath when ES_CLASSPATH isn't set. 2015-07-02 13:16:20 -04:00
javanna 90f32d4d32 Internal: make sure ParseField is always used in combination with parse flags
Removed ParseField#match variant that accepts the field name only, without parse flags. Such a method is harmful as it defaults to empty parse flags, meaning that no deprecation exceptions will be thrown in strict mode, which defeats the purpose of using ParseField. Unfortunately such a method was used in a lot of places were the parse flags weren't easily accessible (outside of query parsing), and in a lot of other places just by mistake.

Parse flags have been introduced now as part of SearchContext and mappers where needed. There are a few places (e.g. java api requests) where it is not possible to retrieve them as they depend on the index settings, in that case we explicitly pass in EMPTY_FLAGS for now, but this has to be seen as an exception.

Closes #11859
2015-07-02 14:30:01 +02:00
Simon Willnauer 753376dba7 [TEST] Remove timeout - it's not needed in this test 2015-07-02 13:11:00 +02:00
Colin Goodheart-Smithe 11e851eb0c Change TODO comments for removing old Script API to 3.0 2015-07-02 11:20:17 +01:00
Adrien Grand 6765635067 Merge pull request #11988 from jpountz/fix/zlib_skip_underlying
Transport: Do not make the buffer skip while a stream is open.
2015-07-02 10:36:11 +02:00
Britta Weber 6bf543dca8 [TEST] mute IndicesStoreIntegrationTests.indexCleanup 2015-07-02 10:24:06 +02:00
Adrien Grand 7482e1362f Transport: Do not make the buffer skip while a stream is open.
This commit changes MessageChannelHandler to not skip the underlying
ChannelBuffer while a StreamInput is open on top of it. In case eg. compression
is enabled, this prevents failures due to the fact that the decompressed
stream input expects a certain structure that it can't verify if the position
of the underlying buffer is changed.
2015-07-02 09:30:28 +02:00
Robert Muir ce900b4d7a Merge pull request #11982 from rmuir/grr_bad_defaults
really ban exitVM with security policy
2015-07-01 17:58:07 -04:00
uboness 6021bd8cca Parameterized exception messages
Added dynamic arguments to `ElasticsearchException`, `ElasticsearchParseException` and `ElasticsearchTimeoutException`.

This helps keeping the exception messages clean and readable and promotes consistency around wrapping dynamic args with `[` and `]`.

This is just the start, we need to propagate this to all exceptions deriving from `ElasticsearchException`. Also, work started on standardizing on lower case logging & exception messages. We need to be consistent here...

 - Uses the same `LoggerMessageFormat` as used by our logging infrastructure.
2015-07-01 23:08:38 +02:00
Robert Muir 5aa5c98826 really ban exitVM with security policy 2015-07-01 15:40:46 -04:00
Martijn van Groningen 53874bf5a6 aliases: Parse aliases at search time and never cache parsed alias filters
The work around for resolving `now` doesn't need to be used for aliases, becuase alias filters are parsed at search time. However it can't be removed, because the percolator relies on it.

Parent/child can be specified again in alias filters, this now works again because alias filters are parsed at search time. Parent/child will also use the late query parse work around, to make sure to do the final preparations when the search context is around. This allows the aliases api to validate the parent/child queries without failing because there is no search context.

Closes #10485
2015-07-01 21:20:54 +02:00
Robert Muir 6b4c51f442 Merge pull request #11979 from rmuir/nosystem
Don't jarhell check system jars
2015-07-01 14:53:20 -04:00
Robert Muir 976ace21e0 let IDEs have jar hell in tests 2015-07-01 14:29:57 -04:00
Robert Muir 41ac191cac Add java.home to debug logging 2015-07-01 14:06:53 -04:00
Robert Muir 30f267f758 Don't jarhell check system jars 2015-07-01 13:26:00 -04:00
Martijn van Groningen 5a2a2861ab Merge pull request #11954 from martijnvg/percolator/stats/typo
Change `percolator.getTime` -> `percolator.time`
2015-07-01 19:08:42 +02:00
Ryan Ernst 41dc1b0120 Mappings: Move short name access out of field type
Eventually, the field type should not need any names, because there
will be only one name which leads to finding it (the full name, which is
also the index name). However, the short or "simple" name (using java
terminology for class names) is needed just in a couple places, for
serialization.

This change moves the simple name out of MappedFieldType.Names, into
Mapper, and makes Mapper and FieldMapper abstract classes.
2015-07-01 09:46:52 -07:00
Ryan Ernst ac8896c10e Merge pull request #11962 from rjernst/pr/root-mappers-rename
Rename "root" mappers to "metadata" mappers
2015-07-01 09:13:14 -07:00
Robert Muir b55a3f32b8 Add debug logging in case classpath is crazy 2015-07-01 10:04:36 -04:00
Simon Willnauer 93acb98bd6 Carry on rest status if exceptions are not serializable
Today we loose the RestStatus code for non-serializable exceptions.
This can be tricky if they are supposed to signal certain situations
like authentication errors etc. This commit adds support for carrying on
the exceptions in the NotSerializableExceptoinWrapper
2015-07-01 13:57:54 +02:00
Colin Goodheart-Smithe e366d0380d Aggregations: Adds other bucket to filters aggregation
The filters aggregation now has an option to add an 'other' bucket which will, when turned on, contain all documents which do not match any of the defined filters. There is also an option to change the name of the 'other' bucket from the default of '_other_'

Closes #11289
2015-07-01 10:44:04 +01:00
Colin Goodheart-Smithe ab80130c10 Aggregations: Makes SKIP Gap Policy work correctly for Bucket Script aggregation
This change means that when the skip gap policy is used, the bucket script aggregation will skip executing the script on a bucket if any of the required bucket_paths are missing for the bucket. No aggregation will be added to the bucket, and the aggregation will move to the next bucket.
2015-07-01 10:18:18 +01:00
Tanguy Leroux 737440b580 Postrm script should not fail
This commit changes the postrm script so that it prints error messages instead of failing & exiting when the deletion of a directory failed while removing a RPM/DEB package.

Closes #11373
2015-07-01 11:03:38 +02:00
Colin Goodheart-Smithe d73ce3492a Aggregations: Makes ValueFormat and ValueFormatter never null
This allows a lot of null checks to be removed where we were always falling back to the ValueFormat.RAW anyway. Now the format is set to ValueFormat.RAW when no alternative is suitable.

Closes #10594
2015-07-01 08:57:03 +01:00
szroland 68d658a30f Recovery: Fix wrong reused file bytes in Recovery API reports
Simple snapshot.restore test case that also exposes the bug mentioned in #11876

Fix #11876
Closes #11965
2015-07-01 09:08:53 +02:00
Martijn van Groningen ef9d70b9b3 field stats: added index constraints
Field stats index constraints allows to omit all field stats for indices that don't match with the constraint. An index
constraint can exclude indices' field stats based on the `min_value` and `max_value` statistic. This option is only
useful if the `level` option is set to `indices`.

For example index constraints can be useful to find out the min and max value of a particular property of your data in
a time based scenario. The following request only returns field stats for the `answer_count` property for indices
holding questions created in the year 2014:

curl -XPOST 'http://localhost:9200/_field_stats?level=indices' -d '{
   "fields" : ["answer_count"] <1>
   "index_constraints" : { <2>
      "creation_date" : { <3>
         "min_value" : { <4>
            "gte" : "2014-01-01T00:00:00.000Z",
         },
         "max_value" : {
            "lt" : "2015-01-01T00:00:00.000Z"
         }
      }
   }
}'

Closes #11187
2015-07-01 08:47:03 +02:00
Boaz Leskes 3cf78669f2 Mute testInstallPluginWithElasticsearchDownloadService & testInstallPluginWithMavenCentral
Example failure: http://build-us-00.elastic.co/job/es_core_master_oracle_6/519/testReport/
2015-07-01 08:34:47 +02:00
Robert Muir 5294b1ae40 Merge pull request #11963 from rmuir/prevent_hell
detect jar hell before installing a plugin
2015-07-01 00:00:12 -04:00
Robert Muir f312a4e30d cleanup uwe's unzip code 2015-06-30 23:59:45 -04:00
Igor Motov 2a9f26dfdc Improve repository verification failure message
Closes #11922
2015-06-30 20:08:15 -04:00
Simon Willnauer 04631d6948 Carry on headers if unknow exception is a WithRestHeadersException 2015-06-30 23:02:45 +02:00
Simon Willnauer c6837f56d9 Serialize AlreadyClosedException first before it's superclass 2015-06-30 23:02:45 +02:00
Robert Muir a839644826 detect jar hell before installing a plugin 2015-06-30 16:42:01 -04:00
Ryan Ernst 4b3dba527b Mappings: Rename "root" mappers to "metadata" mappers
"Root" is a very confusing term for meta field mappers. This change
renames "RootMapper" to "MetadataFieldMapper" and simplifies
how metadata mappers are setup.

It also requires that metadata mappers are now a FieldMapper
(MetadataFieldMapper extends from AbstractFieldMapper). The only
use of a root mapper that wasn't a field mapper was the theoretical
"external" root mapper (just a test mapper). But it doesn't make
sense to not have an actual field, and this falls inline with
the hopefully eventual collapsing of AbstractFieldMapper/FieldMapper/Mapper.
2015-06-30 13:31:17 -07:00
Boaz Leskes 182c59f5b4 Discovery: reroute after node join is processed
- shard listing actions underpinning shard allocation do not have access to that new node yet (causing errors during shard allocation see #11923
 - the very first cluster state published to a node already has shard assignments to it. This surfaced other issues we are working to fix separately

 This commit changes the reroute to be done post processing the initial join cluster state to side step these issues while we work on a longer term solution.

Closes #11960
2015-06-30 22:09:29 +02:00
Simon Willnauer dab2a76eae Cut over to writeable for TransportAddress
This will allow us to move away from reflection hacks to serialize
transport exceptions.
2015-06-30 20:59:47 +02:00
Martijn van Groningen c6ae6fc6d9 percolator: `getTime` -> `time` 2015-06-30 18:44:58 +02:00
Colin Goodheart-Smithe d9ab3cba77 Search Templates: Adds API endpoint to render search templates as a response
Closes #6821
2015-06-30 16:57:23 +01:00
Simon Willnauer bbaf4710cb FileName is optional in StacktraceElement 2015-06-30 15:51:48 +02:00
Simon Willnauer 0ac8c1bc55 Carry on stacktrace and suppressed exceptions if exception is not serializable 2015-06-30 14:51:44 +02:00
Simon Willnauer 6ee9a3d5f2 add more essential exceptions 2015-06-30 14:51:44 +02:00
Simon Willnauer 971ac0475e add IOOB to supported exceptions 2015-06-30 14:51:44 +02:00
Simon Willnauer 0148c462e0 remove leftover ObjectOutputStream references 2015-06-30 14:51:44 +02:00
Simon Willnauer 325cc6bc61 serialize as not-serializable if exception is not registered 2015-06-30 14:51:44 +02:00
Simon Willnauer 4e2621ca21 Cleanup non-serializable code 2015-06-30 14:51:44 +02:00
Simon Willnauer c4f15c2c91 add dedicated unittests for exceptions that have a writeTo method (custom serialization) 2015-06-30 14:51:44 +02:00
Simon Willnauer 62d29ff6a4 simplify exception reading / writing - now use a named exception for unknonw exceptions 2015-06-30 14:51:43 +02:00
Simon Willnauer fa016a2b09 Don't implement Serializable in core classes
We can't add this to forbidden API since literally everything implements this.
Any API that would for instance use a String would not compile anymore.
2015-06-30 14:51:43 +02:00
Simon Willnauer e7eb9cf4de Ban java serialization
We had several problems with Java Serializatin in the past. At some point
in the Java 1.7.x series JDKs where not compatible anymore when java
serialization (ObjectStream) was used to exchange objects. In elasticsearch
we used this to serialize exceptions across the wire which caused several problems
with incompatible JDKs. While causing lot of trouble this essentially prevented
users from moving forward and upgrade their JVMs. To prevent these kind of issues
this commit removes the dependency on java serialization entirely and bans the
usage of ObjectOutputStream and ObjectInputStream entirely.

Yet, we can't fully serialize all exception anymore such that this commit
is best effort and adds hand written serialization to all elasticsearch exceptions
as well to a selected set of JDK and Lucene exceptions. (see StreamOutput#writeThrowable /
StreamInput.readThrowable). Stacktraces should be preserved for all exceptions while
several names might be replaced with ElasticsearchException if there is no mapping for
the given exception.
2015-06-30 14:51:43 +02:00
Alexander Reelsen f26672c184 Release: Build two RPMS, signed and unsigned
In order to support older RPM based distributions like CentOS5,
we should have one RPM available, which is not signed.

This commit creates an unsigned RPM first, then moves it over to
target/releases during the build, then builds a signed RPM.

The unsigned one is uploaded via S3, where as the signed one is
used for the repositories.

In addition, you can now build an RPM without having to specify
any gpg credentials due to offloading this into a maven profile
that is only activated when specifying `rpm.sign` property.

Closes #11587
2015-06-30 14:22:20 +02:00
Simon Willnauer d1a29a63d2 Decode URL.getPath before resolving a real file
URL with spaces are URL-Encoded and need to be decoded before
they can be used to open files etc.
2015-06-30 13:33:40 +02:00
Boaz Leskes 4102e263df AwaitFix testTracerLog 2015-06-30 13:25:27 +02:00
Boaz Leskes 4649650ab2 Merge pull request #11928 from szroland/#11897
Create Snapshot: remove _create from POST path to match PUT

Closes #11928
2015-06-30 12:16:22 +02:00
Martijn van Groningen 1df8e56a83 Merge pull request #11929 from martijnvg/percolator/drop__own_qpc_cache
The query parse context should be fetched from the IndexQueryParseService
2015-06-30 11:04:32 +02:00
Adrien Grand ed561cd434 Merge pull request #11768 from jpountz/fix/stream_eos
Make sure messages are fully read even in case of EOS markers.
2015-06-30 09:00:59 +02:00
Alexander Reelsen 185dbac8fe Testing: Fix running of rest tests against external cluster
The TransportClient specified in the ExternalCluster was missing a
`path.home` setting and thus would not start.
2015-06-30 08:17:45 +02:00
Robert Muir f20e23e6fb jar hell check should fail, if jars require higher java version 2015-06-30 01:49:27 -04:00
Robert Muir 0b2070e350 Merge pull request #11932 from rmuir/jarhell
Fail startup (and tests) on jar hell
2015-06-30 00:58:44 -04:00
Robert Muir 4059d7c18d tests: fix test bugs 2015-06-30 00:16:39 -04:00
Robert Muir b58eb35dfa detect jar hell in tests better and fix more stuff 2015-06-30 00:12:40 -04:00
Robert Muir f8fbf1fa0c Fail startup (and tests) on jar hell 2015-06-29 23:13:45 -04:00
Martijn van Groningen 07090b8ee8 percolator: The query parse context should be fetched from the IndexQueryParseService
instead of maintaining a thread local cache in the PercolatorQueriesRegistry.
Before PercolatorQueriesRegistry had its own cache, because all the queries had to forcefully opt out of caching. Nowadays in master small segments are never cached by the query cache, so the reason for the dedicated cache is no longer valid.
2015-06-30 00:32:09 +02:00
Robert Muir 0bb65f710e tests: simple tests for things we should kick out 2015-06-29 17:21:25 -04:00
szroland e2d76aaa20 remove _create from POST to match PUT path 2015-06-29 21:55:52 +02:00
Simon Willnauer d274e585e6 Add jsr-166e license, notice and sha1 hash 2015-06-29 21:17:22 +02:00
Simon Willnauer fcbd0dddcd Don't pull the engine for access to the QueryCachingPolicy 2015-06-29 21:07:47 +02:00
Robert Muir 0ae638fbb2 Merge pull request #11918 from rmuir/plugin_manager_classpath
Load plugins into classpath in bootstrap
2015-06-29 14:45:38 -04:00
Robert Muir 3f95275f23 Add back check with a note, so anyone reviewing isn't confused. 2015-06-29 13:20:53 -04:00
Robert Muir b93af3c27c Remove try/finally, its not necessary here 2015-06-29 12:35:59 -04:00
Robert Muir f9598030fd Load plugins into classpath in bootstrap 2015-06-29 12:11:04 -04:00
Martijn van Groningen c9f21a7ecc Merge pull request #11909 from martijnvg/nested_query_min_score_mode
Added `min` score mode.
2015-06-29 17:48:48 +02:00
Robert Muir 55c33b29f6 Merge pull request #11898 from rmuir/lockdown
steps to remove dangerous security permissions
2015-06-29 11:13:16 -04:00
Simon Willnauer 2dfd6df369 Merge pull request #11911 from s1monw/no_utf_uuid
Don't convert possibly corrupted bytes to UTF-8
2015-06-29 15:00:40 +02:00
Colin Goodheart-Smithe b5e958b2e0 Aggregations: Fix bucket_script builder method name 2015-06-29 12:53:38 +01:00
Adrien Grand ccaf846f1e Store filter cache statistics at the shard level instead of index.
Today we keep track of how often filters are used at the index level in order
to decide whether they should be cached or not. This is an issue if you have
several shards of the same index on the same node as it will multiply statistics
by the number of shards that you have for this index on the node, which defeats
the purpose of waiting for a filter to be reused before caching them.
2015-06-29 13:34:31 +02:00
Simon Willnauer 1a44afed05 Don't convert possibly corrupted bytes to UTF-8
If the translog UUID is corrupted we should not convert it
to UTF-8 since it might be invalid. Instead we should compare
the UTF-8 byte representation directly.
2015-06-29 13:16:52 +02:00
Adrien Grand 21ffd931ec Fix missing renamings.
Relates to #11893
2015-06-29 12:47:10 +02:00
Martijn van Groningen 47a43e4063 nested query: Added `min` score mode.
This score mode was added with the Lucene 5.2 release, but the `nested` query parser hasn't been changed to use it.
2015-06-29 12:26:30 +02:00
Masaru Hasegawa 80d0f1e5ea Fix RecoveryState timestamps
Use System.currentTimeMillis() for wall clock time and use System.nanoTime() for delta.
Fixes #11870.
2015-06-29 19:24:23 +09:00