Commit Graph

881 Commits

Author SHA1 Message Date
David Pilato ed45ad6327 Fix Azure repository with only one primary account
Using a single azure account is now rejected.
This commit fixes this issue and adds a test for it.

This regression was introduced with #13779. Hopefully no elasticsearch version has been released since then.

Needs to be merged in 2.2, 2.x and master branches.
2016-01-14 13:50:02 +01:00
Nik Everett 01ce49e94e Ban Serializable
1. Uses forbidden patterns to prevent things from referencing
java.io.Serializable or from mentioning serialVersionUID.
2. Uses -Xlint:-serial so we don't have to hear from javac that we aren't
declaring serialVersionUID on any classes that we make that happen to extend
Serializable.
3. Remove Serializable and serialVersionUID declarations.

I didn't use forbidden apis because it doesn't look like it has a way to ban
explicitly implementing Serializable. If you try to ban Serializable with
forbidden apis you end up banning all Exceptions and all Strings.

Closes #15847
2016-01-11 16:57:31 -05:00
Nik Everett b29c416b7c Merge pull request #15848 from nik9000/xlint3
Remove -Xlint:-deprecation from all but core
2016-01-11 16:55:13 -05:00
Nik Everett 6250f4dbaa Remove deprecated azure settings 2016-01-08 13:13:14 -05:00
Jason Tedor 871d1b4885 Remove and forbid use of j.u.c.ThreadLocalRandom
This commit removes and now forbids all uses of
java.util.concurrent.ThreadLocalRandom across the codebase. The
underlying issue with ThreadLocalRandom is that it can not be
seeded. This means that if ThreadLocalRandom is used in production code,
then tests that cover any code path containing ThreadLocalRandom will be
prevented from being reproducible by use of ThreadLocalRandom. Instead,
using org.elasticsearch.common.random.Randomness#get will give
reproducible sources of random when running under tests and otherwise
still give an instance of ThreadLocalRandom when running as production
code.
2016-01-08 12:23:48 -05:00
Nik Everett 98fdb39d3d Remove deprecated settings 2016-01-08 11:17:56 -05:00
Nik Everett 81a7607256 Remove -Xlint:-deprecation from plugins
Instead we suppress warnings about using deprecated stuff near the usage
site with a comment about why its ok.
2016-01-07 20:44:46 -05:00
Nik Everett 42cb2a5f02 Merge pull request #15811 from nik9000/def_cleanup
Cleanups for Def
2016-01-07 11:03:01 -05:00
Nik Everett 52f28888d5 Merge pull request #15813 from nik9000/xlint1
Remove Xlint:-override,-fallthrough,-static
2016-01-07 08:34:40 -05:00
Nik Everett 244120a065 Remove more Xlint skips 2016-01-06 23:53:05 -05:00
Nik Everett 0786c506dc Remove a few more Xlint skips 2016-01-06 23:28:13 -05:00
Nik Everett 20e7fa97db Remove Xlint:-override,-fallthrough,-static
Adds `@SuppressWarnings("fallthrough")` in two places where the fallthrough
is used to implement well known hashing algorithms.
2016-01-06 22:27:14 -05:00
Nik Everett 74c132afc6 Standardize some methods on varargs
Right now we define the same sort of methods as taking String arrays and
string varargs. We should standardize on one and varargs is easier to
call so lets use varargs!
2016-01-06 21:01:58 -05:00
Nik Everett 32605ecb4f Cleanups for Def
Manually I:
1. Added some missing raw types warnings suppressions.
2. Removed some unused unchecked cast warning suppressions.
3. Added <?> to Class.

I let my IDE:
1. Remove unneeded casts.
2. Reorder imports (just ignore these, everyone does).
2016-01-06 20:28:49 -05:00
Martijn van Groningen 2d6adf6428 Percolator refactoring:
* Added percolator field mapper that extracts the query terms and indexes these terms with the percolator query.
* At percolate time these extracted terms are used to query percolator queries that are like to be evaluated. This can significantly cut down the time it takes to percolate. Whereas before all percolator queries were evaluated if they matches with the document being percolated.
* Changes made to percolator queries are no longer immediately visible, a refresh needs to happen before the changes are visible.
* By default the percolate api only returns upto 10 matches instead of returning all matching percolator queries.
* Made percolate more modular, so that it is easier to add unit tests.
* Added unit tests for the percolator.

Closes #12664
Closes #13646
2016-01-06 16:08:10 +01:00
Igor Motov a89dba27c2 Task Management: Add framework for registering and communicating with tasks
Adds task manager class and enables all activities to register with the task manager. Currently, the immutable Transport*Activity class represents activity itself shared across all requests. This PR adds and an additional structure Task that keeps track of currently running requests and can be used to communicate with these requests using TransportTaskAction.

Related to #15117
2016-01-05 12:24:43 -05:00
Adrien Grand 1a47226d9a Merge pull request #15663 from jpountz/remove/mapping_backcompat
Remove mapping backward compatibilit with pre-2.0.
2016-01-04 10:05:39 +01:00
Robert Muir 25914ae879 Merge pull request #15688 from rmuir/thirdPartyAudit3
Improve thirdPartyAudit check, round 3
2015-12-29 09:24:51 -05:00
David Pilato 96b3166c6d Add timeout settings (default to 5 minutes)
By default, azure does not timeout. This commit adds support for a timeout settings which defaults to 5 minutes.
It's a timeout **per request** not a global timeout for a snapshot request.

It can be defined globally, per account or both. Defaults to `5m`.

```yml
cloud:
    azure:
        storage:
            timeout: 10s
            my_account1:
                account: your_azure_storage_account1
                key: your_azure_storage_key1
                default: true
            my_account2:
                account: your_azure_storage_account2
                key: your_azure_storage_key2
                timeout: 30s
```

In this example, timeout will be 10s for `my_account1` and 30s for `my_account2`.

Closes #14277.
2015-12-29 11:40:48 +01:00
David Pilato a49fe189b0 Support global `repositories.azure.` settings
All those repository settings can also be defined globally in `elasticsearch.yml` file using prefix `repositories.azure.`. For example:

```yml
repositories.azure:
    container: backup-container
    base_path: backups
    chunk_size: 32m
    compress": true
```

Closes #13776.
2015-12-29 10:43:01 +01:00
Robert Muir 180ab2493e Improve thirdPartyAudit check, round 3 2015-12-28 22:38:55 -05:00
Adrien Grand 5eb7555ffb Make text parsing less lenient.
It now requires that the parser is on a value.
2015-12-28 16:06:42 +01:00
Adrien Grand af122f4151 Remove mapping backward compatibilit with pre-2.0.
This removes the backward compatibility layer with pre-2.0 indices, notably
the extraction of _id, _routing or _timestamp from the source document when a
path is defined.
2015-12-24 13:47:37 +01:00
Robert Muir d144ba24a5 Merge pull request #15588 from rmuir/hdfs2-only
merge current hdfs improvements to master
2015-12-23 18:17:22 -05:00
Robert Muir f14a21639c add cleanups from simon 2015-12-23 18:15:33 -05:00
Adrien Grand d8d8666877 Remove `index_name` back compat.
Since 2.0 we enforce that fields have the same full and index names. So in 3.x
we can remove the ability to have different names on the same field.
2015-12-23 14:55:26 +01:00
Adrien Grand 56d2dd701e Fix SizeMappingTests failure. 2015-12-23 10:48:00 +01:00
Adrien Grand a2072fe927 Merge pull request #15539 from jpountz/fix/immutable_document_mapper
Make mapping updates more robust.
2015-12-23 09:55:42 +01:00
Adrien Grand f535c27024 Make mapping updates more robust.
This changes a couple of things:

Mappings are truly immutable. Before, each field mapper stored a
MappedFieldTypeReference that was shared across fields that have the same name
across types. This means that a mapping update could have the side-effect of
changing the field type in other types when updateAllTypes is true. This works
differently now: after a mapping update, a new copy of the mappings is created
in such a way that fields across different types have the same MappedFieldType.
See the new Mapper.updateFieldType API which replaces MappedFieldTypeReference.

DocumentMapper is now immutable and MapperService.merge has been refactored in
such a way that if an exception is thrown while eg. lookup structures are being
updated, then the whole mapping update will be aborted. As a consequence,
FieldTypeLookup's checkCompatibility has been folded into copyAndAddAll.

Synchronization was simplified: given that mappings are truly immutable, we
don't need the read/write lock so that no documents can be parsed while a
mapping update is being processed. Document parsing is not performed under a
lock anymore, and mapping merging uses a simple synchronized block.
2015-12-23 09:55:07 +01:00
Lee Hinman 482843e27b Fix build to run correctly on FreeBSD
This adds the required changes/checks so that the build can run on
FreeBSD.

There are a few things that differ between FreeBSD and Linux:

- CPU probes return -1 for CPU usage
- `hot_threads` cannot be supported on FreeBSD

From OpenJDK's `os_bsd.cpp`:

```c++
bool os::is_thread_cpu_time_supported() {
  #ifdef __APPLE__
  return true;
  #else
  return false;
  #endif
}
```

So this API now returns (for each FreeBSD node):

```
curl -s localhost:9200/_nodes/hot_threads
::: {Devil Hunter Gabriel}{q8OJnKCcQS6EB9fygU4R4g}{127.0.0.1}{127.0.0.1:9300}
   hot_threads is not supported on FreeBSD
```

- multicast fails in native `join` method - known bug:
  https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=193246

Which causes:

```
1> Caused by: java.net.SocketException: Invalid argument
1>    at java.net.PlainDatagramSocketImpl.join(Native Method)
1>    at java.net.AbstractPlainDatagramSocketImpl.join(AbstractPlainDatagramSocketImpl.java:179)
1>    at java.net.MulticastSocket.joinGroup(MulticastSocket.java:323)
1>    at org.elasticsearch.plugin.discovery.multicast.MulticastChannel$Plain.buildMulticastSocket(MulticastChannel.java:309)
```

So these tests are skipped on FreeBSD.

Resolves #15562
2015-12-22 12:36:04 -07:00
Robert Muir 7abd051734 better containing of hadoop for actual blobstore operations 2015-12-22 12:07:37 -05:00
Robert Muir 010d1a89c5 Merge branch 'master' into hdfs2-only 2015-12-22 00:40:54 -05:00
Robert Muir 91dcc9e073 tidy up 2015-12-22 00:28:53 -05:00
Robert Muir 9573bb9f15 make sure BlobStore.close always triggers ACE on any access afterwards 2015-12-22 00:21:03 -05:00
Robert Muir a04268e42e reorder checks 2015-12-21 23:52:16 -05:00
Robert Muir a587ba110c add some safety around repository 2015-12-21 23:48:22 -05:00
Ryan Ernst d104d6d652 Refactor hdfs unit tests to be simple and check every configuration
error condition
2015-12-21 19:38:54 -08:00
Ryan Ernst af7d6b629c Change hdfs unit tests to be a single node test instead of integ test 2015-12-21 18:32:28 -08:00
Robert Muir c54d53c8d5 streamline these classes a bit 2015-12-21 21:11:06 -05:00
Robert Muir 795869c345 remove filecontextfactory 2015-12-21 20:55:17 -05:00
Robert Muir 956281f039 remove shitton of permissions 2015-12-21 20:26:13 -05:00
Ryan Ernst d0e9306413 Remove reading node settings as defaults for hdfs repository settings 2015-12-21 16:25:55 -08:00
Robert Muir 3c07a427dc fix exc handling 2015-12-21 19:06:56 -05:00
Ryan Ernst 26eaa16a89 Remove "additional config" from hdfs repositories 2015-12-21 15:55:13 -08:00
Robert Muir 2cbfc54a81 avoid too-long classpath so it works on windows 2015-12-21 18:25:08 -05:00
Robert Muir 7065639a26 add test for listing 2015-12-21 17:25:15 -05:00
Robert Muir 5ebcf183e5 tests 2015-12-21 17:02:50 -05:00
Robert Muir b8524bdb11 add tests 2015-12-21 16:16:24 -05:00
Robert Muir 3a2464b80e improve build logic on windows without native libraries 2015-12-21 15:37:34 -05:00
Robert Muir 0ed45c5bfb remove filesystem leniency 2015-12-21 14:16:53 -05:00