Commit Graph

2172 Commits

Author SHA1 Message Date
Jim Ferenczi e852eb0548 Revert accidentally pushed changes in NoriAnalysisTests 2018-05-30 15:34:55 +02:00
Jim Ferenczi f582418ada Fix missing option serialization after backport
Relates 
2018-05-30 12:55:31 +02:00
Martijn van Groningen 56472d6505
Merge remote-tracking branch 'es/master' into ccr
* es/master: (24 commits)
  Add missing_bucket option in the composite agg ()
  Rename index_prefix to index_prefixes ()
  Rename methods in PersistentTasksService ()
  [DOCS] Fix watcher file location
  Update the version checks around range bucket keys, now that the change was backported.
  Use dedicated ML APIs in tests ()
  [DOCS] Remove reference to platinum Docker image ()
  Minor clean-up in InternalRange. ()
  stable filemode for zip distributions ()
  [DOCS] Adds missing TLS settings for auditing ()
  [test] packaging: use shell when running commands ()
  Fix location of AbstractHttpServerTransport ()
  [test] packaging test logging for suse distros
  Moved keyword tokenizer to analysis-common module ()
  Upgrade to Lucene-7.4-snapshot-1cbadda4d3 ()
  Limit the scope of BouncyCastle dependency ()
  [DOCS] Reset edit links ()
  Fix IndexTemplateMetaData parsing from xContent ()
  Remove log traces in AzureStorageServiceImpl and fix test ()
  Deprecate accepting malformed requests in stored script API ()
  ...
2018-05-30 09:55:34 +02:00
Tim Brooks ad0dc580c5
Fix location of AbstractHttpServerTransport ()
Currently AbstractHttpServerTransport is in a netty4 module. This is the
incorrect location. This commit moves it out of netty4 module.
Additionally, it moves unit tests that test AbstractHttpServerTransport
logic to server.
2018-05-29 13:14:23 -06:00
Martijn van Groningen 544822c78b
Moved keyword tokenizer to analysis-common module ()
Relates to 
2018-05-29 19:22:28 +02:00
Nhat Nguyen 363f1e84ca
Upgrade to Lucene-7.4-snapshot-1cbadda4d3 ()
This snapshot includes LUCENE-8328 which is needed to stabilize CCR builds.
2018-05-29 12:29:52 -04:00
Tanguy Leroux bfa784e5cd
Remove log traces in AzureStorageServiceImpl and fix test ()
This commit removes some log traces in AzureStorageServiceImpl and also
fixes the AzureStorageServiceTests so that is uses the real
implementation to create Azure clients.
2018-05-29 16:50:37 +02:00
Martijn van Groningen e6dd4144ce
Merge remote-tracking branch 'es/master' into ccr
* es/master:
  Move score script context from SearchScript to its own class ()
  Fix bad version check writing Repository nodes ()
  [docs] explainer for java packaging tests ()
  Remove Throwable usage from transport modules ()
  REST high-level client: add put ingest pipeline API ()
  Update the version checks around ip_range bucket keys, now that the change was backported.
  Mute IndexMasterFailoverIT.testMasterFailoverDuringIndexingWithMappingChanges
  Use geohash cell instead of just a corner in geo_bounding_box ()
  Limit user to single concurrent auth per realm ()
  [Tests] Move templated _rank_eval tests ()
  Security: fix dynamic mapping updates with aliases ()
  Ensure that ip_range aggregations always return bucket keys. ()
  Use remote client in TransportFieldCapsAction ()
  Move Watcher versioning setting to meta field ()
  [Docs] Explain incomplete dates in range queries ()
  Move persistent task registrations to core ()
  Decouple ClusterStateTaskListener & ClusterApplier ()
  Send client headers from TransportClient ()
  Packaging: Ensure upgrade_is_oss flag file is always deleted ()
  Force stable file modes for built packages ()
2018-05-25 11:07:45 +02:00
Martijn van Groningen ae2f021f1c
Move score script context from SearchScript to its own class () 2018-05-25 07:17:50 +02:00
Tim Brooks e8b70273c1
Remove Throwable usage from transport modules ()
Currently nio and netty modules use the CompletableFuture class for
managing listeners. This is unfortunate as that class accepts
Throwable. This commit adds a class CompletableContext that wraps
the CompletableFuture but does not accept Throwable. This allows the
modification of netty and nio logic to no longer handle Throwable.
2018-05-24 17:33:29 -06:00
Nhat Nguyen 3fb2c45fa7 Merge branch 'master' into ccr
* master:
  [DOCS] Fixes typos in security settings
  Fix GeoShapeQueryBuilder serialization after backport
  [DOCS] Splits auditing.asciidoc into smaller files
  Reintroduce mandatory http pipelining support ()
  Painless: Types Section Clean Up ()
  Add support for indexed shape routing in geo_shape query ()
  [test] java tests for archive packaging ()
  Revert "Make http pipelining support mandatory ()" ()
  [DOCS] Fix more edit URLs in Stack Overview ()
  Use correct cluster state version for node fault detection ()
  Change serialization version of doc-value fields.
  [DOCS] Fixes broken link for native realm
  [DOCS] Clarified audit.index.client.hosts ()
  [TEST] Don't expect acks when isolating nodes
  Add a `format` option to `docvalue_fields`. ()
  Fixes UpdateSettingsRequestStreamableTests mutate bug
  Mustes {p0=snapshot.get_repository/10_basic/*} YAML test
  Revert "Mutes MachineLearningTests.testNoAttributes_givenSameAndMlEnabled"
  Only allow x-pack metadata if all nodes are ready ()
  Mutes MachineLearningTests.testNoAttributes_givenSameAndMlEnabled
  Use original settings on full-cluster restart ()
  Only ack cluster state updates successfully applied on all nodes ()
  Expose Lucene's FeatureField. ()
  Fix a grammatical error in the 'search types' documentation.
  Remove http pipelining from integration test case ()
2018-05-23 23:05:14 -04:00
Tim Brooks d7040ad7b4
Reintroduce mandatory http pipelining support ()
This commit reintroduces 31251c9 and 63a5799. These commits introduced a
memory leak and were reverted. This commit brings those commits back
and fixes the memory leak by removing unnecessary retain method calls.
2018-05-23 14:38:52 -06:00
Colin Goodheart-Smithe 4fd0a3e492 Revert "Make http pipelining support mandatory ()" ()
This reverts commit 31251c9 introduced in .

We suspect this commit is causing the OOME's reported in  and we will use this PR to test this assertion.
2018-05-23 10:54:46 -06:00
Nhat Nguyen fb48b029e8 Merge branch 'master' into ccr
* master:
  QA: Add xpack tests to rolling upgrade ()
  Modify state of VerifyRepositoryResponse for bwc ()
  Reduce CLI scripts to one-liners on Windows ()
  Simplify number of shards setting ()
  Replace Request#setHeaders with addHeader ()
  [TEST] remove endless wait in RestClientTests ()
  [Docs] Fix script-fields snippet execution ()
  Upgrade to Lucene-7.4.0-snapshot-cc2ee23050 ()
  [DOCS] Add SAML configuration information ()
  [DOCS] Remove X-Pack references from SQL CLI ()
  Make http pipelining support mandatory ()
  [Docs] Fix typo in circuit breaker docs ()
  [Feature] Adding a char_group tokenizer ()
  [Docs] Fix broken cross link in documentation
  Test: wait for netty threads in a JUnit ClassRule ()
  Increase the maximum number of filters that may be in the cache. ()
  [Security] Include an empty json object in an json array when FLS filters out all fields ()
  [TEST] Wait for CS to be fully applied in testDeleteCreateInOneBulk
  Add more yaml tests for get alias API ()
  Ignore empty completion input ()
  [DOCS] fixed incorrect default
  [ML] Filter undefined job groups from update calendar actions ()
  Fix docs failure on language analyzers ()
  [Docs] Fix inconsistencies in snapshot/restore doc ()
  Enable installing plugins from snapshots.elastic.co ()
  Remove fedora 26, add 28 ()
  Accept Gradle build scan agreement ()
  Remove logging from elasticsearch-nio jar ()
  Add Delete Repository High Level REST API ()
2018-05-22 17:24:00 -04:00
Nhat Nguyen 1918a30237
Upgrade to Lucene-7.4.0-snapshot-cc2ee23050 ()
The new snapshot includes LUCENE-8324 which fixes missing checkpoint
after a fully deletes segment is dropped on flush. This snapshot should
resolves failed tests in the CorruptedFileIT suite.

Closes 
Closes 
2018-05-22 13:11:48 -04:00
Tim Brooks 31251c9a6d
Make http pipelining support mandatory ()
This is related to  and . This commit removes the abilitiy
to disable http pipelining. After this commit, any elasticsearch node
will support pipelined requests from a client. Additionally, it extracts
some of the http pipelining work to the server module. This extracted
work is used to implement pipelining for the nio plugin.
2018-05-22 09:29:31 -06:00
Tim Brooks abf8c56a37
Remove logging from elasticsearch-nio jar ()
This is related to . The elasticsearch-nio jar is supposed to be
a library opposed to a framework. Currently it internally logs certain
exceptions. This commit modifies it to not rely on logging. Instead
exception handlers are passed by the applications that use the jar.
2018-05-21 20:18:12 -06:00
Nhat Nguyen 3f6434cd6d Merge branch 'master' into ccr
* master:
  Reduce CLI scripts to one-liners ()
  SQL: Preserve scoring in bool queries ()
  QA: Switch rolling upgrade to 3 nodes ()
  [TEST] Enable DEBUG logging on testAutoQueueSizingWithMax
  [ML] Don't install empty ML metadata on startup ()
  Add assertion on removing copy_settings ()
  bump lucene version for 6_3_0
  [DOCS] Mark painless execute api as experimental ()
  disable annotation processor for docs ()
  Add more script contexts ()
  Fix default shards count in create index docs ()
  Mute testCorruptFileThenSnapshotAndRestore
2018-05-21 18:53:46 -04:00
Martijn van Groningen 314cd6feaf
Add more script contexts ()
Added dedicated script contexts for:
* script function score
* script sorting
* terms_set query

Scripts for these contexts will either have a specific return value or
use scoring and therefor in the future will need their own scripting classes.

Relates to 
2018-05-20 21:31:50 +02:00
Nhat Nguyen 3245e78b78 Merge branch 'master' into ccr
* master:
  Scripting: Remove getDate methods from ScriptDocValues ()
  Upgrade to Lucene-7.4.0-snapshot-59f2b7aec2 ()
  [Docs] Fix single page :docs:check invocation ()
  Docs: Add uptasticsearch to list of clients ()
  [DOCS] Removes out-dated x-pack/docs/en/index.asciidoc
  [DOCS] Removes redundant index.asciidoc files ()
  [TEST] Reduce forecast overflow to disk test memory limit ()
  Plugins: Remove meta plugins ()
  [DOCS] Moves X-Pack configurationg pages in table of contents ()
  TEST: Add engine log to testCorruptFileThenSnapshotAndRestore
  [ML][TEST] Fix bucket count assertion in ModelPlotsIT ()
  [ML][TEST] Make AutodetectMemoryLimitIT less fragile ()
  Default copy settings to true and deprecate on the REST layer ()
  [Build] Add test admin when starting gradle run with trial license and
  This implementation lazily (on 1st forecast request) checks for available diskspace and creates a subfolder for storing data outside of Lucene indexes, but as part of the ES data paths.
  Tests: Fail if test watches could not be triggered ()
  [ML] add version information in case of crash of native ML process ()
  Make TransportClusterStateAction abide to our style ()
  Change required version for Get Settings transport API changes to 6.4.0 ()
  [DOCS] Fixes edit URLs for stack overview ()
  Silence sleep based watcher test
  [TEST] Adjust version skips for movavg/movfn tests
  [DOCS] Replace X-Pack terms with attributes
  [ML] Clean left behind model state docs ()
  Correct typos
  filters agg docs duplicated 'bucket' word removal ()
  top_hits doc example description update ()
  [Docs] Replace InetSocketTransportAddress with TransportAdress ()
  [TEST] Account for increase in ML C++ memory usage ()
  User proper write-once semantics for GCS repository ()
  Remove bogus file accidentally added
  Add detailed assert message to IndexAuditUpgradeIT ()
  Adjust fast forward for token expiration test  ()
  Improve explanation in rescore ()
  Deprecate `nGram` and `edgeNGram` names for ngram filters ()
  Watcher: Fix watch history template for dynamic slack attachments ()
  Fix _cluster/state to always return cluster_uuid ()
  [Tests] Add debug information to CorruptedFileIT

# Conflicts:
#	test/framework/src/main/java/org/elasticsearch/indices/analysis/AnalysisFactoryTestCase.java
2018-05-19 07:38:17 -04:00
Nhat Nguyen 67d8fc222d
Upgrade to Lucene-7.4.0-snapshot-59f2b7aec2 ()
This snapshot resolves issues related to ShrinkIndexIT.
2018-05-18 18:21:39 -04:00
Ryan Ernst b3f3a4312b
Plugins: Remove meta plugins ()
Meta plugins existed only for a short time, in order to enable breaking
up x-pack into multiple plugins. However, now that x-pack is no longer
installed as a plugin, the need for them has disappeared. This commit
removes the meta plugins infrastructure.
2018-05-18 10:56:08 -07:00
Yannick Welsch b57d21bab1
User proper write-once semantics for GCS repository ()
There's no need for an extra blobExists() call when writing a blob to the GCS service. GCS provides
an option (with stronger consistency guarantees) on the insert method that guarantees that the
blob that's uploaded does not already exist.

Relates to 
2018-05-17 13:58:10 +02:00
Martijn van Groningen 5298237847
Merge remote-tracking branch 'es/master' into ccr
* es/master: (74 commits)
  Preserve REST client auth despite 401 response ()
  [test] packaging: add windows boxes ()
  Make xpack modules instead of a meta plugin ()
  Mute ShrinkIndexIT
  [ML] DeleteExpiredDataAction should use client with origin ()
  Reindex: Fixed typo in assertion failure message ()
  [DOCS] Fixes list of unconverted snippets in build.gradle
  [DOCS] Reorganizes RBAC documentation
  SQL: Remove dependency for server's version from JDBC driver ()
  Test: increase search logging for LicensingTests
  Adjust serialization version in IndicesOptions
  [TEST] Fix compilation
  Remove version argument in RangeFieldType ()
  Remove unused DirectoryUtils class. ()
  Mitigate date histogram slowdowns with non-fixed timezones. ()
  Add a MovingFunction pipeline aggregation, deprecate MovingAvg agg ()
  Removes AwaitsFix on IndicesOptionsTests
  Template upgrades should happen in a system context ()
  Fix bug in BucketMetrics path traversal ()
  Fixes IndiceOptionsTests to serialise correctly ()
  ...
2018-05-17 10:34:44 +02:00
Albert Zaharovits 8e9d2b1e28
S3 repo plugin populate SettingsFilter ()
The accessKey and secretKey repo settings (in the cluster state)
of the s3 client are registered and will populate the SettingsFilter.
2018-05-16 16:12:24 +03:00
Tim Brooks 99b9ab58e2
Add nio http server transport ()
This commit is related to . It adds an nio driven http server
transport. Currently it only supports basic http features. Cors,
pipeling, and read timeouts will need to be added in future PRs.
2018-05-15 16:37:14 -06:00
Albert Zaharovits 801973fa9f
Repository GCS plugin new client library ()
This does away with the deprecated `com.google.api-client:google-api-client:1.23`
and replaces it with `com.google.cloud:google-cloud-storage:1.28.0`.
It also changes security permissions for the repository-gcs plugin.
2018-05-15 18:22:58 +03:00
Nhat Nguyen b971a81e70 Merge branch 'master' into ccr
* master:
  Default to one shard ()
  Unmute IndexUpgradeIT tests
  Forbid expensive query parts in ranking evaluation ()
  Docs: Update HighLevelRestClient migration docs ()
  Clients: Switch to new performRequest ()
  [TEST] Fix typo in MovAvgIT test
  Add missing dependencies on testClasses ()
  [TEST] Mute ML test that needs updating to following ml-cpp changes
  Document woes between auto-expand-replicas and allocation filtering ()
  Moved tokenizers to analysis common module ()
  Adjust copy settings versions
  Mute ShrinkIndexIT suite
  SQL: SYS TABLES ordered according to *DBC specs ()
  Deprecate not copy settings and explicitly disallow ()
  [ML] Improve state persistence log message
  Build: Add mavenPlugin cluster configuration method ()
  Re-enable FlushIT tests
  Bump Gradle heap to 2 GB ()
  SQL: Use request flavored methods in tests ()
  Suppress hdfsFixture if there are spaces in the path ()
  Delete temporary blobs before creating index file ()
  Watcher: Remove TriggerEngine.getJobCount() ()
  [ML] Fix wire BWC for JobUpdate ()
  Use simpler write-once semantics for FS repository ()
  Derive max composite buffers from max content len
  Use simpler write-once semantics for HDFS repository ()
  SQL: Improve correctness of SYS COLUMNS & TYPES ()
  Mute two tests in FlushIT with @AwaitsFix.
  Fix incorrect template name in test case
  Build: Remove legacy bwc files from xpack ()
  Mute UnicastZenPingTests#testSimplePings with @AwaitsFix.
  Security: cleanup code in file stores ()
  Security: fix TokenMetaData equals and hashcode ()
  Mute two tests from SmokeTestWatcherWithSecurityClientYamlTestSuiteIT.
  Mute SharedClusterSnapshotRestoreIT#testSnapshotSucceedsAfterSnapshotFailure with @AwaitsFix.
  SQL: Improve compatibility with MS query ()
  SQL: Fix parsing of dates with milliseconds ()
2018-05-14 13:23:23 -04:00
David Turner 15df911e41
Suppress hdfsFixture if there are spaces in the path ()
HDFS sets its thread-name format based partly on a URL-encoded version of the
path, but the URL-encoding of spaces as `%20` is interpreted as a field in the
formatted string of type `2`, which is nonsensical. This change simply skips 
these tests in this case.
2018-05-11 13:36:31 +01:00
Yannick Welsch fc870fdb4c
Use simpler write-once semantics for HDFS repository ()
There's no need for an extra `blobExists()` call when writing a blob to the HDFS service. The writeBlob implementation for the HDFS repository already uses the `CreateFlag.CREATE` option on the file creation, which ensures that the blob that's uploaded does not already exist. This saves one network roundtrip.
2018-05-11 09:50:37 +02:00
Nhat Nguyen a5be4149a3 Merge branch 'master' into ccr
* master:
  Upgrade to Lucene-7.4-snapshot-6705632810 ()
  add version compatibility from 6.4.0 after backport, see  ()
  Security: Simplify security index listeners ()
  Add proper longitude validation in geo_polygon_query ()
  Remove Discovery.AckListener.onTimeout() ()
  Build: move generated-resources to build ()
  Reindex: Fold "with all deps" project into reindex ()
  Isolate REST client single host tests ()
  Solve Gradle deprecation warnings around shadowJar ()
  SAML: Process only signed data ()
  Remove BWC repository test ()
  Build: Remove xpack specific run task ()
  AwaitsFix IntegTestZipClientYamlTestSuiteIT#indices.split tests
  LLClient: Add setJsonEntity ()
  Expose CommonStatsFlags directly in IndicesStatsRequest. ()
  Silence IndexUpgradeIT test failures. ()
  Bump Gradle heap to 1792m ()
  [docs] add warning for read-write indices in force merge documentation ()
  Avoid deadlocks in cache ()
  Test: remove hardcoded list of unconfigured ciphers ()
  mute SplitIndexIT due to https://github.com/elastic/elasticsearch/issues/30416
  Docs: Test examples that recreate lang analyzers  ()
  BulkProcessor to retry based on status code ()
  Add GET Repository High Level REST API ()
  add a comment explaining the need for RetryOnReplicaException on missing mappings
  Add `coordinating_only` node selector ()
  Stop forking groovyc ()
  Avoid setting connection request timeout ()
  Use date format in `date_range` mapping before fallback to default ()
  Watcher: Increase HttpClient parallel sent requests ()

# Conflicts:
#	x-pack/plugin/core/src/test/java/org/elasticsearch/xpack/core/LocalStateCompositeXPackPlugin.java
2018-05-10 13:04:08 -04:00
Nhat Nguyen 519768b5d3
Upgrade to Lucene-7.4-snapshot-6705632810 ()
This snapshot is to include LUCENE-8298 which allows DocValues updates
to reset a value. This is needed for the Lucene rollback work.
2018-05-10 12:31:45 -04:00
Nhat Nguyen d9b9d7d107 Merge branch 'master' into ccr
* elastic-master:
  Watcher: Mark watcher as started only after loading watches ()
  Pass the task to broadcast actions ()
  Disable REST default settings testing until  is back-ported
  Correct wording in log message ()
  Do not fail snapshot when deleting a missing snapshotted file ()
  AwaitsFix testCreateShrinkIndexToN
  DOCS: Correct mapping tags in put-template api
  DOCS: Fix broken link in the put index template api
  Add put index template api to high level rest client ()
  Relax testAckedIndexing to allow document updating
  [Docs] Add snippets for POS stop tags default value
  Move respect accept header on no handler to 6.3.1
  Respect accept header on no handler ()
  [Test] Add analysis-nori plugin to the vagrant tests
  [Docs] Fix bad link
  [Docs] Fix end of section in the korean plugin docs
  Expose the Lucene Korean analyzer module in a plugin ()
  Docs: remove transport_client from CCS role example ()
  [Rollup] Validate timezone in range queries ()
  Use readFully() to read bytes from CipherInputStream ()
  Fix  docs Recently merged  had a doc bug that broke the doc build. This commit fixes.
  Test: remove cluster permission from CCS user ()
  Add Get Settings API support to java high-level rest client ()
  Watcher: Remove unneeded index deletion in tests
2018-05-08 09:04:01 -04:00
Jim Ferenczi 891d3bd9c3
Expose the Lucene Korean analyzer module in a plugin ()
This change adds a new plugin called `analysis-nori` that exposes
Korean text analysis in es using the new Lucene Korean analyzer module named (`nori`).
The plugin adds:
* a Korean analyzer: `nori`
* a Korean tokenizer: `nori_tokenizer`
* a part of speech stop filter: `nori_part_of_speech`
* a filter that can replace Hanja characters with their Hangul transcription: `nori_readingform`
2018-05-04 20:46:13 +02:00
Nhat Nguyen f3a87be74a Remove old sha files from dated Lucene snapshot
This old snapshot was used this branch only.
2018-05-04 10:57:54 -04:00
Nhat Nguyen db14717098 Merge branch 'master' into ccr
* master:
  Set the new lucene version for 6.4.0
  [ML][TEST] Clean up jobs in ModelPlotIT
  Upgrade to 7.4.0-snapshot-1ed95c097b ()
  Watcher: Ensure trigger service pauses execution ()
  [DOCS] Added coming qualifiers in changelog
  [DOCS] Commented out empty sections in the changelog to fix the doc build. ()
  Security: reduce garbage during index resolution ()
  Make RepositoriesMetaData contents unmodifiable ()
  Change quad tree max levels to 29. Closes  ()
  Test: use trial license in qa tests with security
  [ML] Add integration test for model plots ()
  SQL: Fix bug caused by empty composites ()
  [ML] Account for gaps in data counts after job is reopened ()
  InternalEngineTests.testConcurrentOutOfOrderDocsOnReplica should use two documents ()
  Change signature of Get Repositories Response ()
  Tests: Use different watch ids per test in smoke test ()
  [Docs] Add term query with normalizer example
  Adds Eclipse config for xpack licence headers ()
  Watcher: Make start/stop cycle more predictable and synchronous ()
  [test] add debug logging for packaging test
  [DOCS] Removed X-Pack Breaking Changes
  [DOCS] Fixes link to TLS LDAP info
  Update versions for start_trial after backport ()
  Packaging: Set elasticsearch user to have non-existent homedir ()
  [DOCS] Fixes broken links to bootstrap user ()
  Fix NPE when CumulativeSum agg encounters null/empty bucket ()
  Make licensing FIPS-140 compliant ()
  [DOCS] Reorganizes authentication details in Stack Overview ()
  Network: Remove http.enabled setting ()
  Fix merging logic of Suggester Options ()
  [DOCS] Adds LDAP realm configuration details ()
  [DOCS] Adds native realm configuration details ()
  ReplicationTracker.markAllocationIdAsInSync may hang if allocation is cancelled ()
  [DOCS] Enables edit links for X-Pack pages ()
  Packaging: Unmark systemd service file as a config file ()
  SQL: Reduce number of ranges generated for comparisons ()
  Tests: Simplify VersionUtils released version splitting ()
  Cancelling a peer recovery on the source can leak a primary permit ()
  Added changelog entry for deb prerelease version change ()
  Convert server javadoc to html5 ()
  Create default ES_TMPDIR on Windows ()
  [Docs] Clarify `fuzzy_like_this` redirect ()
  Post backport of .
  Fix docs of the `_ignored` meta field.
  Remove MapperService#types(). ()
  Remove useless version checks in REST tests. ()
  Add a new `_ignored` meta field. ()
  Move repository-azure fixture test to QA project ()

# Conflicts:
#	buildSrc/version.properties
#	server/src/test/java/org/elasticsearch/index/engine/InternalEngineTests.java
2018-05-04 09:40:57 -04:00
Jim Ferenczi dbd857341f
Upgrade to 7.4.0-snapshot-1ed95c097b ()
Upgrade to lucene-7.4.0-snapshot-1ed95c097b

This version contains:
* An Analyzer for Korean
* An IntervalQuery and IntervalsSource that retrieve minimum intervals of positional queries.
* A new API to retrieve matches (offsets and positions) of a query for a single document.
* Support for soft deletes in the index writer.
* A fixed shingle filter that handles index time synonyms.
* Support for emoji sequence in ICUTokenizer (with an upgrade to icu 61.1)
2018-05-04 11:44:22 +02:00
Tanguy Leroux 0aad5fd0f5
Move repository-azure fixture test to QA project ()
Similarly to what has been done in for the repository-s3 plugin, this
pull request moves the fixture test into a dedicated
repository-azure/qa/microsoft-azure-storage project.

It also exposes some environment variables which allows to execute the
integration tests against the real Azure Storage service. When the
environment variables are not defined, the integration tests are
executed using the fixture added in .

Closes 
2018-05-02 09:16:51 +02:00
Nhat Nguyen d52ca33bd9 Merge branch 'master' into ccr
* master: (68 commits)
  [DOCS] Removes X-Pack Elasticsearch release notes ()
  Correct an example in the top-level suggester documentation. ()
  [DOCS] Removes broken link
  [DOCS] Adds file realm configuration details ()
  [DOCS] Adds PKI realm configuration details ()
  Fix a reference to match_phrase_prefix in the match query docs. ()
  Fix failure for validate API on a terms query ()
  [DOCS] Fix 6.4-specific link in changelog ()
  Remove RepositoriesMetaData variadic constructor ()
  Test: increase authentication logging for debugging
  [DOCS] Removes redundant SAML realm settings ()
  REST Client: Add Request object flavored methods ()
  [DOCS] Adds changelog to Elasticsearch Reference ()
  [DOCS] Fixes section error
  SQL: Teach the CLI to ignore empty commands ()
  [DOCS] Adds Active Directory realm configuration details ()
  [DOCS] Removes redundant file realm settings ()
  [DOCS] Fixes users command name ()
  Build: Move gradle wrapper jar to a dot dir ()
  Build: Log a warning if disabling reindex-from-old ()
2018-05-01 21:07:54 -04:00
Tanguy Leroux 60b18b5510
Move repository-gcs fixture test to QA project ()
Similarly to what has been done in for the repository-s3 plugin, 
this commit moves the fixture test into a dedicated 
repository-gcs/qa/google-cloud-storage project.

It also exposes some environment variables which allows to 
execute the integration tests against the real Google Cloud 
Storage service. When the environment variables are not 
defined, the integration tests are executed using the fixture 
added in . Related to .
2018-04-30 13:31:14 +02:00
Nik Everett 9c8e015552
Build: Mostly silence warning about html4 javadoc ()
This *mostly* silences `javadoc`'s warning about defaulting to
generating html4 files by enabling generating html5 file for the
projects for which that works. It didn't work in a half dozen projects,
about half of which I've fixed in this PR, entirely by replacing
`<tt>thing</tt>` with `{@code thing}`.

There are a few remaining projects that contain javadoc with invalid
html5. I'll fix those projects in a followup.
2018-04-28 09:50:54 -04:00
Tanguy Leroux 7ae3b3b155
Move repository-s3 fixture tests to QA test project ()
This commit moves the repository-s3 fixture test added in  in a
new `repository-s3/qa/amazon-s3` project. This new project allows the
REST integration tests to be executed using the real S3 service when
all the required environment variables are provided. When no env var
is provided, then the tests are executed using the fixture added
in .

The REST tests located at the `repository-s3`plugin  project now only 
verify that the plugin is correctly loaded.

The REST tests have been adapted to allow a bucket name and a base 
path to be specified as env vars. This way it is possible to run the tests
with different base paths (could be anything, like a CI job name or a
branch name) without multiplicating buckets.

Related to 
2018-04-27 16:49:06 +02:00
Martijn van Groningen f683f9babe
Merge elasticsearch master into ccr 2018-04-25 08:03:25 +02:00
Ryan Ernst 0d8aa7527e Reorganize license files
This commit moves the apache and elastic license files into a new
root level `licenses` directory and rewrites the top level LICENSE.txt
to clarify the repository has a mix of apache and elastic licensed code.
2018-04-20 15:33:59 -07:00
Ryan Ernst fab5e21e7d Build: Split distributions into oss and default
This commit makes x-pack a module and adds it to the default
distrubtion. It also creates distributions for zip, tar, deb and rpm
which contain only oss code.
2018-04-20 15:33:57 -07:00
Nhat Nguyen 566d1826c8
Upgrade to Lucene-7.4-snapshot-330fd18f20 ()
This snapshot includes the fix for LUCENE-8253.
See: https://git-wip-us.apache.org/repos/asf?p=lucene-solr.git;h=330fd18
2018-04-17 18:11:26 -04:00
Nhat Nguyen 7ede88fe4b
Upgrade to Lucene-7.4.0-snapshot-bcf9f5c36b ()
This snapshot includes LUCENE-8253 and improvements for LUCENE-8233.
2018-04-16 13:34:06 -04:00
Martijn van Groningen 9da3e739fb
Merge remote-tracking branch 'es/master' into ccr
* es/master:
  Add remote cluster client ()
  Ensure flush happens on shard idle
  Adds SpanGapQueryBuilder in the query DSL ()
  Control max size and count of warning headers ()
  Make index APIs work without types. ()
  Deprecate filtering on `_type`. ()
  Fix auto-generated ID example format ()
  Fix typo in max number of threads check docs ()
  Add primary term to translog header ()
  Add a helper method to get a random java.util.TimeZone ()
  Move TimeValue into elasticsearch-core project ()
  Fix NPE in InternalGeoCentroidTests#testReduceRandom ()
  Build: introduce keystoreFile for cluster config ()
  test: Index more docs, so that it is less likely the search request does not time out.
2018-04-13 15:31:43 +02:00
Jay Modi e0ec8571ea
Build: introduce keystoreFile for cluster config ()
This commit introduces built in support for adding files to the
keystore when configuring the integration test cluster for a project.

In order to use this support, simply add `keystoreFile` followed by the
secure setting name and the path to the source file inside the
integTestCluster closure for a project. The built in support will
handle the creation of the keystore and the addition of the file to the
keystore.
2018-04-12 07:28:34 -06:00
Nhat Nguyen a90b3eadf3
Upgrade to Lucene-7.4.0-snapshot-bcf9f5c36b ()
This snapshot includes LUCENE-8246 which allows to customize the number 
of deletes a merge claims
2018-04-12 08:12:57 -04:00
Nhat Nguyen b24ca650ad Merge branch 'master' into ccr 2018-04-11 11:35:15 -04:00
Adrien Grand 4918924fae
Remove legacy mapping code. ()
Some features have been deprecated since `6.0` like the `_parent` field or the
ability to have multiple types per index. This allows to remove quite some
code, which in-turn will hopefully make it easier to proceed with the removal
of types.
2018-04-11 09:41:37 +02:00
Nhat Nguyen 30c06a6f8c
Upgrade to Lucene-7.4.0-snapshot-2b27dd846a ()
This snapshot version supports soft delete and the merge policy.
2018-04-09 12:25:49 -04:00
Lee Hinman a07ba9e400
Move Streams.copy into elasticsearch-core and make a multi-release jar ()
* Move Streams.copy into elasticsearch-core and make a multi-release jar

This moves the method `Streams.copy(InputStream in, OutputStream out)` into the
`elasticsearch-core` project (inside the `o.e.core.internal.io` package). It
also makes this class into a multi-release class where the Java 9 equivalent
uses `InputStream#transferTo`.

This is a followup from
https://github.com/elastic/elasticsearch/pull/29300#discussion_r178147495
2018-04-06 11:07:20 -06:00
Tanguy Leroux 26fc8ad109
Use fixture to test repository-azure plugin ()
This commit adds a new fixture that emulates an
Azure Storage service in order to improve the
existing integration tests. This is very similar
to what has been made for Google Cloud Storage
in  and for Amazon S3 in , and it
would have helped a lot to catch bugs like .
2018-04-06 11:06:20 +02:00
Tanguy Leroux 7d29087442
[Tests] Use mock storage in repository-gcs unit tests ()
The repository-gcs unit tests rely on the GoogleCloudStorageTestServer
but it would be better if they rely on a mocked Storage client instead.

That would also help to extract the GoogleCloudStorageFixture and the
GoogleCloudStorageTestServer classes in a QA third party project.

Closes 
2018-04-06 09:13:07 +02:00
Tanguy Leroux d813a05b9f
Use ESBlobStoreRepositoryIntegTestCase to test the repository-s3 plugin ()
This commit adds the S3BlobStoreRepositoryTests class that extends the
base testing class for S3. It also removes some usage of socket servers 
that emulate socket connections in unit tests. It was added to trigger 
security exceptions, but this won't be needed anymore since  
is merged.
2018-04-05 13:34:02 +02:00
Alan Woodward dccd43af47
Upgrade to lucene 7.3.0 () 2018-04-05 10:34:44 +01:00
Jason Tedor c95e7539e7
Enhance error for out of bounds byte size settings ()
Today when you input a byte size setting that is out of bounds for the
setting, you get an error message that indicates the maximum value of
the setting. The problem is that because we use ByteSize#toString, we
end up with a representation of the value that does not really tell you
what the bound is. For example, if the bound is 2^31 - 1 bytes, the
output would be 1.9gb which does not really tell you want the limit as
there are many byte size values that we format to the same 1.9gb with
ByteSize#toString. We have a method ByteSize#getStringRep that uses the
input units to the value as the output units for the string
representation, so we end up with no loss if we use this to report the
bound. This commit does this.
2018-04-04 07:22:13 -04:00
Tanguy Leroux 989e465964
Use fixture to test repository-s3 plugin ()
This commit adds a new fixture that emulates a S3 service in order to
improve the existing integration tests. This is very similar to what has
 been made for Google Cloud Storage in , and such tests would 
have helped a lot to catch bugs like .

The AmazonS3Fixture is brittle and only implements the very necessary
stuff for the S3 repository to work, but at least it works and can be
adapted for specific tests needs.
2018-04-03 11:30:43 +02:00
Adrien Grand 3bdfc8f3fb
Upgrade to lucene-7.3.0-snapshot-98a6b3d. ()
Most notable changes include:
 - this release doesn't have the 7.2.1 version constant so I had to create one
 - spatial4j and jts were upgraded
2018-04-03 09:27:14 +02:00
Christoph Büscher 318b0af953 Remove execute mode bit from source files
Some source files seem to have the execute bit (a+x) set, which doesn't
really seem to hurt but is a bit odd. This change removes those, making
the permissions similar to other source files in the repository.
2018-03-26 13:37:55 +02:00
Lee Hinman 8e8fdc4f0e
Decouple XContentBuilder from BytesReference ()
* Decouple XContentBuilder from BytesReference

This commit removes all mentions of `BytesReference` from `XContentBuilder`.
This is needed so that we can completely decouple the XContent code and move it
into its own dependency.

While this change appears large, it is due to two main changes, moving
`.bytes()` and `.string()` out of XContentBuilder itself into static methods
`BytesReference.bytes` and `Strings.toString` respectively. The rest of the
change is code reacting to these changes (the majority of it in tests).

Relates to 
2018-03-14 13:47:57 -06:00
David Pilato 87553bba16
Add ingest-attachment support for per document `indexed_chars` limit ()
We today support a global `indexed_chars` processor parameter. But in some cases, users would like to set this limit depending on the document itself.
It used to be supported in mapper-attachments plugin by extracting the limit value from a meta field in the document sent to indexation process.

We add an option which reads this limit value from the document itself
by adding a setting named `indexed_chars_field`.

Which allows running:

```
PUT _ingest/pipeline/attachment
{
  "description" : "Extract attachment information. Used to parse pdf and office files",
  "processors" : [
    {
      "attachment" : {
        "field" : "data",
        "indexed_chars_field" : "size"
      }
    }
  ]
}
```

Then index either:

```
PUT index/doc/1?pipeline=attachment
{
  "data": "BASE64"
}
```

Which will use the default value (or the one defined by `indexed_chars`)

Or

```
PUT index/doc/2?pipeline=attachment
{
  "data": "BASE64",
  "size": 1000
}
```

Closes 
2018-03-14 19:07:20 +01:00
Jason Tedor 5904d936fa
Copy Lucene IOUtils ()
As we have factored Elasticsearch into smaller libraries, we have ended
up in a situation that some of the dependencies of Elasticsearch are not
available to code that depends on these smaller libraries but not server
Elasticsearch. This is a good thing, this was one of the goals of
separating Elasticsearch into smaller libraries, to shed some of the
dependencies from other components of the system. However, this now
means that simple utility methods from Lucene that we rely on are no
longer available everywhere. This commit copies IOUtils (with some small
formatting changes for our codebase) into the fold so that other
components of the system can rely on these methods where they no longer
depend on Lucene.
2018-03-13 12:49:33 -04:00
Daniel Mitterdorfer b2557b9c11
Skip GeoIpProcessorFactoryTests on Windows ()
With this commit we skip all GeoIpProcessorFactoryTests on Windows.
These tests use a MappedByteBuffer which will keep its file mappings
until it is garbage-collected. As a consequence, the corresponding
file appears to be still in use, Windows cannot delete it and the test
will fail in teardown.

Closes 
2018-03-13 09:10:40 +01:00
Tanguy Leroux 5a65db153e
[Test] GoogleCloudStorageFixture command line is too long on Windows ()
Windows has some strong limitations on command line arguments,
specially when it's too long. In the googlecloudstoragefixture anttask
the classpath argument is very long and the command fails. This commit
removes the classpath as an argument and uses the CLASSPATH
environment variable instead.
2018-03-12 18:02:30 +01:00
Daniel Mitterdorfer 0d78a5890e
Reduce heap-memory usage of ingest-geoip plugin ()
With this commit we reduce heap usage of the ingest-geoip plugin by
memory-mapping the database files. Previously, we have stored these
files gzip-compressed but this has resulted that data are loaded on the
heap.

Closes 
2018-03-12 08:07:33 +01:00
Tanguy Leroux d9cc6b9270 Remove temporary file 10_basic.yml~ 2018-03-09 17:44:10 +01:00
Tanguy Leroux 4756790d6e
Use fixture to test the repository-gcs plugin ()
This commit adds a GoogleCloudStorageFixture that uses the
logic of a GoogleCloudStorageTestServer (added in )
to emulate a remote Google Cloud Storage service.

By adding this fixture and a more complete integration test, we 
should be able to catch more bugs when upgrading the client library.

The fixture is started by the googleCloudStorageFixture task
and a custom Service Account file is created and added to the
Elasticsearch keystore for each test.
2018-03-09 13:57:27 +01:00
Tim Brooks 7d434c16f9
Remove NioNotEnabledBootstrapCheck bootstrap check ()
This is related to . This commit removes the bootstrap check that
prevents nio from being enabled.
2018-03-08 11:06:36 -07:00
Tim Brooks d8d1f0d4f0
Give transport-nio plugin socket permissions ()
This is related to . The transport-nio plugin needs socket
permissions to operate as a transport. This commit gives it these
permissions in the policy file.
2018-03-08 09:33:39 -07:00
Tim Brooks 5a8ec9b762
Selectors operate on channel contexts ()
This commit is related to . Currently there is a weird
relationship between channel contexts and nio channels. The selectors
use the context for read and writing. But the selector operates directly
on the nio channel for registering, closing, and connecting.

This commit works on improving this relationship. The selector operates
directly on the context which wraps the low level java.nio.channels. The
NioChannel class is simply an API that is used to interact with the
channel (sending messages from outside the selector event loop,
scheduling a close, adding listeners, etc). The context is only used
internally by the channel to implement these apis and by the selector to
perform these operations.
2018-02-22 09:44:52 -07:00
Tanguy Leroux a6a138905d
Use client settings in repository-gcs ()
Similarly to what has been done for s3 and azure, this commit removes
the repository settings `application_name` and `connect/read_timeout`
in favor of client settings. It introduce a GoogleCloudStorageClientSettings
class (similar to S3ClientSettings) and a bunch of unit tests for that,
it aligns the documentation to be more coherent with the S3 one, it
documents the connect/read timeouts that were not documented at all and
also adds a new client setting that allows to define a custom endpoint.
2018-02-22 15:40:20 +01:00
Tim Brooks de2a0dfa6e
Ensure that azure stream has socket privileges ()
This is related to . It wraps the azure repository inputstream in
an inputstream that ensures `read` calls have socket permissions. This
is because the azure inputstream internally makes service calls.
2018-02-21 11:20:06 -07:00
Tanguy Leroux 9a95be35cf
[Tests] Extract the testing logic for Google Cloud Storage ()
This pull request extracts in a dedicated class the request/response 
logic that "emulates" a Google Cloud Storage service in our 
repository-gcs tests.

The idea behind this is to make the logic more reusable. The class 
MockHttpTransport has been renamed to MockStorage which now 
only takes care of instantiating a Storage client and does the low-level 
request/response plumbing needed by this client.

The "Google Cloud Storage" logic has been extracted from 
MockHttpTransport and put in a new GoogleCloudStorageTestServer 
that is now independent from the google client testing framework.
2018-02-21 13:20:35 +01:00
Tanguy Leroux 9485b43167 [Tests] Fix RetryHttpInitializerWrapperTests.testIOExceptionRetry
This commit gives more time to the IO exception handler to retry the
request.
2018-02-20 14:54:53 +01:00
Tanguy Leroux 207ca1cc38
[Tests] Simplify GceDiscoverTests ()
GceDiscoverTests can be simplified in a similar manner than . It
now uses a mocked GceInstancesService that exposes internal test cluster
nodes as if they were real GCE nodes. It should also make the test more
robust by not using a HTTP server anymore.

closes 
2018-02-20 09:38:22 +01:00
Christoph Büscher 231fd3c9be [Docs] Remove misleading comment
The TikaImpl#parse method comment sounds like this method is only used
in the same package for testing, but AttachmentProcessor uses it outside
of testing, so we should remove this comment.
2018-02-09 15:47:38 +01:00
Jason Tedor 641a6c9e62
Guard accessDeclaredMembers for Tika on JDK 10
Tika parsers need accessDeclaredMembers because ZipFile needs
accessDeclaredMembers on JDK 10. This commit guards adding this
permission to parsers so that the permission is only granted on JDK
10. Additionally, we add an assertion that forces us to check if the
permission is still needed in JDK 11.

Relates 
2018-02-09 09:08:07 -05:00
Christoph Büscher cc9cb5356a
Add missing runtime permission to TikaImpl ()
Tests on jdk10 were failing because of a change in its ZipFile implementation 
that now needs `accessDeclaredMembers` permissions. This change adds 
the missing permission to the plugins security policy and TikaImpl.

Closes 
2018-02-09 14:41:24 +01:00
Lee Hinman eebff4d2b3
Use non deprecated xcontenthelper ()
* Move to non-deprecated XContentHelper.createParser(...)

This moves away from one of the now-deprecated XContentHelper.createParser
methods in favor of specifying the deprecation logger at parser creation time.

Relates to 

Note that this doesn't move all the `createParser` calls because some of them
use the already-deprecated method that doesn't specify the XContentType.

* Remove the deprecated (and now non-needed) createParser method
2018-02-05 16:18:18 -07:00
Tanguy Leroux be74f11517
Replace jvm-example by two plugin examples ()
This pull request replaces the jvm-example plugin (from the jvm/site plugins era) by two new plugins: a custom-settings that shows how to register and use custom settings (including secured settings) in a plugin, and rest-handler plugin that shows how to register a rest handler.

The two plugins now reside in the plugins/examples project. They can serve as sample plugins for users, a special attention has been put on documentation. The packaging tests have been adapted to use the custom-settings plugin.
2018-01-26 17:34:24 +01:00
kel c675407a70 Remove redundant argument for buildConfiguration of s3 plugin () 2018-01-23 22:32:46 -08:00
Adrien Grand 700d9ecc95
Remove the `update_all_types` option. ()
This option is not useful in 7.x since no indices may have more than one type
anymore.
2018-01-22 12:03:07 +01:00
Tim Brooks a6a57a71d3
Implement socket and server ChannelContexts ()
This commit is related to . Currently have a channel context that
implements reading and writing logic for socket channels. Additionally,
we have exception contexts to handle exceptions. And accepting contexts
to handle accepted channels. This PR introduces a ChannelContext that
handles close and exception handling for all channel types.
Additionally, it has implementers that provide specific functionality
for socket channels (read and writing). And specific functionality for
server channels (accepting).
2018-01-18 13:06:40 -07:00
Ryan Ernst cefea1a7c9
Build: Add gradle plugin for configuring meta plugin ()
This commit adds a gradle plugin to ease development of meta plugins.
Applying the plugin will generated the meta plugin properties based on
the es_meta_plugin configuration object, which includes name and
description. The plugins to include within the meta plugin are
configured through the `plugins` list. An integ test task is also
automatically added.
2018-01-17 19:47:37 -08:00
Tim Brooks 4ea9ddb7d3
Unify nio read / write channel contexts ()
This commit is related to . Right now we have separate read and
write contexts for implementing specific protocol logic. However, some
protocols require a closer relationship between read and write
operations than is allowed by our current model. An example is HTTP
which might require a write if some problem with request parsing was
encountered.

Additionally, some protocols require close messages to be sent when a
channel is shutdown. This is also problematic in our current model,
where we assume that channels should simply be queued for close and
forgotten.

This commit transitions to a single ChannelContext which implements
all read, write, and close logic for protocols. It is the job of the
context to tell the selector when to close the channel. A channel can
still be manually queued for close with a selector. This is how server
channels are closed for now. And this route allows timeout mechanisms on
normal channel closes to be implemented.
2018-01-17 09:44:21 -07:00
Jason Tedor aded32f48f
Fix third-party audit tasks on JDK 8
This one is interesting. The third party audit task runs inside the
Gradle JVM. This means that if Gradle is started on JDK 8, the third
party audit tasks will fail as a result of the changes to support
building Elasticsearch with the JDK 9 compiler. This commit reverts the
third party audit changes to support running this task when Gradle is
started with JDK 8.

Relates 
2018-01-16 22:59:29 -05:00
Jason Tedor 0a79555a12
Require JDK 9 for compilation ()
This commit modifies the build to require JDK 9 for
compilation. Henceforth, we will compile with a JDK 9 compiler targeting
JDK 8 as the class file format. Optionally, RUNTIME_JAVA_HOME can be set
as the runtime JDK used for running tests. To enable this change, we
separate the meaning of the compiler Java home versus the runtime Java
home. If the runtime Java home is not set (via RUNTIME_JAVA_HOME) then
we fallback to using JAVA_HOME as the runtime Java home. This enables:
 - developers only have to set one Java home (JAVA_HOME)
 - developers can set an optional Java home (RUNTIME_JAVA_HOME) to test
   on the minimum supported runtime
 - we can test compiling with JDK 9 running on JDK 8 and compiling with
   JDK 9 running on JDK 9 in CI
2018-01-16 13:45:13 -05:00
Ryan Ernst 18463e7e9f
Painless: Add whitelist extensions ()
This commit adds a PainlessExtension which may be plugged in via SPI to
add additional classes, methods and members to the painless whitelist on
a per context basis. An example plugin adding and using a whitelist is
also added.
2018-01-15 11:28:31 -08:00
Jim Ferenczi b82017cbfe
Fix daitch_mokotoff phonetic filter to use the dedicated Lucene filter ()
This commit changes the phonetic filter factory to use a DaitchMokotoffSoundexFilter
instead of a PhoneticFilter with a daitch_mokotoff encoder when daitch_mokotoff is selected.
The latter does not hanlde branching when computing the soundex and fails to encode multiple
variations when possible.

Closes 
2018-01-15 19:35:54 +01:00
Tim Brooks ee7eac8dc1
`MockTcpTransport` to connect asynchronously ()
The method `initiateChannel` on `TcpTransport` is explicit in that
channels can be connect asynchronously. All production implementations
do connect asynchronously. Only the blocking `MockTcpTransport`
connects in a synchronous manner. This avoids testing some of the
blocking code in `TcpTransport` that waits on connections to complete.
Additionally, it requires a more extensive method signature than
required for other transports.

This commit modifies the `MockTcpTransport` to make these connections
asynchronously on a different thread. Additionally, it simplifies that
`initiateChannel` method signature.
2018-01-15 10:20:30 -07:00
Jim Ferenczi be012b1326
upgrade to lucene 7.2.1 () 2018-01-15 16:47:46 +01:00
Igor Motov c75ac319a6
Add ability to associate an ID with tasks ()
Adds support for capturing the X-Opaque-Id header from a REST request and storing it's value in the tasks that this request started. It works for all user-initiated tasks (not only search).

Closes 

Usage:
```
$ curl -H "X-Opaque-Id: imotov" -H "foo:bar" "localhost:9200/_tasks?pretty&group_by=parents"
{
  "tasks" : {
    "7qrTVbiDQKiZfubUP7DPkg:6998" : {
      "node" : "7qrTVbiDQKiZfubUP7DPkg",
      "id" : 6998,
      "type" : "transport",
      "action" : "cluster:monitor/tasks/lists",
      "start_time_in_millis" : 1513029940042,
      "running_time_in_nanos" : 266794,
      "cancellable" : false,
      "headers" : {
        "X-Opaque-Id" : "imotov"
      },
      "children" : [
        {
          "node" : "V-PuCjPhRp2ryuEsNw6V1g",
          "id" : 6088,
          "type" : "netty",
          "action" : "cluster:monitor/tasks/lists[n]",
          "start_time_in_millis" : 1513029940043,
          "running_time_in_nanos" : 67785,
          "cancellable" : false,
          "parent_task_id" : "7qrTVbiDQKiZfubUP7DPkg:6998",
          "headers" : {
            "X-Opaque-Id" : "imotov"
          }
        },
        {
          "node" : "7qrTVbiDQKiZfubUP7DPkg",
          "id" : 6999,
          "type" : "direct",
          "action" : "cluster:monitor/tasks/lists[n]",
          "start_time_in_millis" : 1513029940043,
          "running_time_in_nanos" : 98754,
          "cancellable" : false,
          "parent_task_id" : "7qrTVbiDQKiZfubUP7DPkg:6998",
          "headers" : {
            "X-Opaque-Id" : "imotov"
          }
        }
      ]
    }
  }
}
```
2018-01-12 15:34:17 -05:00
Jim Ferenczi fcf4114adc
Make sure that we don't detect files as maven coordinate when installing a plugin ()
* This change makes sure that we don't detect a file path containing a ':' as
a maven coordinate (e.g.: `file:C:\path\to\zip`)

* restore test muted on master
2018-01-10 14:59:37 +01:00
Jim Ferenczi ca6b15bf7c [Tests] temporary disable meta plugin rest tests 2018-01-10 14:31:07 +01:00
Jim Ferenczi 36729d1c46
Add the ability to bundle multiple plugins into a meta plugin ()
This commit adds the ability to package multiple plugins in a single zip.
The zip file for a meta plugin must contains the following structure:

|____elasticsearch/
| |____   <plugin1> <-- The plugin files for plugin1 (the content of the elastisearch directory)
| |____   <plugin2>  <-- The plugin files for plugin2
| |____   meta-plugin-descriptor.properties <-- example contents below
The meta plugin properties descriptor is mandatory and must contain the following properties:

description: simple summary of the meta plugin.
name: the meta plugin name
The installation process installs each plugin in a sub-folder inside the meta plugin directory.
The example above would create the following structure in the plugins directory:

|_____ plugins
| |____   <name_of_the_meta_plugin>
| | |____   meta-plugin-descriptor.properties
| | |____   <plugin1>
| | |____   <plugin2>
If the sub plugins contain a config or a bin directory, they are copied in a sub folder inside the meta plugin config/bin directory.

|_____ config
| |____   <name_of_the_meta_plugin>
| | |____   <plugin1>
| | |____   <plugin2>

|_____ bin
| |____   <name_of_the_meta_plugin>
| | |____   <plugin1>
| | |____   <plugin2>
The sub-plugins are loaded at startup like normal plugins with the same restrictions; they have a separate class loader and a sub-plugin
cannot have the same name than another plugin (or a sub-plugin inside another meta plugin).

It is also not possible to remove a sub-plugin inside a meta plugin, only full removal of the meta plugin is allowed.

Closes 
2018-01-09 18:28:43 +01:00
Tim Brooks ff3db0b50e
Cleanup TcpChannelFactory and remove classes ()
This commit is related to . It moves the TcpChannelFactory into
NioTransport so that consumers do not have to be passed around.
Additionally it deletes an unused read handler.
2018-01-08 10:18:19 -07:00
Martijn van Groningen b46bb2efae
test: do not use asn fields
Closes 
2018-01-07 23:21:01 +01:00
Tim Brooks 38701fb6ee
Create nio-transport plugin for NioTransport ()
This is related to . This commit moves the NioTransport from
:test:framework to a new nio-transport plugin. Additionally, supporting
tcp decoding classes are moved to this plugin. Generic byte reading and
writing contexts are moved to the nio library.

Additionally, this commit adds a basic MockNioTransport to
:test:framework that is a TcpTransport implementation for testing that
is driven by nio.
2018-01-05 09:41:29 -07:00
Martijn van Groningen fdb9b50747
test: replaced try-catch statements with expectThrows(...) 2018-01-05 14:29:53 +01:00
Sian Lerk Lau a4a7150b56
Added ASN support for Ingest GeoIP plugin.
Closes 
2018-01-05 14:07:04 +01:00
Ryan Ernst d36ec18029
Plugins: Add plugin extension capabilities ()
This commit adds the infrastructure to plugin building and loading to
allow one plugin to extend another. That is, one plugin may extend
another by the "parent" plugin allowing itself to be extended through
java SPI. When all plugins extending a plugin are finished loading, the
"parent" plugin has a callback (through the ExtensiblePlugin interface)
allowing it to reload SPI.

This commit also adds an example plugin which uses as-yet implemented
extensibility (adding to the painless whitelist).
2018-01-03 11:12:43 -08:00
Tanguy Leroux 098f82f086
[Test] Do not rely on MockZenPing for Azure tests ()
This commit changes some Azure tests so that they do not rely on
MockZenPing and TestZenDiscovery anymore, but instead use a mocked
AzureComputeService that exposes internal test cluster nodes as if
they were real Azure nodes.

Related to 

Closes , 
2017-12-22 09:58:02 +01:00
Martijn van Groningen 90ee35930a
ingest: upgraded ingest geoip's geoip2's dependencies. 2017-12-21 08:43:02 +01:00
Nhat Nguyen 3c865d6d04 TEST: reduce blob size #testExecuteMultipartUpload
If a large blob size and small buffer size are picked, this test causes out of memory.

https://elasticsearch-ci.elastic.co/job/elastic+elasticsearch+master+intake/1061/
2017-12-20 12:43:05 -05:00
Adrien Grand 77711508b0
Upgrade to Lucene 7.2.0. () 2017-12-20 14:17:40 +01:00
Boaz Leskes 0ca141880f Disable TestZenDiscovery in cloud providers integrations test
TestZenDiscovery is used to allow discovery based on in memory structures. This isn't a relevant for the cloud providers tests (but isn't a problem at the moment either)
2017-12-20 14:02:55 +01:00
Martijn van Groningen 4585cc8312
ingest: Upgraded the geolite2 databases. 2017-12-20 10:42:46 +01:00
Tal Levy 43ff38c5da
update ingest-attachment to use Tika 1.17 and newer deps ()
- this pr updates tika and its dependencies
- updates the SHAs
- updates the class excludes
2017-12-15 13:47:26 -08:00
Colin Goodheart-Smithe 579d1fea57
Fixes ByteSizeValue to serialise correctly ()
* Fixes ByteSizeValue to serialise correctly

This fix makes a few fixes to ByteSizeValue to make it possible to perform round-trip serialisation:
* Changes wire serialisation to use Zlong methods instead of VLong methods. This is needed because the value `-1` is accepted but previously if `-1` is supplied it cannot be serialised using the wire protocol.
* Limits the supplied size to be no more than Long.MAX_VALUE when converted to bytes. Previously values greater than Long.MAX_VALUE bytes were accepted but would be silently interpreted as Long.MAX_VALUE bytes rather than erroring so the user had no idea the value was not being used the way they had intended. I consider this a bug and so fine to include this bug fix in a minor version but I am open to other points of view.
* Adds a `getStringRep()` method that can be used when serialising the value to JSON. This will print the bytes value if the size is positive, `”0”` if the size is `0` and `”-1”` if the size is `-1`.
* Adds logic to detect fractional values when parsing from a String and emits a deprecation warning in this case.
* Modifies hashCode and equals methods to work with long values rather than doubles so they don’t run into precision problems when dealing with large values. Previous to this change the equals method would not detect small differences in the values (e.g. 1-1000 bytes ranges) if the actual values where very large (e.g. PBs). This was due to the values being in the order of 10^18 but doubles only maintaining a precision of ~10^15.

Closes 

* Fix bytes settings default value to not use fractional values

* Fixes test

* Addresses review comments

* Modifies parsing to preserve unit

This should be bwc since in the case that the input is fractional it reverts back to the old method of parsing it to the bytes value.

* Addresses more review comments

* Fixes tests

* Temporarily changes version check to 7.0.0

This will be changed to 6.2 when the fix has been backported
2017-12-14 12:17:17 +00:00
Tanguy Leroux b69923f112
Remove some unused code ()
This commit removes some unused code.
2017-12-13 16:45:55 +01:00
Tanguy Leroux f27cb96a64
Use AmazonS3.doesObjectExist() method in S3BlobContainer ()
This pull request changes the S3BlobContainer.blobExists() method implementation 
to make it use the AmazonS3.doesObjectExist() method instead of 
AmazonS3.getObjectMetadata(). The AmazonS3 implementation takes care of 
catching any thrown AmazonS3Exception and compares its response code with 404, 
returning false (object does not exist) or lets the exception be propagated.
2017-12-12 09:30:36 +01:00
Luca Cavanna f4fb4d3bf5
Add support for filtering mappings fields ()
Add support for filtering fields returned as part of mappings in get index, get mappings, get field mappings and field capabilities API.

Plugins can plug in their own function, which receives the index as argument, and return a predicate which controls whether each field is included or not in the returned output.
2017-12-05 20:31:29 +01:00
Jason Tedor 42a4ad35da
Add node name to thread pool executor name
This commit adds the node name to the names of thread pool executors so
that the node name is visible in rejected execution exception messages.

Relates 
2017-12-05 07:45:40 -05:00
Henrik Lindström 7a44596446 Catch InvalidPathException in IcuCollationTokenFilterFactory ()
Using custom rules in the icu_collation filter can fail on Windows. If the rules are interpreted 
as a file location, this leads to an InvalidPathException when trying to read the rules from a file.
2017-12-04 10:29:08 +01:00
Adrien Grand 6323bb0d97
Upgrade to lucene-7.2.0-snapshot-8c94404. ()
This new snapshot mostly brings a change to TopFieldCollector which can now
early terminate collection when trackTotalHits is `false`.

As a follow-up, we should replace our usage of
`EarlyTerminatingSortingCollector` with this new option.
2017-12-04 09:40:08 +01:00
James Baiera e16f1271b6
Fix SecurityException when HDFS Repository used against HA Namenodes ()
* Sense HA HDFS settings and remove permission restrictions during regular execution.

This PR adds integration tests for HA-Enabled HDFS deployments, both regular and secured. 
The Mini HDFS fixture has been updated to optionally run in HA-Mode. A new test suite has 
been added for reproducing the effects of a Namenode failing over during regular repository 
usage. Going forward, the HDFS Repository will still be subject to its self imposed permission 
restrictions during normal use, but will no longer restrict them when running against an HA 
enabled HDFS cluster. Instead, the plugin will rely on the provided security policy and not 
further restrict the permissions so that the transparent operation to failover to a different 
Namenode in the client does not raise security exceptions. Additionally, we are now testing the 
secure mode with SASL based wire encryption of data between Elasticsearch and HDFS. This 
includes a missing library (commons codec) in order to support this change.
2017-12-01 14:26:05 -05:00
Jason Tedor d0cd18169e Remove stale awaits fix on azure master nodes test
This awaits fix has been there forever and no one seems to know what to
do with this test. I say let CI churn on it because it passed for me
three out of three times. If there is something wrong with it, we will
know quickly and can then address with the new information that we have.
2017-11-28 22:43:34 -05:00
Adrien Grand 996990ad1f
Upgrade to lucene-7.2.0-snapshot-8c94404. ()
The main highlight of this new snapshot is that it introduces the opportunity
for queries to opt out of caching. In case a query opts out of caching, not only
will it never be cached, but also no compound query that wraps it will be
cached.
2017-11-28 14:52:42 +01:00
David Turner 7ac361d86e Update @AwaitsFix URL to point at an issue in the current repo 2017-11-28 09:35:46 +00:00
Tanguy Leroux 50a2459adf
Update Google SDK to version 1.23 ()
This commit updates the google-api-client library to version 1.23.0.

Related to 
2017-11-15 15:30:27 +01:00
Tanguy Leroux dd51c231ac
Create new handlers for every new request in GoogleCloudStorageService ()
This commit changes the DefaultHttpRequestInitializer in order to make
it create new HttpIOExceptionHandler and HttpUnsuccessfulResponseHandler
for every new HTTP request instead of reusing the same two handlers for
all requests.

Closes 
2017-11-14 11:43:32 +01:00
Jason Tedor d375cef73c
Upgrade AWS SDK Jackson Databind to 2.6.7.1
The AWS SDK has a transitive dependency on Jackson Databind. While the
AWS SDK was recently upgraded, the Jackson Databind dependency was not
pulled along with it to the version that the AWS SDK depends on. This
commit upgrades the dependencies for discovery-ec2 and repository-s3
plugins to match versions on the AWS SDK transitive dependencies.

Relates 
2017-11-13 12:05:14 -05:00
Simon Willnauer 2299c70371
Allow affix settings to specify dependencies ()
We use affix settings to group settings / values under a certain namespace.
In some cases like login information for instance a setting is only valid if
one or more other settings are present. For instance `x.test.user` is only valid
if there is an `x.test.passwd` present and vice versa. This change allows to specify
such a dependency to prevent settings updates that leave settings in an inconsistent
state.
2017-11-13 12:06:36 +01:00
Tanguy Leroux f6c2ea0f7d [Test] Fix S3BlobStoreContainerTests.testNumberOfMultiparts() 2017-11-10 15:45:20 +01:00
Tanguy Leroux 9c4d6c629a
Remove S3 output stream ()
Now the blob size information is available before writing anything, 
the repository implementation can know upfront what will be the 
more suitable API to upload the blob to S3.

This commit removes the DefaultS3OutputStream and S3OutputStream 
classes and moves the implementation of the upload logic directly in the 
S3BlobContainer.

related 
closes 
2017-11-10 12:22:33 +01:00
Guillaume Le Floch ac5fd6a7d9 Update Tika version to 1.15
This commit upgrades the Tika dependency to version 1.15.

Relates 
2017-11-09 13:16:44 -05:00
Tanguy Leroux 184dda9eb0
Update to AWS SDK 1.11.223 () 2017-11-09 13:25:51 +01:00
Jason Tedor 58a28dacbd
Remove colons from task and configuration names
Gradle 5.0 will remove support for colons in configuration and task
names. This commit fixes this for our build by removing all current uses
of colons in configuration and task names.

Relates 
2017-11-08 15:22:31 -05:00
David Roberts 749c3ec716
Remove the single argument Environment constructor ()
Only tests should use the single argument Environment constructor.  To
enforce this the single arg Environment constructor has been replaced with
a test framework factory method.

Production code (beyond initial Bootstrap) should always use the same
Environment object that Node.getEnvironment() returns.  This Environment
is also available via dependency injection.
2017-11-04 13:25:09 +00:00
kel 0f21262b36 Do not create directories if repository is readonly ()
For FsBlobStore and HdfsBlobStore, if the repository is read only, the blob store should be aware of the readonly setting and do not create directories if they don't exist.

Closes 
2017-11-03 13:10:50 +01:00
Colin Goodheart-Smithe c1b8140c83
Upgrade to Lucene 7.1 () 2017-11-02 13:25:33 +00:00
Colin Goodheart-Smithe 99aca9cdfc
Enhances exists queries to reduce need for `_field_names` ()
* Enhances exists queries to reduce need for `_field_names`

Before this change we wrote the name all the fields in a document to a `_field_names` field and then implemented exists queries as a term query on this field. The problem with this approach is that it bloats the index and also affects indexing performance.

This change adds a new method `existsQuery()` to `MappedFieldType` which is implemented by each sub-class. For most field types if doc values are available a `DocValuesFieldExistsQuery` is used, falling back to using `_field_names` if doc values are disabled. Note that only fields where no doc values are available are written to `_field_names`.

Closes 

* Addresses review comments

* Addresses more review comments

* implements existsQuery explicitly on every mapper

* Reinstates ability to perform term query on `_field_names`

* Added bwc depending on index created version

* Review Comments

* Skips tests that are not supported in 6.1.0

These values will need to be changed after backporting this PR to 6.x
2017-11-01 10:46:59 +00:00
Christoph Büscher 9253ea8aec Fix beidermorse phonetic token filter for unspecified `languageset` ()
Currently, when we create a BeiderMorseFilter with an unspecified `languageset`,
the filter will not guess the language, which should be the default behaviour.
This change fixes this and adds a simple test for the cases with and without
provided `languageset` settings.

Closes 
2017-10-27 10:07:36 +02:00
Tanguy Leroux 463e7e6fa3 Revert "Upgrade to Jackson 2.9.2 ()"
This reverts commit 0b9acc5ace.
2017-10-20 08:25:41 +02:00
Tanguy Leroux f78b2e5bc9 Fix ingest-attachment yaml REST test 2017-10-19 22:20:50 +02:00
Simon Willnauer cdd7c1e6c2 Return List instead of an array from settings ()
Today we return a `String[]` that requires copying values for every
access. Yet, we already store the setting as a list so we can also directly
return the unmodifiable list directly. This makes list / array access in settings
a much cheaper operation especially if lists are large.
2017-10-09 09:52:08 +02:00
Md. Abdulla-Al-Sun a40c474e10
Added Bengali Analyzer to Elasticsearch with respect to the lucene update(PR#238) 2017-10-05 13:25:05 +02:00
Simon Willnauer 00dfdf50cf Represent lists as actual lists inside Settings ()
Today we represent each value of a list setting with it's own dedicated key
that ends with the index of the value in the list. Aside of the obvious
weirdness this has several issues especially if lists are massive since it
causes massive runtime penalties when validating settings. Like a list of 100k
words will literally cause a create index call to timeout and in-turn massive
slowdown on all subsequent validations runs.

With this change we use a simple string list to represent the list. This change
also forbids to add a settings that ends with a .0 which was internally used to
detect a list setting.  Once this has been rolled out for an entire major
version all the internal .0 handling can be removed since all settings will be
converted.

Relates to 
2017-10-05 09:27:08 +02:00
Martijn van Groningen dca787ed8a
upgrade to Lucene 7.1.0 snapshot version 2017-10-05 09:06:56 +02:00
David Pilato 84a3899550 Simplify Azure blobStore method signatures ()
While working on , I found that we are passing the container name on every single method although we don't need it as it is stored within the blobstore object already.

This commit simplifies a bit that part of the code.

It also removes `repositoryName` from AzureBlobStore which was not used anymore.
Also we move some properties in AzureBlobContainer to `private` members.
2017-10-04 20:17:50 +02:00
Simon Willnauer d1533e2397 Remove Settings#getAsMap() ()
Since `#getAsMap` exposes internal representation we are trying to remove it
step by step. This commit is cleaning up some xcontent writing as well as
usage in tests
2017-10-04 01:21:38 -06:00
Simon Willnauer 7b8d036ab5 Replace group map settings with affix setting ()
We use group settings historically instead of using a prefix setting which is more restrictive and type safe. The majority of the usecases needs to access a key, value map based on the _leave node_ of the setting ie. the setting `index.tag.*` might be used to tag an index with `index.tag.test=42` and `index.tag.staging=12` which then would be turned into a `{"test": 42, "staging": 12}` map. The group settings would always use `Settings#getAsMap` which is loosing type information and uses internal representation of the settings. Using prefix settings allows now to access such a method type-safe and natively.
2017-09-30 14:27:21 +02:00
David Pilato 9ba5e168e4 Don't use create static storage service
Even though you annotate the Test class with `@ThirdParty` the static
code is initialized.

In that case it fails with:

```
==> Test Info: seed=529C3C6977F695FC; jvms=3; suites=6
Suite: org.elasticsearch.repositories.azure.AzureSnapshotRestoreTests
ERROR   0.00s J2 | AzureSnapshotRestoreTests (suite) <<< FAILURES!
   > Throwable : java.lang.IllegalStateException: to run integration tests, you need to set -Dtests.thirdparty=true and -Dtests.azure.account=azure-account -Dtests.azure.key=azure-key
   >    at org.elasticsearch.cloud.azure.AzureTestUtils.generateMockSecureSettings(AzureTestUtils.java:37)
   >    at org.elasticsearch.repositories.azure.AzureSnapshotRestoreTests.generateMockSettings(AzureSnapshotRestoreTests.java:81)
   >    at org.elasticsearch.repositories.azure.AzureSnapshotRestoreTests.<clinit>(AzureSnapshotRestoreTests.java:84)
   >    at java.lang.Class.forName0(Native Method)
   >    at java.lang.Class.forName(Class.java:348)
Completed [1/6] on J2 in 2.21s, 0 tests, 1 error <<< FAILURES!
```

Closes .

(cherry picked from commit eb6d714 for master branch)
2017-09-28 16:40:16 +01:00
Hendrik Muhs 0358bb5f34 fix of disabling 2017-09-28 16:14:34 +02:00
Hendrik Muhs bf4d3123bd disable AzureSnapshotRestoreTests, see 2017-09-28 15:42:53 +02:00
David Pilato 1ccb497c0d Use Azure upload method instead of our own implementation ()
* Use Azure upload method instead of our own implementation

We are not following the Azure documentation about uploading blobs to Azure storage. https://docs.microsoft.com/en-us/azure/storage/blobs/storage-java-how-to-use-blob-storage#upload-a-blob-into-a-container

Instead we are using our own implementation which might cause some troubles and rarely some blobs can be not immediately commited just after we close the stream. Using the standard implementation provided by Azure team should allow us to benefit from all the magic Azure SDK team already wrote.

And well... Let's just read the doc!

* Adapt integration tests to secure settings

That was a missing part in .

* Simplify all the integration tests and *extends ESBlobStoreRepositoryIntegTestCase tests

    * removes IT `testForbiddenContainerName()` as it is useless. The plugin does not create anymore the container but expects that the user has created it before registering the repository
   * merges 2 IT classes so all IT tests are ran from one single class
   * We don't remove/create anymore the container between each single test but only for the test suite
2017-09-28 13:15:37 +02:00
olcbean 6952f7b560 Validate top-level keys for create index request () ()
This commit ensures create index requests do not ignore unknown keys passed to the request.

closes 
2017-09-26 09:49:20 -07:00
David Pilato 3f71772cd2 Azure snapshots can not be restored anymore ()
While working on  and doing some manual integration testing I found that this  removed an important line of our code:

`AzureRepository` overrides default `initializeSnapshot` method which creates metadata files and do other stuff.

But with PR , I wrote:

```java
    @Override
    public void initializeSnapshot(SnapshotId snapshotId, List<IndexId> indices, MetaData clusterMetadata) {
        if (blobStore.doesContainerExist(blobStore.container()) == false) {
            throw new IllegalArgumentException("The bucket [" + blobStore.container() + "] does not exist. Please create it before " +
                " creating an azure snapshot repository backed by it.");
        }
    }
```

instead of

```java
    @Override
    public void initializeSnapshot(SnapshotId snapshotId, List<IndexId> indices, MetaData clusterMetadata) {
        if (blobStore.doesContainerExist(blobStore.container()) == false) {
            throw new IllegalArgumentException("The bucket [" + blobStore.container() + "] does not exist. Please create it before " +
                " creating an azure snapshot repository backed by it.");
        }
        super.initializeSnapshot(snapshotId, indices, clusterMetadata);
    }
```

As we never call `super.initializeSnapshot(...)` files are not created and we can't restore what we saved.

Closes .
2017-09-25 14:35:27 -04:00
Simon Willnauer aab4655e63 Unify Settings xcontent reading and writing ()
This change adds a fromXContent method to Settings that allows to read
the xcontent that is produced by toXContent. It also replaces the entire settings
loader infrastructure and removes the structured map representation. Future PRs will
also tackle the `getAsMap` that exposes the internal represenation of settings for
better encapsulation.
2017-09-25 13:23:01 +02:00
Jason Tedor 2e63a13c0a Upgrade to Log4j 2.9.1
This commit upgrades the Log4j dependency, picking up a fix for an issue
with handling stack traces on JDK 9.

Relates 
2017-09-22 11:57:06 -04:00
Jason Tedor e0db89bc35 Upgrade to Lucene 7.0.0
This commit upgrades to the GA release of Luence 7!

Relates 
2017-09-21 19:19:33 -04:00
James Baiera c760eec054 Add permission checks before reading from HDFS stream ()
Add checks for special permissions before reading hdfs stream data. Also adds test from 
readonly repository fix. MiniHDFS will now start with an existing repository with a single snapshot 
contained within. Readonly Repository is created in tests and attempts to list the snapshots 
within this repo.
2017-09-21 11:55:07 -04:00
kel 601be4f83e Add azure storage endpoint suffix ()
Allow specifying azure storage endpoint suffix for an azure client.
2017-09-20 22:26:19 -07:00
Ryan Ernst bebff47b5b File Discovery: Remove fallback with zen discovery ()
When adding file based discovery, we added a fallback when the discovery
type was set to zen (the default, so everyone got this warning). This
commit removes the fallback for 6.0. Setting file discovery should now
happen explicitly through the hosts_provider setting.

closes 
2017-09-19 16:32:34 -07:00
Jason Tedor bdd9953aa4 Fix discovery-file plugin to use custom config path
The discovery-file plugin was not config path aware, so it always picked
up the default config path (from Elasticsearch home) rather than a
custom config path. This commit fixes the discovery-file plugin to
respect a custom config path.

Relates 
2017-09-16 11:00:33 -04:00
Claudio Bley 7184cf8b5b Fix kuromoji default stoptags ()
Initialize the default stop-tags in `KuromojiPartOfSpeechFilterFactory` if the
`stoptags` are not given in the config. Also adding a test which checks that 
part-of-speech tokens are removed when using the kuromoji_part_of_speech 
filter.
2017-09-15 12:25:09 +02:00
Christoph Büscher c7c6443b10 [Docs] "The the" is a great band, but ... ()
Removing several occurrences of this typo in the docs and javadocs, seems to be
a common mistake. Corrections turn up once in a while in PRs, better to correct
some of this in one sweep.
2017-09-14 15:08:20 +02:00
David Pilato b6c6effa2a Move all repository-azure classes under one single package ()
As we did for S3, we can collapse all packages within one single `org.elasticsearch.repositories.azure` package name.

Follow up for https://github.com/elastic/elasticsearch/pull/23518#issuecomment-328903585
2017-09-14 11:56:02 +02:00
David Pilato a34db4e09f Support for accessing Azure repositories through a proxy ()
You can define a proxy using the following settings:

```yml
azure.client.default.proxy.host: proxy.host
azure.client.default.proxy.port: 8888
azure.client.default.proxy.type: http
```

Supported values for `proxy.type` are `direct`, `http` or `socks`. Defaults to `direct` (no proxy).

Closes 

BTW I changed a test `testGetSelectedClientBackoffPolicyNbRetries` as it was using an old setting name `cloud.azure.storage.azure.max_retries` instead of `azure.client.azure1.max_retries`.
2017-09-13 11:51:55 +02:00
David Pilato b01b1c2a58 Remove azure deprecated settings ()
Follow up for .

We remove azure deprecated settings in 7.0:

* The legacy azure settings which where starting with `cloud.azure.storage.` prefix have been removed.
This includes `account`, `key`, `default` and `timeout`.
You need to use settings which are starting with `azure.client.` prefix instead.

* Global timeout setting `cloud.azure.storage.timeout` has been removed.
You must set it per azure client instead. Like `azure.client.default.timeout: 10s` for example.
2017-09-12 16:51:44 +02:00
mohit 06150d40a2 update AWS SDK for ECS Task IAM support in discovery-ec2 ()
This commit contains:

* update AWS SDK for ECS Task IAM support
* ignore dependencies not essential to `discovery-ec2`:
  * jmespath seems to be used for `waiters`
  * amazon ion is a protocol not used by EC2 or IAM
2017-09-12 10:34:12 +02:00
etiennecarriere 706067211a Azure repository: Accelerate the listing of files (used in delete snapshot) ()
This commit reworks the azure listing of snapshot files to do a single listing, instead of once per blob.

closes 
2017-09-11 16:09:27 -07:00
Adrien Grand 1adee8b5a8 Fix the MapperFieldType.rangeQuery API. ()
RangeQueryBuilder needs to perform too many `instanceof` checks in order to
check for `date` or `range` fields in order to know what it should do with the
shape relation, time zone and date format.

This commit adds those 3 parameters to the `rangeQuery` factory method so that
those instanceof checks are not necessary anymore.
2017-09-11 11:02:05 +02:00
Andy Bristol 33faf5ec70 forbid ICU Collator creation with default locale () 2017-09-07 14:47:52 -07:00
Jason Tedor f6a489f323 Add Log4j to SLF4J binding for repository-hdfs
This commit adds the Log4j to SLF4J binding JAR to the repository-hdfs
plugin so that SLF4J can detect Log4j at runtime and therefore use the
server Log4j implementation for logging (and the usual Elasticsearch
APIs can be used for setting logging levels).

Relates 
2017-09-05 19:38:17 -04:00
Adrien Grand 78681bc9e5 Upgrade to lucene-7.0.0-snapshot-d94a5f0. () 2017-08-31 09:06:40 +02:00
Andy Bristol e00366ba95 ICU plugin: use root locale by default for collators ()
Calls to Collator.getInstance without arguments returns a
collator that uses the system's default locale, which we don't
want because it makes behavior harder to reproduce. Change it
to always use the root locale instead.

For 
2017-08-29 08:58:36 -07:00
Jim Ferenczi 86d97971a4 Remove the _all metadata field ()
* Remove the _all metadata field

This change removes the `_all` metadata field. This field is deprecated in 6
and cannot be activated for indices created in 6 so it can be safely removed in
the next major version (e.g. 7).
2017-08-28 17:43:59 +02:00
Adrien Grand eb782492be Remove support for lenient booleans.
Closes 
2017-08-28 09:56:01 +02:00
Nik Everett b3edd11aa0 Allow plugins to plug rescore implementations ()
This allows plugins to plug rescore implementations into
Elasticsearch. While this is a fairly expert thing to do I've
done my best to point folks to the QueryRescorer as one that at
least documents the tradeoffs that it makes. I've attempted to
limit the API surface area by removing `SearchContext` from the
exposed interface, instead exposing just the IndexSearcher and
`QueryShardContext`. I also tried to make some of the class names
more consistent and do some general cleanup while I was there.

I entertained the notion of moving the `QueryRescorer` to module.
After all, it'd be a wonderful test to prove that you can plug
rescore implementation into Elasticsearch if the only built in
rescore implementation is in the module. But I decided against it
because the new module would require a client jar and it'd require
moving some more things around. I think if we really want to do
it, we should do it as a followup.

I did, on the other hand, create an "example" rescore plugin which
should both be a nice example for anyone wanting to plug in their
own rescore implementation and servers as a good integration test
to make sure that you can indeed plug one in.

Closes 
2017-08-25 13:46:57 -04:00
Yannick Welsch 3d8feff66e Use Java 9 FilePermission model ()
This commit makes the security code aware of the Java 9 FilePermission changes (see ) and allows us to remove the `jdk.io.permissionsUseCanonicalPath` system property.
2017-08-22 11:22:00 +09:30
Matt Weber e89d9400c9 ICUCollationKeywordFieldMapper use SortedSetDocValuesField ()
Switch ICUCollationKeywordFieldMapper from using SortedDocValuesField to SortedSetDocValuesField
so we can support fields with multiple values.
2017-08-21 10:40:56 +02:00
desmorto 292dd8f992 (refactor) some opportunities to use diamond operator ()
* (refactor) some opportunities to use diamond operator

* Update ExceptionRetryIT.java

update typo
2017-08-15 16:36:42 -06:00
David Pilato 80b142d218 Azure repository: Move to named configurations as we do for S3 repository
We should have the same behavior for Azure repositories as we have for S3 (see ).

Instead of:

```yml
cloud:
    azure:
        storage:
            my_account1:
                account: your_azure_storage_account1
                key: your_azure_storage_key1
                default: true
            my_account2:
                account: your_azure_storage_account2
                key: your_azure_storage_key2
```

Support something like:

```
azure.client:
            default:
                account: your_azure_storage_account1
                key: your_azure_storage_key1
            my_account2:
                account: your_azure_storage_account2
                key: your_azure_storage_key2
```

Then instead of:

```
PUT _snapshot/my_backup3
{
    "type": "azure",
    "settings": {
        "account": "my_account2"
    }
}
```

Use:

```
PUT _snapshot/my_backup3
{
    "type": "azure",
    "settings": {
        "config": "my_account2"
    }
}
```

If someone uses:

```
PUT _snapshot/my_backup3
{
    "type": "azure"
}
```

It will use the `default` azure repository settings.

And mark as deprecated old settings.

Closes .
2017-08-08 15:14:47 +02:00
Adrien Grand f0c1e30544 Upgrade to lucene-7.0.0-snapshot-a128fcb. () 2017-08-08 13:03:19 +02:00
Simon Willnauer 82fa531ab4 Remove `_index` fielddata hack if cluster alias is present ()
We introduced a hack in  to respect the cluster alias if available on the `_index` field. This is important if aggregations or other field data related operations are executed. Yet, we added a small hack that duplicated an implementation detail from the `_index` field data builder to make this work. This change adds a necessary but simple API change that allows us to remove the hack and only have a single implementation.
2017-08-08 09:24:24 +02:00
Yannick Welsch 1a01514081 Move tribe to a module ()
This commit moves tribe to a module, stripping core from the tribe functionality.
2017-07-28 11:23:50 +02:00
Tim Brooks 7d2d6bd752 Make calls to CloudBlobContainer#exists privileged ()
This is related to . In CloudBlobContainer#exists it is possible
that a socket connection will be opened. This commit ensures that those
calls have the proper socket privileges.
2017-07-27 22:29:24 -05:00
Tim Brooks 71f58e6f26 Ensure that gcs client creation is privileged ()
This is related to . Currently when we create the
`GoogleCloudStorageService` client we do not wrap that call in a
doPrivileged block. The call might open a connection. This commit
ensures that the creation is wrapped in a doPrivileged block.
2017-07-27 22:28:47 -05:00
Yannick Welsch efd79882a2 Allow build to directly run under JDK 9 ()
With Gradle 4.1 and newer JDK versions, we can finally invoke Gradle directly using a JDK9 JAVA_HOME without requiring a JDK8 to "bootstrap" the build. As the thirdPartyAudit task runs within the JVM that Gradle runs in, it needs to be adapted now to be JDK9 aware.

This commit also changes the `JavaCompile` tasks to only fork if necessary (i.e. when Gradle's JVM and JAVA_HOME's JVM differ).
2017-07-27 16:14:04 +02:00
Simon Willnauer 634ce90dc0 Respect cluster alias in `_index` aggs and queries ()
Today when we aggregate on the `_index` field the cross cluster search
alias is not taken into account. Neither is it respected when we search
on the field. This change adds support for cluster alias when the cluster
alias is present on the `_index` field.

Closes 
2017-07-26 09:16:52 +02:00
Adrien Grand 481d5d09b2 Upgrade to lucene-7.0.0-snapshot-00142c9. ()
Lucene 7.0 is feature-frozen now, so there should not be many changes until GA.
2017-07-11 13:58:55 +02:00
Tal Levy 8cf0528001 update ingest-user-agent regexes.yml ()
This new regexes are from:

3153c2f2ae/regexes.yaml
2017-07-10 08:43:11 -07:00
joachimdraeger 1ff2c13472 Avoid SecurityException in repository-S3 on DefaultS3OutputStream.flush() ()
Moved SocketAccess.doPrivileged up the stack to DefaultS3OutputStream in repository-S3 plugin to avoid SecurityException by Streams.copy(). A plugin is only allowed to use its own jars when performing privileged operations. The S3 client might open a new Socket on close(). 
2017-07-07 09:26:50 -05:00
Jason Tedor 5f2a0118b8 Fix third party audit for repository-hdfs
This commit fixes the third party audit check for the repository-hdfs
plugin; a class was excluded on JDK 9 that does not need to be.
2017-07-02 16:14:05 -04:00
Simon Willnauer 5a7c8bb04e Cleanup network / transport related settings ()
This commit makes the use of the global network settings explicit instead
of implicit within NetworkService. It cleans up several places where we fall
back to the global settings while we should have used tcp or http ones.

In addition this change also removes unnecessary settings classes
2017-07-02 10:16:50 +02:00
James Baiera 74f4a14d82 Upgrading HDFS Repository Plugin to use HDFS 2.8.1 Client ()
Hadoop 2.7.x libraries fail when running on JDK9 due to the version string changing to a single 
character. On Hadoop 2.8, this is no longer a problem, and it is unclear on whether the fix will be 
backported to the 2.7 branch. This commit upgrades our dependency of Hadoop for the HDFS 
Repository to 2.8.1.
2017-06-30 17:57:56 -04:00
olcbean 3518e313b8 Unify the result interfaces from get and search in Java client ()
As GetField and SearchHitField have the same members, they have been unified into
DocumentField.

Closes 
2017-06-29 11:35:28 +02:00
Jason Tedor 5a9fc8aa2a Remove path.conf setting
This commit removes path.conf as a valid setting and replaces it with a
command-line flag for specifying a non-default path for configuration.

Relates 
2017-06-26 15:18:29 -04:00
Adrien Grand 44e9c0b947 Upgrade to lucene-7.0.0-snapshot-ad2cb77. ()
Most notable changes:
 - better update concurrency: LUCENE-7868
 - TopDocs.totalHits is now a long: LUCENE-7872
 - QueryBuilder does not remove the boolean query around multi-term synonyms:
   LUCENE-7878
 - removal of Fields: LUCENE-7500

For the `TopDocs.totalHits` change, this PR relies on the fact that the encoding
of vInts and vLongs are compatible: you can write and read with any of them as
long as the value can be represented by a positive int.
2017-06-22 12:35:33 +02:00
joachimdraeger 98b02676d8 Remove redundant and broken MD5 checksum from repository-s3 ()
Remove redundant and not resettable (fails on retries) check-summing. Checksums are calculated and compared by the S3 client already. 

Closes 
2017-06-21 15:41:17 -04:00
Nik Everett 21b1db2965 Remove assemble from build task when assemble removed
Removes the `assemble` task from the `build` task when we have
removed `assemble` from the project. We removed `assemble` from
projects that aren't published so our releases will be faster. But
That broke CI because CI builds with `gradle precommit build` and,
it turns out, that `build` includes `check` and `assemble`. With
this change CI will only run `check` for projects without an
`assemble`.
2017-06-16 17:19:14 -04:00
Nik Everett 7b358190d6 Remove assemble task when not used for publishing ()
Removes the `assemble` task from projects that are not published.
This should speed up `gradle assemble` by skipping projects that
don't need to be built. Which is useful because `gradle assemble`
is how we cut releases.
2017-06-16 11:46:34 -04:00
David Causse ff9edb627e [analysis-icu] Allow setting unicodeSetFilter ()
UnicodeSetFilter was only allowed in the icu_folding token filter.
It seems useful to expose this setting in icu_normalizer token filter
and char filter.
2017-06-16 11:08:39 +02:00
Jim Ferenczi 0036f28a6a Upgrade icu4j for the ICU analysis plugin to 59.1 ()
* Upgrade icu4j for the ICU analysis plugin to 59.1

Lucene upgraded to 59.1 so we should use the same.

Closes 

* Add breaking change for the icu upgrade
2017-06-15 13:26:48 +02:00
Adrien Grand 0c117145f6 Upgrade to lucene-7.0.0-snapshot-92b1783. ()
This snapshot has faster range queries on range fields (LUCENE-7828), more
accurate norms (LUCENE-7730) and the ability to use fake term frequencies
(LUCENE-7854).
2017-06-15 09:52:07 +02:00
Ryan Ernst caf7792db1 Scripting: Rename SearchScript.needsScores to needs_score ()
This commit renames the needsScores method so as to make it
automatically generatable, based on the name of the `_score` variable
which is available in search scripts. It also adds documentation to
ScriptContext to explain the naming and signature of such methods.
2017-06-14 22:01:19 -07:00
John Murphy c652b586c4 Remove `discovery.type` BWC layer from the EC2/Azure/GCE plugins
Those plugins don't replace the discovery logic but rather only provide a custom unicast host provider for their respective platforms. in 5.1 we introduced the  `discovery.zen.hosts_provider` setting to better reflect it. This PR removes BWC code in those plugins as it is not needed anymore

Fixes 
2017-06-14 13:52:48 +02:00
Adis Nezirović 82897e2636 Port support for commercial GeoIP2 databases from Logstash. ()
* Port support for commercial GeoIP2 databases from Logstash.

* Match GeoIP databases according to the database name suffix.

* Rename CITY/COUNTRY_DB_TYPE, since they are suffixes now.
2017-06-13 14:20:01 -07:00
Jason Tedor 8de6f4e608 Fix secure repository-hdfs tests on JDK 9
The secure repository-hdfs tests fail on JDK 9 because some Hadoop code
reaches into sun.security.krb5. This commit adds the necessary flags to
open the java.security.jgss module. Note that these flags are actually
needed at runtime as well when using secure repository-hdfs. For now we
will punt on how best to help users obtain this when running on JDK 9
with this plugin.

Relates 
2017-06-13 13:26:48 -04:00
James Baiera 2e29b69f6a Revert "Revert "Sense for VirtualBox and $HOME when deciding to turn on vagrant testing. ()""
This reverts commit b9e2a1f989.
2017-06-12 09:41:35 -04:00
Ryan Ernst a03b6c2fa5 Scripting: Change keys for inline/stored scripts to source/id ()
This commit adds back "id" as the key within a script to specify a
stored script (which with file scripts now gone is no longer ambiguous).
It also adds "source" as a replacement for "code". This is in an attempt
to normalize how scripts are specified across both put stored scripts and script usages, including search template requests. This also deprecates the old inline/stored keys.
2017-06-09 08:29:25 -07:00
Jason Tedor 2f5f27fafa Remove unnecessary callback interface
We have a callback interface that is not needed because it is
effectively the same as java.util.function.Consumer. This commit removes
it.

Relates 
2017-06-06 20:50:03 -04:00
Lee Hinman b9e2a1f989 Revert "Sense for VirtualBox and $HOME when deciding to turn on vagrant testing. ()"
This reverts commit 4ed0abe72d.
2017-06-02 14:42:52 -06:00
James Baiera 4ed0abe72d Sense for VirtualBox and $HOME when deciding to turn on vagrant testing. ()
We're using Vagrant in more places now than before. This commit includes a plugin that verifies
the Vagrant and Virtualbox installations for projects that depend on them. This shared code
should fix up the errors we've seen from CI builds relating to the new Kerberos fixture.
2017-06-02 16:26:11 -04:00
Colin Goodheart-Smithe 779fb9a1c0 Adds nodes usage API to monitor usages of actions ()
* Adds nodes usage API to monitor usages of actions

The nodes usage API has 2 main endpoints

/_nodes/usage and /_nodes/{nodeIds}/usage return the usage statistics
for all nodes and the specified node(s) respectively.

At the moment only one type of usage statistics is available, the REST
actions usage. This records the number of times each REST action class is
called and when the nodes usage api is called will return a map of rest
action class name to long representing the number of times each of the action
classes has been called.

Still to do:

* [x] Create usage service to store usage statistics
* [x] Record usage in REST layer
* [x] Add Transport Actions
* [x] Add REST Actions
* [x] Tests
* [x] Documentation

* Rafactors UsageService so counts are done by the handlers

* Fixing up docs tests

* Adds a name to all rest actions

* Addresses review comments
2017-06-02 08:46:38 +01:00
Ryan Ernst 7c1211d2ed Scripting: Add StatefulFactoryType as optional intermediate factory in script contexts ()
ScriptContexts currently understand a FactoryType that can produce
instances of the script InstanceType. However, for search scripts, this
does not work as we have the concept of LeafSearchScript that is created
per lucene segment. This commit effectively renames the existing
SearchScript class into SearchScript.LeafFactory, which is a new,
optional, class that can be defined within a ScriptContext.
LeafSearchScript is effectively renamed back into SearchScript. This
change allows the model of stateless factory -> stateful factory ->
script instance to continue, but in a generic way that any script
context may take advantage of.

relates 
2017-05-30 16:32:14 -07:00
Ryan Ernst 74e031e842 Scripting: Rename CompiledType to FactoryType in ScriptContext ()
This commit renames the concept of the "compiled type" to a "factory
type", along with all implementations of this class to be named Factory.
This brings it inline with the classes purpose.
2017-05-26 00:02:54 -07:00
Ryan Ernst 8eab1fefa1 Scripting: Make contexts available to ScriptEngine construction ()
This commit adds collection of all contexts to the parameters of
getScriptEngine. This will allow script engines like painless to
precache extra information about the contexts.
2017-05-25 16:55:47 -07:00
Ryan Ernst 8aaea51a0a Scripting: Move context definitions to instance type classes ()
This is a simple refactoring to move the context definitions into the
type that they use. While we have multiple context names for the same
class at the moment, this will eventually become one ScriptContext per
instance type, so the pattern of a static member on the interface called
CONTEXT can be used. This commit also moves the consolidated list of
contexts provided by core ES into ScriptModule.
2017-05-25 12:18:45 -07:00
Ryan Ernst 59c052e76f Build: Fix hadoop integ test error on windows ()
This commit fixes the error message to escape the dollar sign for
referencing a literal `$HADOOP_HOME`, which caused an error while trying
to generate an error.

closes 
2017-05-25 12:11:33 -07:00
Ryan Ernst 7d03cff820 Scripting: Make ScriptEngine.compile generic on the script context ()
This commit changes the compile method of ScriptEngine to be generic in
the same way it is on ScriptService. This moves the shim of handling the
two existing context classes into each script engine, so that each
engine can be worked on independently to convert to real handling of
contexts.
2017-05-24 20:06:32 -07:00
Ryan Ernst 1daacd97b0 Scripting: Add instance and compiled classes to script contexts ()
This commit modifies the compile method of ScriptService to be context
aware. The ScriptContext is now a generic class which contains both the
instance type and compiled type for a script. Instance type may be
stateful (for example, pre loading field information for the index a
script will execute on, like in expressions), while the compiled type is
stateless and used to construct instance type instances. This change is
only a first step to cutover ScriptService to the new paradigm. It only
converts callers to the script service, and has a small shim to wrap
compilation from the script engines to support the current two fixed
instance types, SearchScript and ExecutableScript.
2017-05-24 14:29:02 -07:00
Ryan Ernst 0ddd219423 Scripting: Add default implementation of close() for ScriptEngine ()
Since groovy was removed, we no longer have any ScriptEngines with
resources to release. We may want to keep the option open for a script
engine to close resources, but this would not be common. This commit
adds a default implementation to ScriptEngine for `close()` to reduce
the boiler plate that must be added for a ScriptEngine implementation.
2017-05-24 13:19:27 -07:00
Jim Ferenczi 4e70235d55 Upgrade icu4j to latest version () 2017-05-22 09:34:50 +02:00
Ryan Ernst 2de748859f Scripting: Remove "inline script enabled" on script engines ()
ScriptEngine implementations have an overridable method to indicate they
are safe to use as inline scripts. Since groovy was removed fro 6.0,
there are no longer any implementations which used the default false
value. Furthermore, the value was not actually read anywhere. This
commit removes the method. The ScriptEngineRegistry was also no longer
necessary as it only was used to build a map from language to engine.
2017-05-20 12:01:25 -07:00
Nicholas Knize deb7caf4d3 Upgrade to lucene-7.0.0-snapshot-a0aef2f
This commit upgrades master to a current lucene snapshot with commit id a0aef2f.
2017-05-19 10:20:55 -05:00
Jack Conradson 1196dfb6bb Remove Deprecated Script Settings ()
Removes all fine-grained script settings replaced by scripts.types_allowed and scripts.contexts_allowed.
2017-05-18 13:32:46 -07:00
Ryan Ernst b214b80e6c GCS Repository: Remove specifying credential file on disk ()
This commit removes the ability to specify the google credential json
file on disk, which is deprecated in 5.5.0.
2017-05-18 10:22:29 -07:00
Ryan Ernst 2a65bed243 Tests: Change rest test extension from .yaml to .yml ()
This commit renames all rest test files to use the .yml extension
instead of .yaml. This way the extension used within all of
elasticsearch for yaml is consistent.
2017-05-16 17:24:35 -07:00
Ryan Ernst d74760c306 GCS Repository: Add secure storage of credentials ()
This commit adds gcs credential settings to the elasticsearch keystore.
The setting name follows the same pattern as the s3 client settings,
beginning with `gcs.client.`, followed by the client name, and then the
setting name, in this case, `credentials_file`. Using the legacy service
file setting is also deprecated.
2017-05-16 17:17:37 -07:00
Koen De Groote 878ae8eb3c Size lists in advance when known
When constructing an array list, if we know the size of the list in
advance (because we are adding objects to it derived from another list),
we should size the array list to the appropriate capacity in advance (to
avoid resizing allocations). This commit does this in various places.

Relates 
2017-05-12 10:36:13 -04:00
Dimitris Athanasiou b7976bd536 [TEST] Temporarily disable the secure fixture for hdfs tests ()
This keeps failing the build so I am temporarily disabling it
until  gets merged.
2017-05-12 12:58:30 +01:00
Ryan Ernst c1f1f66509 Scripting: Replace advanced and native scripts with ScriptEngine docs ()
This commit documents how to write a `ScriptEngine` in order to use
expert internal apis, such as using Lucene directly to find index term
statistics. These documents prepare the way to remove both native
scripts and IndexLookup.

The example java code is actually compiled and tested under a new gradle
subproject for example plugins. This change does not yet breakup
jvm-example into the new examples dir, which should be done separately.

relates 
relates 
2017-05-11 12:15:16 -07:00
Ryan Ernst 17d01550c2 S3 Repository: Add back repository level credentials ()
Specifying s3 access and secret keys inside repository settings are not
secure. However, until there is a way to dynamically update secure
settings, this is the only way to dynamically add repositories with
credentials that are not known at node startup time. This commit adds
back `access_key` and `secret_key` s3 repository settings, but protects
it with a required system property `allow_insecure_settings`.
2017-05-11 12:14:23 -07:00
Ryan Ernst 0789a74055 S3 Repository: Remove deprecated settings ()
These settings are deprecated in 5.5. This change removes them for 6.0.
2017-05-10 20:12:17 -07:00
James Baiera 6a113ae499 Introduce Kerberos Test Fixture for Repository HDFS Security Tests ()
This PR introduces a subproject in test/fixtures that contains a Vagrantfile used for standing up a 
KRB5 KDC (Kerberos). The PR also includes helper scripts for provisioning principals, a few 
changes to the HDFS Fixture to allow it to interface with the KDC, as well as a new suite of 
integration tests for the HDFS Repository plugin.

The HDFS Repository plugin senses if the local environment can support the HDFS Fixture 
(Windows is generally a restricted environment). If it can use the regular fixture, it then tests if 
Vagrant is installed with a compatible version to determine if the secure test fixtures should be 
enabled. If the secure tests are enabled, then we create a Kerberos KDC fixture, tasks for adding 
the required principals, and an HDFS fixture configured for security. A new integration test task is 
also configured to use the KDC and secure HDFS fixture and to run a testing suite that uses 
authentication. At the end of the secure integration test the fixtures are torn down.
2017-05-10 17:42:20 -04:00
Matt Weber b24326271e Add ICUCollationFieldMapper ()
Adds a new "icu_collation" field type that exposes lucene's
ICUCollationDocValuesField.  ICUCollationDocValuesField is the replacement
for ICUCollationKeyFilter which has been deprecated since Lucene 5.
2017-05-10 10:35:11 +02:00
Nik Everett bb06d8ec4f Allow plugins to build pre-configured token filters ()
This changes the way we register pre-configured token filters so that
plugins can declare them and starts to move all of the pre-configured
token filters out of core. It doesn't finish the job because doing
so would make the change unreviewably large. So this PR includes
a shim that keeps the "old" way of registering pre-configured token
filters around.

The Lowercase token filter is special because there is a "special"
interaction between it and the lowercase tokenizer. I'm not sure
exactly what to do about it so for now I'm leaving it alone with
the intent of figuring out what to do with it in a followup.

This also renames these pre-configured token filters from
"pre-built" to "pre-configured" because that seemed like a more
descriptive name.

This is a part of 
2017-05-09 14:50:49 -04:00
Yannick Welsch c8712e9531 Limit AllocationService dependency injection hack ()
Changes the scope of the AllocationService dependency injection hack so that it is at least contained to the AllocationService and does not leak into the Discovery world.
2017-05-05 08:39:18 +02:00
James Baiera f5edd5049a Fixing permission errors for `KERBEROS` security mode for HDFS Repository ()
Added missing permissions required for authenticating with Kerberos to HDFS. Also implemented 
code to support authentication in the form of using a Kerberos keytab file. In order to support 
HDFS authentication, users must install a Kerberos keytab file on each node and transfer it to the 
configuration directory. When a user specifies a Kerberos principal in the repository settings the 
plugin automatically enables security for Hadoop and begins the login process. There will be a 
separate PR and commit for the testing infrastructure to support these changes.
2017-05-04 10:51:31 -04:00
James Baiera d928ae210d Add Vagrant based testing fixture () 2017-05-04 10:17:55 -04:00
Koen De Groote 0fef5acd01 Cleanup collections construction
This commit cleans up some cases where a list or map was being
constructed, and then an existing collection was copied into the new
collection. The clean is to instead use an appropriate constructor to
directly copy the existing collection in during collection
construction. The advantage of this is that the new collection is sized
appropriately.

Relates 
2017-04-30 21:26:51 -04:00
Yannick Welsch 35f78d098a Separate publishing from applying cluster states ()
Separates cluster state publishing from applying cluster states:

- ClusterService is split into two classes MasterService and ClusterApplierService. MasterService has the responsibility to calculate cluster state updates for actions that want to change the cluster state (create index, update shard routing table, etc.). ClusterApplierService has the responsibility to apply cluster states that have been successfully published and invokes the cluster state appliers and listeners.
- ClusterApplierService keeps track of the last applied state, but MasterService is stateless and uses the last cluster state that is provided by the discovery module to calculate the next prospective state. The ClusterService class is still kept around, which now just delegates actions to ClusterApplierService and MasterService.
- The discovery implementation is now responsible for managing the last cluster state that is used by the consensus layer and the master service. It also exposes the initial cluster state which is used by the ClusterApplierService. The discovery implementation is also responsible for adding the right cluster-level blocks to the initial state.
- NoneDiscovery has been renamed to TribeDiscovery as it is exclusively used by TribeService. It adds the tribe blocks to the initial state.
- ZenDiscovery is synchronized on state changes to the last cluster state that is used by the consensus layer and the master service, and does not submit cluster state update tasks anymore to make changes to the disco state (except when becoming master).

Control flow for cluster state updates is now as follows:

- State updates are sent to MasterService
- MasterService gets the latest committed cluster state from the discovery implementation and calculates the next cluster state to publish
- MasterService submits the new prospective cluster state to the discovery implementation for publishing
- Discovery implementation publishes cluster states to all nodes and, once the state is committed, asks the ClusterApplierService to apply the newly committed state.
- ClusterApplierService applies state to local node.
2017-04-28 09:34:31 +02:00
Ryan Ernst 4a5c3c5a4a Test: Write node ports file before starting tribe service ()
The tribe service can take a while to initialize, depending on how many cluster it needs to connect to. This change moves writing the ports file used by tests to before the tribe service is started.
2017-04-27 09:59:54 +02:00
Ryan Ernst 51b33f1fd5 S3 Repository: Deprecate remaining `repositories.s3.*` settings ()
Most of these settings should always be pulled from the repository
settings. A couple were leftover that should be moved to client
settings. The path style access setting should be removed altogether.
This commit adds deprecations for all of these existing settings, as
well as adding new client specific settings for max retries and
throttling.

relates 
2017-04-25 23:43:20 -07:00
Nik Everett caf376c8af Start building analysis-common module ()
Start moving built in analysis components into the new analysis-common
module. The goal of this project is:
1. Remove core's dependency on lucene-analyzers-common.jar which should
shrink the dependencies for transport client and high level rest client.
2. Prove that analysis plugins can do all the "built in" things by moving all
"built in" behavior to a plugin.
3. Force tests not to depend on any oddball analyzer behavior. If tests
need anything more than the standard analyzer they can use the mock
analyzer provided by Lucene's test infrastructure.
2017-04-19 18:51:34 -04:00
Ryan Ernst 151a65ed17 Ec2 Discovery: Cleanup deprecated settings ()
This commit removes the deprecated cloud.aws.* settings. It also removes
backcompat for specifying `discovery.type: ec2`, and unused aws signer
code which was removed in a previous PR.
2017-04-19 12:06:10 -07:00
Ryan Ernst 212f24aa27 Tests: Clean up rest test file handling ()
This change simplifies how the rest test runner finds test files and
removes all leniency.  Previously multiple prefixes and suffixes would
be tried, and tests could exist inside or outside of the classpath,
although outside of the classpath never quite worked. Now only classpath
tests are supported, and only one resource prefix is supported,
`/rest-api-spec/tests`.

closes 
2017-04-18 15:07:08 -07:00
Adrien Grand 4632661bc7 Upgrade to a Lucene 7 snapshot ()
We want to upgrade to Lucene 7 ahead of time in order to be able to check whether it causes any trouble to Elasticsearch before Lucene 7.0 gets released. From a user perspective, the main benefit of this upgrade is the enhanced support for sparse fields, whose resource consumption is now function of the number of docs that have a value rather than the total number of docs in the index.

Some notes about the change:
 - it includes the deprecation of the `disable_coord` parameter of the `bool` and `common_terms` queries: Lucene has removed support for coord factors
 - it includes the deprecation of the `index.similarity.base` expert setting, since it was only useful to configure coords and query norms, which have both been removed
 - two tests have been marked with `@AwaitsFix` because of , which we intend to address after the merge
2017-04-18 15:17:21 +02:00
Ryan Ernst a8083f3d76 S3 Repository: Remove unused files ()
These were leftover from the removal of the signer type setting in
2017-04-18 01:19:25 -07:00
Ryan Ernst a8017ff020 Tests: Move cluster dependencies from runner to cluster ()
After splitting integ tests into cluster configuration and the test
runner task, we still have dependencies of the test runner added as deps
of the cluster. This commit adds dependencies directly to the cluster,
so that the runner can have other dependencies independent of what is
needed for the cluster.
2017-04-17 16:02:46 -07:00
Ryan Ernst 1629c9fd5c S3 Repository: Cleanup deprecated settings ()
This commit removes all deprecated settings which start with
`cloud.aws`, `repositories.s3` and repository level client settings.
2017-04-17 14:18:49 -07:00
Ryan Ernst 1207103b6d S3 Repository: Eagerly load static settings ()
The S3 repostiory has many levels of settings it looks at to create a
repository, and these settings were read at repository creation time.
This meant secure settings like access and secret keys had to be
available after node construction. This change makes setting loading for
every except repository level settings eager, so that secure settings
can be stashed, and the keystore can once again be closed after
bootstrapping the node is complete.
2017-04-11 15:42:56 -07:00
Colin Goodheart-Smithe 0114f0061c Removes version 2.x constants from Version ()
* Removes version 2.x constants from Version

Closes 

* Addresses review comments
2017-04-11 08:31:22 +01:00
Ryan Ernst dd3c1137a4 Repository S3: Simplify client method ()
This commit removes passing the repository metadata object through to
s3 client creation. It is not needed, and in fact in tests was confusing
because you could create the metadata but have it contain different
settings than were passed in as repository settings.
2017-04-10 14:43:34 -07:00
Ryan Ernst 83ba677e7f Discovery EC2: Remove region setting ()
We have both endpoint and region settings. Region was removed from s3 to
simplify configuration. This is the ec2 equivalent.

closes 
2017-04-07 22:06:40 -07:00