Commit Graph

4742 Commits

Author SHA1 Message Date
Nhat Nguyen 5875399738
Upgrade to Lucene-7.5.0-snapshot-b9e064b935 (#31839)
This snapshot includes LUCENE-8384 which blocks work in Lucene rollback.
2018-07-08 15:05:08 -04:00
Nhat Nguyen 6dd3434519 Merge branch 'master' into ccr
* master:
  REST high-level client: add get index API (#31703)
  SQL: Allow long literals (#31777)
  SQL: Fix incorrect message for aliases (#31792)
  Test: Do not remove xpack templates when cleaning (#31642)
  Reduce more raw types warnings (#31780)
  Add unreleased version 6.3.2
  Scripting: Remove support for deprecated StoredScript contexts (#31394)
  [ML][TEST] Use java 11 valid time format in DataDescriptionTests (#31817)
  [ML] Don't treat stale FAILED jobs as OPENING in job allocation (#31800)
  [ML] Fix calendar and filter updates from non-master nodes (#31804)
  Fix license header generation on Windows (#31790)
  mark RollupIT.testTwoJobsStartStopDeleteOne as AwaitsFix
  mark SearchAsyncActionTests.testFanOutAndCollect as AwaitsFix
  Correct exclusion of test on JDK 11
  Fix doclint jdk 11
  Add JDK11 support and enable in CI (#31644)
  Watcher: Fix check for currently executed watches (#31137)
  Watcher: Ensure correct method is used to read secure settings (#31753)
  SQL: Update CLI logo
2018-07-05 14:07:06 -04:00
Christoph Büscher bd1c513422
Reduce more raw types warnings (#31780)
Similar to #31523.
2018-07-05 15:38:06 +02:00
Sohaib Iftikhar 40b822c878 Scripting: Remove support for deprecated StoredScript contexts (#31394)
Removes support for storing scripts without the usual json around the
script. So You can no longer do:
```
POST _scripts/<templatename>
{
    "query": {
        "match": {
            "title": "{{query_string}}"
        }
    }
}
```

and must instead do:
```
POST _scripts/<templatename>
{
    "script": {
        "lang": "mustache",
        "source": {
            "query": {
                "match": {
                    "title": "{{query_string}}"
                }
            }
        }
    }
}
```

This improves error reporting when you attempt to store a script but don't
quite get the syntax right. Before, there was a good chance that we'd
think of it as a "raw" template and just store it. Now we won't do that.
Nice.
2018-07-05 09:30:08 -04:00
Alpar Torok cf2295b408
Add JDK11 support and enable in CI (#31644)
* Upgrade bouncycastle

Required to fix
`bcprov-jdk15on-1.55.jar; invalid manifest format `
on jdk 11

* Downgrade bouncycastle to avoid invalid manifest

* Add checksum for new jars

* Update tika permissions for jdk 11

* Mute test failing on jdk 11

* Add JDK11 to CI

* Thread#stop(Throwable) was removed

http://mail.openjdk.java.net/pipermail/core-libs-dev/2018-June/053536.html

* Disable failing tests #31456

* Temprorarily disable doc tests

To see if there are other failures on JDK11

* Only blacklist specific doc tests

* Disable only failing tests in ingest attachment plugin

* Mute failing HDFS tests #31498

* Mute failing lang-painless tests #31500

* Fix backwards compatability builds

Fix JAVA version to 10 for ES 6.3

* Add 6.x to bwx -> java10

* Prefix out and err from buildBwcVersion for readability

```
> Task :distribution:bwc:next-bugfix-snapshot:buildBwcVersion
  [bwc] :buildSrc:compileJava
  [bwc] WARNING: An illegal reflective access operation has occurred
  [bwc] WARNING: Illegal reflective access by org.codehaus.groovy.reflection.CachedClass (file:/home/alpar/.gradle/wrapper/dists/gradle-4.5-all/cg9lyzfg3iwv6fa00os9gcgj4/gradle-4.5/lib/groovy-all-2.4.12.jar) to method java.lang.Object.finalize()
  [bwc] WARNING: Please consider reporting this to the maintainers of org.codehaus.groovy.reflection.CachedClass
  [bwc] WARNING: Use --illegal-access=warn to enable warnings of further illegal reflective access operations
  [bwc] WARNING: All illegal access operations will be denied in a future release
  [bwc] :buildSrc:compileGroovy
  [bwc] :buildSrc:writeVersionProperties
  [bwc] :buildSrc:processResources
  [bwc] :buildSrc:classes
  [bwc] :buildSrc:jar

```

* Also set RUNTIME_JAVA_HOME for bwcBuild

So that we can make sure it's not too new for the build to understand.

* Align bouncycastle dependency

* fix painles array tets

closes #31500

* Update jar checksums

* Keep 8/10 runtime/compile untill consensus builds on 11

* Only skip failing tests if running on Java 11

* Failures are dependent of compile java version not runtime

* Condition doc test exceptions on compiler java version as well

* Disable hdfs tests based on runtime java

* Set runtime java to minimum supported for bwc

* PR review

* Add comment with ticket for forbidden apis
2018-07-05 03:24:01 +00:00
Nhat Nguyen f736205fd5 Merge branch 'master' into ccr
* master:
  [ML] Rate limit established model memory updates (#31768)
  [Docs] Correct default window_size (#31582)
  S3 fixture should report 404 on unknown bucket (#31782)
  Detach Transport from TransportService (#31727)
  [ML] Limit ML filter items to 10K (#31731)
  [ML] Return statistics about forecasts as part of the jobsstats and usage API (#31647)
  Fixture for Minio testing (#31688)
  [DOCS] Add missing get mappings docs to HLRC (#31765)
  [DOCS] Starting Elasticsearch (#31701)
  Painless: Complete Removal of Painless Type (#31699)
  Fix not waiting for Netty ThreadDeathWatcher in IT (#31758)
  Consolidate watcher setting update registration (#31762)
  Build: re-enabled bwc (#31769)
  ingest: Introduction of a bytes processor (#31733)
  Fix coerce validation_method in GeoBoundingBoxQueryBuilder (#31747)
  Add analyze API to high-level rest client (#31577)
  [DOCS] Typos
  DOC: Add examples to the SQL docs (#31633)
  Add support for AWS session tokens (#30414)
  Watcher: Reenable start/stop yaml tests (#31754)
  Implemented XContent serialisation for GetIndexResponse (#31675)
  JDBC: Fix stackoverflow on getObject and timestamp conversion (#31735)
  resolveHasher defaults to NOOP (#31723)
  Account for XContent overhead in in-flight breaker
  Split CircuitBreaker-related tests (#31659)
  Add write*Blob option to replace existing blob (#31729)
  Painless: Add Context Docs (#31190)
  Watcher: Fix chain input toXcontent serialization (#31721)
  Docs: Match the examples in the description (#31710)
  rest-high-level: added get cluster settings (#31706)
  [Docs] Correct typos (#31720)
  Clean up double semicolon code typos (#31687)
  [DOCS] Check for Windows and *nix file paths (#31648)
  [ML] Validate ML filter_id (#31535)
  Revert long lines
  Fix TransportChangePasswordActionTests
2018-07-04 09:18:27 -04:00
Simon Willnauer 3f2a241b7f
Detach Transport from TransportService (#31727)
Today TransportService is tightly coupled with Transport since it
requires an instance of TransportService in order to receive responses
and send requests. This is mainly due to the Request and Response handlers
being maintained in TransportService but also because of the lack of a proper 
callback interface.

This change moves request handler registry and response handler registration into
Transport and adds all necessary methods to `TransportConnectionListener` in order
to remove the `TransportService` dependency from `Transport`
Transport now accepts one or more `TransportConnectionListener` instances that are
executed sequentially in a blocking fashion.
2018-07-04 11:32:35 +02:00
Jack Conradson a02e5ee740
Painless: Complete Removal of Painless Type (#31699)
This completes the removal of Painless Type. The new data structures in the definition are a map of names (String) to Java Classes and a map of Java Classes to Painless Structs. The names to Java Classes map can contain a 2 to 1 ratio of names to classes depending on whether or not a short (imported) name is used. The Java Classes to Painless Structs is 1 to 1 always where the Java Class name must match the Painless Struct name. This should lead a significantly simpler type system in Painless moving forward since the Painless Type only held redundant information since Painless does not support generics.
2018-07-03 13:31:56 -07:00
Jake Landis c0056cddd8
ingest: Introduction of a bytes processor (#31733)
ingest: Introduction of a bytes processor

This processor allows for human readable byte values (e.g. 1kb) to be converted to value in bytes (e.g. 1024). Internally this processor re-uses "ByteSizeValue.parseBytesSizeValue" which supports conversions up to Long.MAX_VALUE and the following units: "b", "kb", "mb", "gb", "tb", pb".

This change also introduces a generic return type for the AbstractStringProcessor to allow for code reuse while supporting a String -> T conversion. (String -> Long in this case).
2018-07-03 10:40:56 -05:00
Yannick Welsch 2bb4f38371
Add write*Blob option to replace existing blob (#31729)
Adds a new parameter to the BlobContainer#write*Blob methods to specify whether the existing file
should be overridden or not. For some metadata files in the repository, we actually want to replace
the current file. This is currently implemented through an explicit blob delete and then a fresh write.
In case of using a cloud provider (S3, GCS, Azure), this results in 2 API requests instead of just 1.
This change will therefore allow us to achieve the same functionality using less API requests.
2018-07-03 09:13:50 +02:00
Christoph Büscher 31aabe4bf9
Clean up double semicolon code typos (#31687) 2018-07-02 15:14:44 +02:00
Jason Tedor 05b4517f2f
Merge remote-tracking branch 'elastic/master' into ccr
* elastic/master:
  Enable setting client path prefix to / (#30119)
  [DOCS] Secure settings specified per node (#31621)
  has_parent builder: exception message/param fix (#31182)
2018-07-01 14:42:04 -04:00
Nirmal Chidambaram c827a4e8e1 has_parent builder: exception message/param fix (#31182)
has_parent builder throws exception message that it expects a `type`
while parser excepts `parent_type`
2018-06-30 11:17:37 -07:00
Nhat Nguyen 52d9012d31 Merge branch 'master' into ccr
* master:
  Do not check for object existence when deleting repository index files (#31680)
  Remove extra check for object existence in repository-gcs read object (#31661)
  Support multiple system store types (#31650)
  [Test] Clean up some repository-s3 tests (#31601)
  [Docs] Use capital letters in section headings (#31678)
  [DOCS] Add PQL language Plugin (#31237)
  Merge AzureStorageService and AzureStorageServiceImpl and clean up tests (#31607)
  TEST: Fix test task invocation (#31657)
  Revert "[TEST] Mute failing tests in NativeRealmInteg and ReservedRealmInteg"
  Fix RealmInteg test failures
  Extend allowed characters for grok field names (#21745) (#31653)
  [DOCS] Fix licensing API details (#31667)
  [TEST] Mute failing tests in NativeRealmInteg and ReservedRealmInteg
  Fix CreateSnapshotRequestTests Failure (#31630)
  Configurable password hashing algorithm/cost (#31234)
  [TEST] Mute failing NamingConventionsTaskIT tests
  [DOCS] Replace CONFIG_DIR with ES_PATH_CONF (#31635)
  Core: Require all actions have a Task (#31627)
2018-06-29 10:12:03 -04:00
Tanguy Leroux 0ef22db844
[Test] Clean up some repository-s3 tests (#31601)
This commit removes some tests in the repository-s3 plugin that 
have not been executed for 2+ years but have been maintained 
for nothing. Most of the tests in AbstractAwsTestCase were 
obsolete or superseded by fixture based integration tests.
2018-06-29 13:21:29 +02:00
Nhat Nguyen 2c10cd3e1d Merge branch 'master' into ccr
* master:
  Docs: Remove duplicate test setup
  Print output when the name checker IT fails (#31660)
  Fix syntax errors in get-snapshots docs (#31656)
  Docs: Fix description of percentile ranks example example (#31652)
  Add MultiSearchTemplate support to High Level Rest client (#30836)
  Add test for low-level client round-robin behaviour (#31616)
  SQL: Refactor package names of sql-proto and sql-shared-proto projects (#31622)
  Remove deprecation warnings to prepare for Gradle 5 (sourceSets.main.output.classesDirs) (#30389)
  Correct integTest enable logic (#31646)
  Fix missing get-snapshots docs reference #31645
  Do not check for Azure container existence (#31617)
  Merge AwsS3Service and InternalAwsS3Service in a S3Service class (#31580)
  Upgrade gradle wrapper to 4.8 (#31525)
  Only set vm.max_map_count if greater than default (#31512)
  Add Get Snapshots High Level REST API (#31537)
  QA: Merge query-builder-bwc to restart test (#30979)
  Update reindex.asciidoc (#31626)
  Docs: Skip xpack snippet tests if no xpack (#31619)
  mute CreateSnapshotRequestTests
  HLRest: Fix test for explain API
  [TEST] Fix RemoteClusterConnectionTests
  Add Create Snapshot to High-Level Rest Client (#31215)
  Remove legacy MetaDataStateFormat (#31603)
  Add explain API to high-level REST client (#31387)
  Preserve thread context when connecting to remote cluster (#31574)
  Unify headers for full text queries
  Remove redundant 'minimum_should_match'
  JDBC driver prepared statement set* methods  (#31494)
  [TEST] call yaml client close method from test suite (#31591)
2018-06-28 11:13:01 -04:00
markharwood 09dd19a403
Add MultiSearchTemplate support to High Level Rest client (#30836)
Add MultiSearchTemplate support to High Level Rest client.
Addresses part of #27205
2018-06-28 14:05:26 +01:00
Nhat Nguyen a55f614b85 Merge branch 'master' into ccr
* master:
  ingest: Add ignore_missing property to foreach filter (#22147) (#31578)
  Fix a formatting issue in the docvalue_fields documentation. (#31563)
  reduce log level at gradle configuration time
  [TEST] Close additional clients created while running yaml tests (#31575)
  Docs: Clarify sensitive fields watcher encryption (#31551)
  Watcher: Remove never executed code (#31135)
  Add support for switching distribution for all integration tests (#30874)
  Improve robustness of geo shape parser for malformed shapes (#31449)
  QA: Create xpack yaml features (#31403)
  Improve test times for tests using `RandomObjects::addFields` (#31556)
  [Test] Add full cluster restart test for Rollup (#31533)
  Enhance thread context uniqueness assertion
  [DOCS] Fix heading format errors (#31483)
  fix writeIndex evaluation for aliases (#31562)
  Add x-opaque-id to search slow logs (#31539)
  Watcher: Fix put watch action (#31524)
  Add package pre-install check for java binary (#31343)
  Reduce number of raw types warnings (#31523)
  Migrate scripted metric aggregation scripts to ScriptContext design (#30111)
  turn GetFieldMappingsResponse to ToXContentObject (#31544)
  Close xcontent parsers (partial) (#31513)
  Ingest Attachment: Upgrade Tika to 1.18 (#31252)
  TEST: Correct the assertion arguments order (#31540)
2018-06-26 16:25:32 -04:00
Armin Braun 13e1cf6191
ingest: Add ignore_missing property to foreach filter (#22147) (#31578) 2018-06-26 20:04:41 +02:00
Alpar Torok 08b8d11e30
Add support for switching distribution for all integration tests (#30874)
* remove left-over comment

* make sure of the property for plugins

* skip installing modules if these exist in the distribution

* Log the distrbution being ran

* Don't allow running with integ-tests-zip passed externally

* top level x-pack/qa can't run with oss distro

* Add support for matching objects in lists

Makes it possible to have a key that points to a list and assert that a
certain object is present in the list. All keys have to be present and
values have to match. The objects in the source list may have additional
fields.

example:
```
  match:  { 'nodes.$master.plugins': { name: ingest-attachment }  }
```

* Update plugin and module tests to work with other distributions

Some of the tests expected that the integration tests will always be ran
with  the `integ-test-zip` distribution so that there will be no other
plugins loaded.

With this change, we check for the presence of the plugin without
assuming exclusivity.

* Allow modules to run on other distros as well

To match the behavior of tets.distributions

* Add and use a new `contains` assertion

Replaces the  previus changes that caused `match` to do a partial match.

* Implement PR review comments
2018-06-26 06:49:03 -07:00
Igor Motov 237650e9c0
Add x-opaque-id to search slow logs (#31539)
Add x-opaque-id to search slow logs only. Indexing slow log and audit
logs will be handled as separate PRs.

Relates #31521
2018-06-25 12:20:27 -07:00
Christoph Büscher 86ab3a2d1a
Reduce number of raw types warnings (#31523)
A first attempt to reduce the number of raw type warnings, 
most of the time by using the unbounded wildcard.
2018-06-25 15:59:03 +02:00
Jonathan Little 8e4768890a Migrate scripted metric aggregation scripts to ScriptContext design (#30111)
* Migrate scripted metric aggregation scripts to ScriptContext design #29328

* Rename new script context container class and add clarifying comments to remaining references to params._agg(s)

* Misc cleanup: make mock metric agg script inner classes static

* Move _score to an accessor rather than an arg for scripted metric agg scripts

This causes the score to be evaluated only when it's used.

* Documentation changes for params._agg -> agg

* Migration doc addition for scripted metric aggs _agg object change

* Rename "agg" Scripted Metric Aggregation script context variable to "state"

* Rename a private base class from ...Agg to ...State that I missed in my last commit

* Clean up imports after merge
2018-06-25 12:01:33 +01:00
Nhat Nguyen 08ee9b67c5 Merge branch 'master' into ccr
* master:
  Add get field mappings to High Level REST API Client (#31423)
  [DOCS] Updates Watcher examples for code testing (#31152)
  TEST: Add bwc recovery tests with synced-flush index
  [DOCS] Move sql to docs (#31474)
  [DOCS] Move monitoring to docs folder (#31477)
  Core: Combine doExecute methods in TransportAction (#31517)
  IndexShard should not return null stats (#31528)
  fix repository update with the same settings but different type (#31458)
  Fix Mockito trying to mock IOException that isn't thrown by method (#31433) (#31527)
  Node selector per client rather than per request (#31471)
  Core: Combine messageRecieved methods in TransportRequestHandler (#31519)
  Upgrade to Lucene 7.4.0. (#31529)
  [ML] Add ML filter update API (#31437)
  Allow multiple unicast host providers (#31509)
  Avoid deprecation warning when running the ML datafeed extractor. (#31463)
  REST high-level client: add simulate pipeline API (#31158)
  Get Mapping API to honour allow_no_indices and ignore_unavailable (#31507)
  [PkiRealm] Invalidate cache on role mappings change (#31510)
  [Security] Check auth scheme case insensitively (#31490)
  In NumberFieldType equals and hashCode, make sure that NumberType is taken into account. (#31514)
  [DOCS] Fix REST tests in SQL docs
  [DOCS] Add code snippet testing in more ML APIs (#31339)
  Core: Remove ThreadPool from base TransportAction (#31492)
  [DOCS] Remove fixed file from build.gradle
  Rename createNewTranslog to fileBasedRecovery (#31508)
  Test: Skip assertion on windows
  [DOCS] Creates field and document level security overview (#30937)
  [DOCS] Significantly improve SQL docs
  [DOCS] Move migration APIs to docs (#31473)
  Core: Convert TransportAction.execute uses to client calls (#31487)
  Return transport addresses from UnicastHostsProvider (#31426)
  Ensure local addresses aren't null (#31440)
  Remove unused generic type for client execute method (#31444)
  Introduce http and tcp server channels (#31446)
2018-06-23 17:27:17 -04:00
Ryan Ernst 7a150ec06d
Core: Combine doExecute methods in TransportAction (#31517)
TransportAction currently contains 2 doExecute methods, one which takes
a the task, and one that does not. The latter is what some subclasses
implement, while the first one just calls the latter, dropping the given
task. This commit combines these methods, in favor of just always
assuming a task is present.
2018-06-22 15:03:01 -07:00
Ryan Ernst 59e7c6411a
Core: Combine messageRecieved methods in TransportRequestHandler (#31519)
TransportRequestHandler currently contains 2 messageReceived methods,
one which takes a Task, and one that does not. The first just delegates
to the second. This commit changes all existing implementors of
TransportRequestHandler to implement the version which takes Task, thus
allowing the class to be a functional interface, and eliminating the
need to throw exceptions when a task needs to be ensured.
2018-06-22 07:36:03 -07:00
Adrien Grand f023e95ae0
Upgrade to Lucene 7.4.0. (#31529)
This moves Elasticsearch from a recent 7.4.0 snapshot to the GA release.
2018-06-22 16:17:17 +02:00
Ryan Ernst 4f9332ee16
Core: Remove ThreadPool from base TransportAction (#31492)
Most transport actions don't need the node ThreadPool. This commit
removes the ThreadPool as a super constructor parameter for
TransportAction. The actions that do need the thread pool then have a
member added to keep it from their own constructor.
2018-06-21 11:25:26 -07:00
Ryan Ernst 0a324b9943
Core: Convert TransportAction.execute uses to client calls (#31487)
This commit converts some of the existing calls to
TransportAction.execute to use the equivalent client method for the
desired action.
2018-06-21 07:59:55 -07:00
Ryan Ernst 00283a61e1
Remove unused generic type for client execute method (#31444)
This commit removes the request builder generic type for AbstractClient
as it was unused.
2018-06-20 16:26:26 -07:00
Tim Brooks 9ab1325953
Introduce http and tcp server channels (#31446)
Historically in TcpTransport server channels were represented by the
same channel interface as socket channels. This was necessary as
TcpTransport was parameterized by the channel type. This commit
introduces TcpServerChannel and HttpServerChannel classes. Additionally,
it adds the implementations for the various transports. This allows
server channels to have unique functionality and not implement the
methods they do not support (such as send and getRemoteAddress).

Additionally, with the introduction of HttpServerChannel this commit
extracts some of the storing and closing channel work to the abstract
http server transport.
2018-06-20 16:34:56 -06:00
Nhat Nguyen efcb9a3603 Merge branch 'master' into ccr
* master:
  [DOCS] Omit shard failures assertion for incompatible responses  (#31430)
  [DOCS] Move licensing APIs to docs (#31445)
  Add Delete Snapshot High Level REST API
  Remove QueryCachingPolicy#ALWAYS_CACHE (#31451)
  [Docs] Extend Homebrew installation instructions (#28902)
  Choose JVM options ergonomically
  [Docs] Mention ip_range datatypes on ip type page (#31416)
  Multiplexing token filter (#31208)
  Fix use of time zone in date_histogram rewrite (#31407)
  Core: Remove index name resolver from base TransportAction (#31002)
  [DOCS] Fixes code snippet testing for machine learning (#31189)
  [DOCS] Removed  and  params from MLT. Closes #28128 (#31370)
  Security: fix joining cluster with production license (#31341)
  Unify http channels and exception handling (#31379)
  [DOCS] Moves the info API to docs (#31121)
  Preserve response headers on cluster update task (#31421)
  [DOCS] Add code snippet testing for more ML APIs (#31404)
  Do not preallocate bytes for channel buffer (#31400)
  Docs: Advice for reindexing many indices (#31279)
  Mute HttpExporterTests#testHttpExporterShutdown test Tracked by #31433
  Docs: Add note about removing prepareExecute from the java client (#31401)
  Make release notes ignore the `>test-failure` label. (#31309)
2018-06-20 12:43:40 -04:00
Alan Woodward 5683bc60a6
Multiplexing token filter (#31208)
The `multiplexer` filter emits multiple tokens at the same position, each 
version of the token haivng been passed through a different filter chain.
Identical tokens at the same position are removed.

This allows users to, for example, index lowercase and original-case tokens,
or stemmed and unstemmed versions, in the same field, so that they can search
for a stemmed term within x positions of an unstemmed term.
2018-06-20 10:16:26 +01:00
Ryan Ernst 401800d958
Core: Remove index name resolver from base TransportAction (#31002)
Most transport actions don't need to resolve index names. This commit
removes the index name resolver as a super constructor parameter for
TransportAction. The actions that do need the resolver then have a
member added to keep the resolver from their own constructor.
2018-06-19 17:06:09 -07:00
Tim Brooks 529e704b11
Unify http channels and exception handling (#31379)
This is a general cleanup of channels and exception handling in http.
This commit introduces a CloseableChannel that is a superclass of
TcpChannel and HttpChannel. This allows us to unify the closing logic
between tcp and http transports. Additionally, the normal http channels
are extracted to the abstract server transport.

Finally, this commit (mostly) unifies the exception handling between nio
and netty4 http server transports.
2018-06-19 11:50:03 -06:00
Nhat Nguyen ec04366d29 Merge branch 'master' into ccr
* master:
  Add get stored script and delete stored script to high level REST API - post backport fix
  Add get stored script and delete stored script to high level REST API (#31355)
  Core: Combine Action and GenericAction (#31405)
  Fix reference to XContentBuilder.string() (#31337)
  Avoid sending duplicate remote failed shard requests (#31313)
  Fix defaults in GeoShapeFieldMapper output (#31302)
  RestAPI: Reject forcemerge requests with a body (#30792)
  Packaging: Remove windows bin files from the tar distribution (#30596)
  Docs: Use the default distribution to test docs (#31251)
  [DOCS] Adds testing for security APIs (#31345)
  Clarify that IP range data can be specified in CIDR notation. (#31374)
  Use system context for cluster state update tasks (#31241)
  Percentile/Ranks should return null instead of NaN when empty (#30460)
  REST high-level client: add validate query API (#31077)
  Move language analyzers from server to analysis-common module. (#31300)
  [Test] Fix :example-plugins:rest-handler on Windows
  Expose lucene's RemoveDuplicatesTokenFilter (#31275)
  Reload secure settings for plugins (#31383)
  Remove some cases in FieldTypeLookupTests that are no longer relevant. (#31381)
  Ensure we don't use a remote profile if cluster name matches (#31331)
  [TEST] Double write alias fault (#30942)
  [DOCS] Fix version in SQL JDBC Maven template
  [DOCS] Improve install and setup section for SQL JDBC
  SQL: Fix rest endpoint names in node stats (#31371)
  Support for remote path in reindex api - post backport fix Closes #22913
  [ML] Put ML filter API response should contain the filter (#31362)
  Support for remote path in reindex api (#31290)
  Add byte array pooling to nio http transport (#31349)
  Remove trial status info from start trial doc (#31365)
  [DOCS] Adds links to release notes and highlights
  add is-write-index flag to aliases (#30942)
  Add rollover-creation-date setting to rolled over index (#31144)
  [ML] Hold ML filter items in sorted set (#31338)
  [Tests] Fix edge case in ScriptedMetricAggregatorTests (#31357)
2018-06-19 09:46:39 -04:00
Ryan Ernst e67aa96c81
Core: Combine Action and GenericAction (#31405)
Since #30966, Action no longer has anything but a call to the
GenericAction super constructor. This commit renames GenericAction
into Action, thus eliminating the Action class. Additionally, this
commit removes the Request generic parameter of the class, since
it was unused.
2018-06-18 23:53:04 +02:00
Martijn van Groningen 47095357bc
Move language analyzers from server to analysis-common module. (#31300)
The following analyzers were moved from server module to analysis-common module:
`greek`, `hindi`, `hungarian`, `indonesian`, `irish`, `italian`, `latvian`,
`lithuanian`, `norwegian`, `persian`, `portuguese`, `romanian`, `russian`,
`sorani`, `spanish`, `swedish`, `turkish` and `thai`.

Relates to #23658
2018-06-18 11:24:43 +02:00
Alan Woodward 8c0ec05a12
Expose lucene's RemoveDuplicatesTokenFilter (#31275) 2018-06-18 09:46:12 +01:00
Vladimir Dolzhenko dbc9d60260
Support for remote path in reindex api (#31290)
Support for remote path in reindex api
Closes #22913
2018-06-15 22:14:28 +02:00
Tim Brooks a705e1a9e3
Add byte array pooling to nio http transport (#31349)
This is related to #28898. This PR implements pooling of bytes arrays
when reading from the wire in the http server transport. In order to do
this, we must integrate with netty reference counting. That manner in
which this PR implements this is making Pages in InboundChannelBuffer
reference counted. When we accessing the underlying page to pass to
netty, we retain the page. When netty releases its bytebuf, it releases
the underlying pages we have passed to it.
2018-06-15 14:01:03 -06:00
Nhat Nguyen a8abf0fcc0 Merge branch 'master' into ccr
* master:
  Upgrade to Lucene-7.4.0-snapshot-518d303506 (#31360)
  Rankeval: Fold template test project into main module (#31203)
  Add QA project and fixture based test for discovery-ec2 plugin (#31107)
  [Docs] Remove reference to repository-s3 plugin creating an S3 bucket (#31359)
  REST Client: NodeSelector for node attributes (#31296)
  LLClient: Fix assertion on windows
  Add details section for dcg ranking metric (#31177)
  [ML] Re-enable tests muted in #30982
2018-06-15 11:19:35 -04:00
Nhat Nguyen 8453ca638d
Upgrade to Lucene-7.4.0-snapshot-518d303506 (#31360) 2018-06-15 10:58:21 -04:00
Christoph Büscher 02346c20a2
Rankeval: Fold template test project into main module (#31203)
This change moves tests in `smoke-test-rank-eval-with-mustache` into the main
ranking evaluation module by declaring that the integration testing cluster
requires the `lang-mustache` plugin. This avoids having to maintain the qa
project for only one basic test suite.
2018-06-15 15:55:39 +02:00
Christoph Büscher a0d6c19e75
Add details section for dcg ranking metric (#31177)
While the other two ranking evaluation metrics (precicion and reciprocal rank)
already provide a more detailed output for how their score is calculated, the
discounted cumulative gain metric (dcg) and its normalized variant are lacking
this until now. Its not really clear which level of detail might be useful for
debugging and understanding the final metric calculation, but this change adds a
`metric_details` section to REST output that contains some information about the
evaluation details.
2018-06-15 11:56:16 +02:00
Tanguy Leroux 9c03b4844f Merge branch 'master' into ccr
* master:
  992c7889ee Uncouple persistent task state and status (#31031)
  8c6ee7db54 Describe how to add a plugin in Dockerfile (#31340)
  1c5cec0ac7 Remove http status code maps (#31350)
  87a676e4d5 Do not set vm.max_map_count when unnecessary (#31285)
  e5b7137508 TEST: getCapturedRequestsAndClear should be atomic (#31312)
  0324103737 Painless: Fix bug for static method calls on interfaces (#31348)
  d6d0727aac QA: Fix resolution of default distribution (#31351)
  fcf1e41e42 Extract common http logic to server (#31311)
  6dd81ead74 Build: Fix the license in the pom zip and tar (#31336)
  8f886cd4be Treat ack timeout more like a publish timeout (#31303)
  9b293275af [ML] Add description to ML filters (#31330)
  f7a0cafe55 SQL: Fix build on Java 10
  375d09c588 [TEST] Fix RemoteClusterClientTests#testEnsureWeReconnect
  4877cec3e8 More detailed tracing when writing metadata (#31319)
  bbfe1eccc7 [Tests] Mutualize fixtures code in BaseHttpFixture (#31210)
2018-06-15 10:40:31 +02:00
Tim Brooks 1c5cec0ac7
Remove http status code maps (#31350)
Currently we maintain a compatibility map of http status codes in both
the netty4 and nio modules. These maps convert a RestStatus to a netty
HttpResponseStatus. However, as these fundamentally represent integers,
we can just use the netty valueOf method to convert a RestStatus to a
HttpResponseStatus.
2018-06-14 20:16:40 -06:00
Jack Conradson 0324103737
Painless: Fix bug for static method calls on interfaces (#31348)
Static method calls on interfaces were not being called correctly
which was causing JVM crashes.  This change fixes the issue.
2018-06-14 18:30:37 -07:00
Tim Brooks fcf1e41e42
Extract common http logic to server (#31311)
This is related to #28898. With the addition of the http nio transport,
we now have two different modules that provide http transports.
Currently most of the http logic lives at the module level. However,
some of this logic can live in server. In particular, some of the
setting of headers, cors, and pipelining. This commit begins this moving
in that direction by introducing lower level abstraction (HttpChannel,
HttpRequest, and HttpResonse) that is implemented by the modules. The
higher level rest request and rest channel work can live entirely in
server.
2018-06-14 15:10:02 -06:00
Nhat Nguyen 2cffd85f7f Merge branch 'master' into ccr
* master:
  Remove RestGetAllAliasesAction (#31308)
  Temporary fix for broken build
  Reenable Checkstyle's unused import rule (#31270)
  Remove remaining unused imports before merging #31270
  Fix non-REST doc snippet
  [DOC] Extend SQL docs
  Immediately flush channel after writing to buffer (#31301)
  [DOCS] Shortens ML API intros
  Use quotes in the call invocation (#31249)
  move security ingest processors to a sub ingest directory (#31306)
  Add 5.6.11 version constant.
  Fix version detection.
  SQL: Whitelist SQL utility class for better scripting (#30681)
  [Docs] All Rollup docs experimental, agg limitations, clarify DeleteJob (#31299)
  CCS: don't proxy requests for already connected node (#31273)
  Mute ScriptedMetricAggregatorTests testSelfReferencingAggStateAfterMap
  [test] opensuse packaging turn up debug logging
  Add unreleased version 6.3.1
  Removes experimental tag from scripted_metric aggregation (#31298)
  [Rollup] Metric config parser must use builder so validation runs (#31159)
  [ML] Check licence when datafeeds use cross cluster search  (#31247)
  Add notion of internal index settings (#31286)
  Test: Remove broken yml test feature (#31255)
  REST hl client: cluster health to default to cluster level (#31268)
  [ML] Update test thresholds to account for changes to memory control (#31289)
  Log warnings when cluster state publication failed to some nodes (#31233)
  Fix AntFixture waiting condition (#31272)
  Ignore numeric shard count if waiting for ALL (#31265)
  [ML] Implement new rules design (#31110)
  index_prefixes back-compat should test 6.3 (#30951)
  Core: Remove plain execute method on TransportAction (#30998)
  Update checkstyle to 8.10.1 (#31269)
  Set analyzer version in PreBuiltAnalyzerProviderFactory (#31202)
  Modify pipelining handlers to require full requests (#31280)
  Revert upgrade to Netty 4.1.25.Final (#31282)
  Use armored input stream for reading public key (#31229)
  Fix Netty 4 Server Transport tests. Again.
  REST hl client: adjust wait_for_active_shards param in cluster health (#31266)
  REST high-level Client: remove deprecated API methods (#31200)
  [DOCS] Mark SQL feature as experimental
  [DOCS] Updates machine learning custom URL screenshots (#31222)
  Fix naming conventions check for XPackTestCase
  Fix security Netty 4 transport tests
  Fix race in clear scroll (#31259)
  [DOCS] Clarify audit index settings when remote indexing (#30923)
  Delete typos in SAML docs (#31199)
  REST high-level client: add Cluster Health API (#29331)
  [ML][TEST] Mute tests using rules (#31204)
  Support RequestedAuthnContext (#31238)
  SyncedFlushResponse to implement ToXContentObject (#31155)
  Add Get Aliases API to the high-level REST client (#28799)
  Remove some line length supressions (#31209)
  Validate xContentType in PutWatchRequest. (#31088)
  [INGEST] Interrupt the current thread if evaluation grok expressions take too long (#31024)
  Suppress extras FS on caching directory tests
  Revert "[DOCS] Added 6.3 info & updated the upgrade table. (#30940)"
  Revert "Fix snippets in upgrade docs"
  Fix snippets in upgrade docs
  [DOCS] Added 6.3 info & updated the upgrade table. (#30940)
  LLClient: Support host selection (#30523)
  Upgrade to Netty 4.1.25.Final (#31232)
  Enable custom credentials for core REST tests (#31235)
  Move ESIndexLevelReplicationTestCase to test framework (#31243)
  Encapsulate Translog in Engine (#31220)
  HLRest: Add get index templates API (#31161)
  Remove all unused imports and fix CRLF (#31207)
  [Tests] Fix self-referencing tests
  [TEST] Fix testRecoveryAfterPrimaryPromotion
  [Docs] Remove mention pattern files in Grok processor (#31170)
  Use stronger write-once semantics for Azure repository (#30437)
  Don't swallow exceptions on replication (#31179)
  Limit the number of concurrent requests per node (#31206)
  Call ensureNoSelfReferences() on _agg state variable after scripted metric agg script executions (#31044)
  Move java version checker back to its own jar (#30708)
  [test] add fix for rare virtualbox error (#31212)
2018-06-14 10:47:36 -04:00