Commit Graph

48578 Commits

Author SHA1 Message Date
Albert Zaharovits 69fc715bc3
Fix security origin for TokenService#findActiveTokensFor... (#47418) (#48280)
All internal searches (triggered by APIs) across the .security index
must be performed while "under the security origin". Otherwise,
the search is performed in the context of the caller which most
likely does not have privileges to search .security (hopefully).
This commit fixes this in the case of two methods in the
TokenService and corrects an overly done such context switch
in the ApiKeyService.

In addition, this makes all tests from the client/rest-high-level
module execute as an all mighty administrator,
but not a literal superuser.

Closes #47151
2019-10-21 13:15:05 +03:00
Alpar Torok fe265f0308 Mute CliSecurityIT
tracking in #48117
2019-10-21 11:25:56 +03:00
David Turner 672b2a92ca Fix compile error from previous commit (#48230)
The previous commit, 3a6fa0bbdb introduces a
compile error that was fixed locally but not committed. This commit adds the
missing change.
2019-10-21 08:54:04 +01:00
David Turner 3a6fa0bbdb Close query cache on index service creation failure (#48230)
Today it is possible that we create the `QueryCache` and then fail to create
the owning `IndexService` and this means we do not close the `QueryCache`
again. This commit addresses that leak.

Fixes #48186
2019-10-21 08:46:53 +01:00
Alpar Torok c903ac2376
Fix download of 6.x rpm and deb packages (#48228) 2019-10-21 10:03:44 +03:00
Ignacio Vera b1224fca8c
upgrade to Lucene-8.3.0-snapshot-25968e3b75e (#48227) 2019-10-21 08:21:09 +02:00
Hendrik Muhs 4b3a672a0b
blacklist transform tests in mixed cluster prior 7.4 (#48247)
blacklist transform tests in mixed cluster prior 7.4, see #48019

fixes #47943
2019-10-21 08:19:58 +02:00
Gordon Brown a2217f4a91
Fix testRetentionWhileSnapshotInProgress (#48219)
This test could fail for two reasons, both should be fixed by this PR:

1) It hit a timeout for an `assertBusy`. This commit increases the
timeout for that `assertBusy`.

2) The snapshot that was supposed to be blocked could, in fact, be
successful. This is because a previous snapshot had been successfully
been taken, and no new data had been added between the two snapshots.
This means that no new segment files needed to be written for the new
snapshot, so the block on data files was never triggered. This commit
changes two things: First, it indexes some new data before taking the
second snapshot (the one that needs to be blocked), and second,
checks to ensure that the block is actually hit before continuing
with the test.
2019-10-18 14:25:18 -06:00
Christoph Büscher 3ea666d694 Clarify mapping types that support ignore_malformed (#48206)
The `ignore_malformed` setting only works on selected mapping types, otherwise
we throw an mapper_parsing_exception. We should add a list of all the mapping
types that support it, since the number of types not supporting it seems larger.

Closes #47166
2019-10-18 20:39:38 +02:00
Alpar Torok cc26e30281
Increase timeout for yml tests (#48237)
Some of these are larger than what can complete in the regular timeout.

Closes #48212
2019-10-18 11:14:15 -07:00
Mark Vieira 0e24e6fdd1
Upgrade to build scan plugin 2.4.2 (#48220) 2019-10-18 11:04:52 -07:00
Martijn van Groningen 403728dc3d
Use correct policy type in test 2019-10-18 16:58:03 +02:00
Takuya Kajiwara a56daeae2d [DOCS] Fix typos in InternalEngine.java comments (#46861) 2019-10-18 10:36:58 -04:00
Guido Lena Cota 1beeb3a83e [DOCS] Fix typos in Painless execute API docs(#47518) 2019-10-18 10:31:04 -04:00
Martijn van Groningen 5cb44a414c
Fixed links in java docs for EnrichClient (#48233) 2019-10-18 16:26:49 +02:00
Alexander Reelsen 66581d8158
update ingest-user-agent regexes.yml (#47807)
This new regexes are from:
154eba17f5/regexes.yaml
2019-10-18 16:26:48 +02:00
Martijn van Groningen 844825a13f
Validate policy type when storing an enrich policy (#48126) 2019-10-18 16:26:48 +02:00
James Rodewig 1d06be2f05 [DOCS] Note clause limit in `index.mapping.total_fields.limit` docs (#48153)
Notes that you should increase the `indices.query.bool.max_clause_count`
setting if you increase the `index.mapping.total_fields.limit`
setting.

Closes #46928.
2019-10-18 10:20:49 -04:00
markharwood fbd43196a8 [DOCS] Add TermVectors API reference to MLT query docs (#37228)
Adds a reference the use of the TermVectors API following an issue raised about lack of flexibility in the MLT query: https://github.com/elastic/elasticsearch/issues/35509
2019-10-18 10:08:57 -04:00
Benjamin Trent 876f4aafac
[ML] Add logistic_regression output aggregator (#48238) (#48244) 2019-10-18 10:08:17 -04:00
David Turner a8bcbbc38a Quieter logging from the DiskThresholdMonitor (#48115)
Today if an Elasticsearch node reaches a disk watermark then it will repeatedly
emit logging about it, which implies that some action needs to be taken by the
administrator. This is misleading. Elasticsearch strives to keep nodes under
the high watermark, but it is normal to have a few nodes occasionally exceed
this level. Nodes may be over the low watermark for an extended period without
any ill effects.

This commit enhances the logging emitted by the `DiskThresholdMonitor` to be
less misleading. The expected case of hitting the high watermark and
immediately relocating one or more shards that to bring the node back under the
watermark again is reduced in severity to `INFO`. Additionally, `INFO` messages
are not emitted repeatedly.

Fixes #48038
2019-10-18 15:00:14 +01:00
rsarawgi 5e4dd0fd2e [ML] Removing usages of ToXContentParams.INCLUDE_TYPE (#48165)
Removing the option of ToXContentParams.INCLUDE_TYPE and replacing them with ToXContentParams.FOR_INTERNAL_STORAGE
Closes #48057
2019-10-18 14:49:26 +01:00
David Turner 6a97aef8ac Clarify contribution instructions re. license hdrs (#48067)
Today our instructions in `CONTRIBUTING.md` regarding license headers are a
little unclear. This commit clarifies the exact headers that are required for
contributions.
2019-10-18 14:45:03 +01:00
Alpar Torok a675b35680 Mute org.elasticsearch.xpack.sql.jdbc.JdbcConfigurationTests.testDriverConfigurationWithSSLInURL
tracked in #41557
2019-10-18 11:05:56 +01:00
Ioannis Kakavas ce3a06292b Mute flaky testCreateApiKey test (#47973)
see #47958
2019-10-18 09:52:07 +01:00
Armin Braun 1157775074
Remove Support for pre-5.x Indices in Restore (#48181) (#48199)
The logic for handling empty segment files has been
unnecessary ever since #24021 which removes the support
for these files in 6.x -> we can safely remove the
support for restoring these from 7.x+ to simplify the code.
2019-10-18 09:45:07 +02:00
Alpar Torok ca99014d8b Create an upload report once the build completes (#47642)
* Create an upload report once the build completes

We used to have this logic in Jenkins, but that forced us to make it
platform dependent and gave us less control on what to include here.

With this change we create a single archive to be uploaded after the
build completes, and we include a link in the build scan to where we
know this should get uploaded.

* Fix when there's nothign to upload
* Log the directory size
* Switch to ant to walk the project tree
* Collect journlas
* Filter for regular files
* only call journalctl on unix where we have bash
* Grab only logs fro this gradle version
* restrict demon log to relevant one
2019-10-18 08:15:49 +03:00
Przemyslaw Gomulka 02d18f5c1e
[7.x] Slow log must use separate underlying logger for each index BACKPORT(#47234) (#48176)
* Slow log must use separate underlying logger for each index (#47234)

SlowLog instances should not share the same underlying logger, as it would cause different indexes override each other levels. When creating underlying logger, unique per index identifier should be used. Name + IndexSettings.UUID

Closes #42432
2019-10-17 20:04:57 +02:00
Armin Braun 04e3316408
Stop Resolving Fallback IndexId (#48141) (#48204)
There is no reason to still resolve the
fallback `IndexId` here. It only applies to
`2.x` repos and those we can't read anymore
anyway because they use an `/index` instead of
an `/index-N` blob at the repo root for which
at least 7.x+ does not contain the logic to find
it.
2019-10-17 19:27:49 +02:00
Przemysław Witek 28f68fa221
Make num_top_classes parameter's default value equal to 2 (#48119) (#48201) 2019-10-17 18:43:15 +02:00
Dimitris Athanasiou e0489fc328
[7.x][ML] Always refresh dest index before starting analytics process (#48090) (#48196)
If a job stops right after reindexing is finished but before
we refresh the destination index, we don't refresh at all.
If the job is started again right after, it jumps into the analyzing state.
However, the data is still not searchable.
This is why we were seeing test failures that we start the process
expecting X rows (where X is lower than the expected number of docs)
and we end up getting X+.

We fix this by moving the refresh of the dest index right before
we start the process so it always ensures the data is searchable.

Closes #47612

Backport of #48090
2019-10-17 17:20:19 +01:00
Gordon Brown eb7969e8cc
Fix ILM HLRC Javadoc->Documentation links (#48083)
Several links from the ILM HLRC Javadoc to the online documentation were
not updated when the ILM HLRC documentation was written. This commit
fixes those links.
2019-10-17 09:59:40 -06:00
Stuart Tettemer 356eef00c8
Scripting: get context names REST API (#48026) (#48168)
Adds `GET /_script_context`, returning a `contexts` object with each
available context as a key whose value is an empty object. eg.
```
{
  "contexts": {
    "aggregation_selector": {},
    "aggs": {},
    "aggs_combine": {},
...
  }
}
```

refs: #47411
2019-10-17 09:08:55 -06:00
William Brafford f46281a251
Muting test that fails on Windows (#48190)
This test has been failing very frequently on Windows checks for 7.4. We've also seen it for 7.x as well as for the new 7.5 tests. I'm muting during investigation so we can see if this is masking any other issues.
2019-10-17 10:54:12 -04:00
Mark Vieira befc44c145
Cache all rest tests tasks so long as they don't use shared clusters (#48161) 2019-10-17 07:30:55 -07:00
James Rodewig 4c6d56bef0 [DOCS] Expand put mapping API docs examples (#47462) 2019-10-17 09:31:52 -04:00
James Rodewig b4bf47cce4 [DOCS] Document ingest pipeline and processor node stats (#47884) 2019-10-17 09:19:45 -04:00
Anton b54d817f1d [DOCS] Fix typo in intervals query docs (#48180) 2019-10-17 09:15:55 -04:00
Hendrik Muhs 06f54428c2 update internal index name in documentation warning (#48125) 2019-10-17 15:09:47 +02:00
Alpar Torok 13df6beb53 Fix packaging tests on debian 10 (#48138)
* Fix packaging tests on debian 10

- lintian no longer has  `--fail-on-warnings`
- Systemd has a build number / version after it's acutal version
```
alpar@alpar-ci-debian-10-1015-1517:~$ systemd --version
systemd 241 (241)
+PAM +AUDIT +SELINUX +IMA +APPARMOR +SMACK +SYSVINIT +UTMP +LIBCRYPTSETUP +GCRYPT +GNUTLS +ACL +XZ +LZ4 +SECCOMP +BLKID +ELFUTILS +KMOD -IDN2 +IDN -PCRE2 default-hierarchy=hybrid
```
2019-10-17 14:29:33 +03:00
István Zoltán Szabó ecd62934f6 [DOCS] Applies rename of transform related roles and privileges (#48123) 2019-10-17 10:24:56 +02:00
Martijn van Groningen a5fe69c344
Include enrich into the info api as feature (#48157)
This commit also fixes a bug, the enrich enabled setting
was not included in the list of settings.

Backport of #48109
2019-10-17 09:51:32 +02:00
Mark Vieira fa50377dc6 Explicitly declare test fixture usage (#48166) 2019-10-17 10:05:40 +03:00
Armin Braun 9bf8e1e060
Fix SLMSnapshotBlockingIntegTest (#47941) (#47963)
The after snapshot action is interfering with SLM deleting snapshots
here it seems, causing concurrent delete exceptions.
Since these tests are now test-scoped there is no reason to run
snapshot deletes after each test so we can remove them to avoid this issue.

Closes #47937
2019-10-17 08:55:56 +02:00
Armin Braun 0ca7cc1848
Safely Close Repositories on Node Shutdown (#48020) (#48107)
We were not closing repositories on Node shutdown.
In production, this has little effect but in tests
shutting down a node using `MockRepository` and is
currently stuck in a simulated blocked-IO situation
will only unblock when the node's threadpool is
interrupted. This might in some edge cases (many
snapshot threads and some CI slowness) result
in the execution taking longer than 5s to release
all the shard stores and thus we fail the assertion
about unreleased shard stores in the internal test cluster.

Regardless of tests, I think we should close repositories
and release resources associated with them when closing
a node and not just when removing a repository from the CS
with running nodes as this behavior is really unexpected.

Fixes #47689
2019-10-17 07:55:05 +02:00
Armin Braun f1bc3a0753
Remove TestLogging for #46701 (#48156) (#48160)
This hasn't failed in 5 weeks now. Removing
the test logging and closing the issue.

Closes #46701
2019-10-17 07:54:20 +02:00
Lee Hinman 5af66d79ef
Add SLM support to xpack usage and info APIs (#48149)
* Add SLM support to xpack usage and info APIs

This is a backport of #48096

This adds the missing xpack usage and info information into the
`/_xpack` and `/_xpack/usage` APIs. The output now looks like:

```
GET /_xpack/usage
{
  ...
  "slm" : {
    "available" : true,
    "enabled" : true,
    "policy_count" : 1,
    "policy_stats" : {
      "retention_runs" : 0,
      ...
    }
  }
```

and

```
GET /_xpack
{
  ...
  "features" : {
    ...
    "slm" : {
      "available" : true,
      "enabled" : true
    },
    ...
  }
}
```

Relates to #43663

* Fix missing license
2019-10-16 21:06:27 -06:00
Jack Conradson fa99721295 Drop stored scripts with the old style-id (#48078)
This PR fixes (#47593). Stored scripts with the old-style id of lang#id are 
saved through the upgrade process but are no longer accessible in recent 
versions. This fix will drop those scripts altogether since there is no way for 
a user to access them.
2019-10-16 16:10:31 -07:00
William Brafford ac1ed6e740 Update TESTING.asciidoc for new :qa:os test style (#48023)
* Update testing document for new packaging tests

The TESTING.asciidoc document had gotten out of date due to some new and
wonderful changes in our vagrant testing code. I've removed all of the
instructions that no longer work, and added working examples and descriptions
in their place.
2019-10-16 17:26:06 -04:00
David Woods 73f5fef678 Add a note to query_string docs around spaces in field names. (#47326) 2019-10-16 13:28:55 -07:00