Commit Graph

244 Commits

Author SHA1 Message Date
Martijn van Groningen 43773a32a4
Replace Streamable w/ Writeable in BaseTasksRequest and subclasses (#35854)
* Replace Streamable w/ Writeable in BaseTasksRequest and subclasses

This commit replaces usages of Streamable with Writeable for the
BaseTasksRequest / TransportTasksAction classes and subclasses of
these classes.

Relates to #34389
2018-12-03 08:04:29 +01:00
Dimitris Athanasiou 54cf1f9d74
[ML] Refactor control message writer to allow reuse for other processes (#36070) 2018-11-30 09:25:35 +00:00
Zachary Tong 61c2db5ebb Revert "Deprecate X-Pack centric rollup endpoints (#35962)"
This reverts commit b84f1f6a3a.
2018-11-29 12:58:23 -05:00
Jason Tedor b84f1f6a3a
Deprecate X-Pack centric rollup endpoints (#35962)
This commit is part of our plan to deprecate and ultimately remove the
use of _xpack in the REST APIs.
2018-11-27 20:34:17 -05:00
Jay Modi 2061eeb122
Remove use of AbstractComponent in xpack (#35394)
This commit removes the use of AbstractComponent in xpack where it was
still being extended. It has been replaced with explicit logger
declarations.

See #34488
2018-11-27 11:28:26 -07:00
David Roberts 110c4fdd65
[ML] Adjust file structure finder parser config (#35935) 2018-11-27 12:52:52 +00:00
Benjamin Trent 6d4a3f8fce
Removes two unused AnalysisConfig options (#35645)
* ML: Removing result_finalization_window && overlapping_buckets

* Reverting bad method deletions

* Setting to current before backport to try and get a green build

* fixing testBuildAutodetectCommand test

* disabling bwc tests for backport
2018-11-19 08:29:53 -06:00
Benjamin Trent bc7dea4480
ML: changing automatic check_window calculation (#35643)
* ML: changing automatic check_window calculation

* adding docs on how we calculate the default
2018-11-19 08:03:34 -06:00
Arthur Gavlyukovskiy 022726011c Remove use of AbstractComponent in server (#35444)
Removed extending of AbstractComponent and changed logger usage to
explicit declaration. Abstract classes still have logger
declaration using this.getClass() in order to show implementation class
name in its logs.

See #34488
2018-11-16 16:10:32 -05:00
Benjamin Trent f7ada9b29b
Add delayed datacheck to the datafeed job runner (#35387)
* ML: Adding missing datacheck to datafeedjob

* Adding client side and docs

* Making adjustments to validations

* Making values default to on, having more sensible limits

* Intermittent commit, still need to figure out interval

* Adjusting delayed data check interval

* updating docs

* Making parameter Boolean, so it is nullable

* bumping bwc to 7 before backport

* changing to version current

* moving delayed data check config its own object

* Separation of duties for delayed data detection

* fixing checkstyles

* fixing checkstyles

* Adjusting default behavior so that null windows are allowed

* Mentioning the default value

* Fixing comments, syncing up validations
2018-11-15 13:32:45 -06:00
Tanguy Leroux c9b4ef0dfd
Use RunOnce when appropriate (#35553)
This pull request replaces some blocks of code that must be run once 
and that are currently based on AtomicBoolean by the convient RunOnce 
class added in #35489.
2018-11-15 09:24:40 +01:00
David Roberts 09965cb370
[ML] Fix find_file_structure NPE with should_trim_fields (#35465)
The NPE would occur if should_trim_field was overridden to
true and any field value was completely blank.  This change
defends against this situation.

Fixes #35462
2018-11-13 08:49:24 +00:00
David Kyle 9494e046e7
[ML] Prevent notifications on deletion of a non existent job (#35337) 2018-11-08 09:57:07 +00:00
Jason Tedor 4f4fc3b8f8
Replicate index settings to followers (#35089)
This commit uses the index settings version so that a follower can
replicate index settings changes as needed from the leader.

Co-authored-by: Martijn van Groningen <martijn.v.groningen@gmail.com>
2018-11-07 21:20:51 -05:00
Benjamin Trent 2117f4f358
[ML] Add Missing data checking class (#35310)
* ML: Adding missing data check class

* reverting bad change

* Adding bucket + missing data object for returns

* reverting unnecessary change

* adding license header

* Make client calls synchronous, akin to DatafeedJob

* Fixing line length

* Renaming things, addressing PR comments
2018-11-07 12:48:15 -06:00
Nik Everett f72ef9b5fd
Build: Pull "skip assemble on qa" to common build (#35214)
Pull all of the logic that we use to skip the `assemble` and
`dependenciesInfo` tasks on `qa` projects into one spot in our root
build file.
2018-11-05 16:16:00 -05:00
Alexander Reelsen 409050e8de
Refactor: Remove settings from transport action CTOR (#35208)
As settings are not used in the transport action constructor, this
removes the passing of the settings in all the transport actions.
2018-11-05 13:08:18 +01:00
David Kyle 85f8458f06 [ML] Add comment describing test behaviour 2018-11-05 11:21:59 +00:00
Tal Levy c3cf7dd305 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-11-01 10:13:02 -07:00
Benjamin Trent 2fadec5c3d
ML: Add support for rollup Indexes in Datafeeds (#34654)
* Adding rollup support for datafeeds

* Fixing tests and adjusting formatting

* minor formatting chagne

* fixing some syntax and removing redundancies

* Refactoring and fixing failing test

* Refactoring, adding paranoid null check

* Moving rollup into the aggregation package

* making AggregationToJsonProcessor package private again

* Addressing test failure

* Fixing validations, chunking

* Addressing failing test

* rolling back RollupJobCaps changes

* Adding comment and cleaning up test

* Addressing review comments and test failures

* Moving builder logic into separate methods

* Addressing PR comments, adding test for rollup permissions

* Fixing test failure

* Adding rollup priv check on datafeed put

* Handling missing index when getting caps

* Fixing unused import
2018-11-01 10:02:24 -05:00
Nik Everett e28509fbfe
Core: Less settings to AbstractComponent (#35140)
Stop passing `Settings` to `AbstractComponent`'s ctor. This allows us to
stop passing around `Settings` in a *ton* of places. While this change
touches many files, it touches them all in fairly small, mechanical
ways, doing a few things per file:
1. Drop the `super(settings);` line on everything that extends
`AbstractComponent`.
2. Drop the `settings` argument to the ctor if it is no longer used.
3. If the file doesn't use `logger` then drop `extends
AbstractComponent` from it.
4. Clean up all compilation failure caused by the `settings` removal
and drop any now unused `settings` isntances and method arguments.

I've intentionally *not* removed the `settings` argument from a few
files:
1. TransportAction
2. AbstractLifecycleComponent
3. BaseRestHandler

These files don't *need* `settings` either, but this change is large
enough as is.

Relates to #34488
2018-10-31 21:23:20 -04:00
David Turner 0072c90e2a
Pre-populate unicast hosts files (#35136)
Today when ESIntegTestCase starts some nodes it writes out the unicast hosts
files each time a node starts its transport service. This does mean that a
number of nodes can start and perform their first pinging round without any
unicast hosts which, if the timing is unlucky and a lot of nodes are all
started at the same time, can lead to a split brain as in #35052.

Prior to #33554 this was unlikely to happen since the MockUncasedHostsProvider
would always have yielded the existing hosts, so the timing would have to have
been implausibly unlucky. Since #33554, however, it's more likely because the
race occurs between the start of the first round of pinging and the writing of
the unicast hosts file. It is realistic that new nodes will be configured with
the existing nodes from startup, so this change reinstates that behaviour.

Closes #35052.
2018-10-31 19:21:24 +00:00
Tal Levy d5d28420b6 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-31 10:47:07 -07:00
Dimitris Athanasiou 00dc2ba36f
[ML] Enable reusing field extraction logic when no time field is required (#35100) 2018-10-31 10:55:11 +00:00
Nik Everett 086ada4c08
Core: Drop settings member from AbstractComponent (#35083)
Drops the `Settings` member from `AbstractComponent`, moving it from the
base class on to the classes that use it. For the most part this is a
mechanical change that doesn't drop `Settings` accesses. The one
exception to this is naming threads where it switches from an invocation
that passes `Settings` and extracts the node name to one that explicitly
passes the node name.

This change doesn't drop the `Settings` argument from
`AbstractComponent`'s ctor because this change is big enough as is.
We'll do that in a follow up change.
2018-10-30 16:10:38 -04:00
Tal Levy 18c72e86c5 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-30 08:09:57 -07:00
David Turner f7760ddacb Increase discovery log level in NetworkDisruptionIT
Sometimes the cluster forming here will split-brain when it grows up to 5
nodes. This could be a timing issue or could be something going wrong in
discovery, so this asks for more logs. Relates #35052
2018-10-30 08:45:23 +00:00
Dimitris Athanasiou d85a654ebb
[ML] Refactor doc value format into ExtractedField (#35053)
This commit moves the knowledge of which doc value format
to be used down to the `ExtractedField` instead of being
in the data extractor.
2018-10-29 22:56:53 +00:00
Tal Levy c9e4d26a53 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-29 14:03:55 -07:00
Pratik Sanglikar f1135ef0ce Core: Replace deprecated Loggers calls with LogManager. (#34691)
Replace deprecated Loggers calls with LogManager.

Relates to #32174
2018-10-29 15:52:30 -04:00
David Roberts c455be7bc2
[ML] Rename the json file structure to ndjson (#34901)
The file structure finder endpoint can find the NDJSON
(newline-delimited JSON) file format, but called it
`json`.  This change renames the `format` for this file
structure to `ndjson`, which is more precise and will
hopefully avoid confusion.
2018-10-29 10:06:12 +01:00
Tal Levy d8322ca069 Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-26 12:46:21 -07:00
Nik Everett 10295b306d
Core: Drop nodeName from AbstractComponent (#34487)
`AbstractComponent` is trouble because its name implies that
*everything* should extend from it. It *is* useful, but maybe too
broadly useful. The things it offers access too, the `Settings` instance
for the entire server and a logger are nice to have around, but not
really needed *everywhere*. The `Settings` instance especially adds a
fair bit of ceremony to testing without any value.

This removes the `nodeName` method from `AbstractComponent` so it is
more clear where we actually need the node name.
2018-10-26 15:26:14 -04:00
Dimitris Athanasiou a39a67cd38
[ML] Extract common native process base class (#34856)
We currently have two different native processes:
autodetect & normalizer. There are plans for introducing
a new process. All these share many things in common.
This commit refactors the processes to extend an
`AbstractNativeProcess` class that encapsulates those
commonalities with the purpose of reusing the code
for new processes in the future.
2018-10-26 15:34:48 +01:00
David Roberts 734088673e
[ML] Include message in field_stats for text log files (#34861)
This change ensures the `message` field is always included
in the `field_stats` for the semi-structured text log file
file structure.  Previously it was not, as it will almost
certainly contain all distinct values.  However, for
consistency in the UI it's useful to include it.
2018-10-26 07:45:02 +02:00
Tal Levy e1fdd00420
Lowercase static final DeprecationLogger instance names (#34887)
After discussing on the team's FixItFriday, we concluded that
static final instance variables that are mutable should be lowercased.

Historically, DeprecationLogger was uppercased more frequently than lowercased.
2018-10-25 21:12:19 -07:00
Lee Hinman 3e7042832a Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-10-25 11:00:36 -06:00
Alpar Torok 795d57b4f9
Auto configure all test tasks (#34666)
With this change, we apply the common test config automatically to all
newly created tasks instead of opting in specifically.

For plugin authors using the plugin externally this means that the
configuration will be applied to their RandomizedTestingTasks as well.

The purpose of the task is to simplify setup and make it easier to
change projects that use the `test` task but actually run integration
tests to use a task called `integTest` for clarity, but also because
we may want to configure and run them differently.
E.x. using different levels of concurrency.
2018-10-24 16:05:50 +03:00
lipsill d5ad3de42e [test] Introduce strict deprecation mode for REST tests (#34338)
#33708 introduced a strict deprecation mode that makes a REST request
fail if there is a warning header in the response returned by
Elasticsearch (usually a deprecation message signaling that a feature
or a field has been deprecated).

This change adds the strict deprecation mode into the REST integration
tests, and makes the tests fail if a deprecated feature is used. Also
any test using a deprecated feature has been modified to pass the build.

The YAML integration tests already analyzed HTTP warnings so they do
not use this mode, keeping their "expected vs actual" behavior.
2018-10-24 08:21:24 -04:00
Ryan Ernst 596b5cf108
Test: Fix last reference to SearchScript (#34731)
This was accidentally left over when converting to FieldScript.

closes #34683
2018-10-23 17:26:17 -07:00
Tal Levy 67bfdb16ad Merge remote-tracking branch 'upstream/master' into index-lifecycle 2018-10-22 13:09:37 -07:00
jaymode ae1e46b852
Test: add empty test to PainlessDomainSplitIT
All of the tests in PainlessDomainSplitIT have an awaitsfix, which
causes the build to fail since no tests are run. This adds an empty
test to get the build going again.

Relates #34683
Relates #32966
2018-10-22 14:01:44 -06:00
Jason Tedor 7af19b8f81
Migrate wait for pending tasks helper to server (#34675)
In some of our X-Pack REST tests we have to wait for pending tasks to
complete. We are now needing this functionality in ESRestTestCase for
the docs tests where we run against X-Pack features. This commit moves
the helper method that we have in X-Pack to ESRestTestCase, and removes
duplicate logic from waiting for rollup tasks to complete.
2018-10-22 11:14:02 -04:00
Jason Tedor e562afad69
Awaits fix PainlessDomainSplitIT#testIsolated
This test fails reliably with a compilation error. This commit awaits
fix the test.
2018-10-21 19:13:53 -04:00
Colin Goodheart-Smithe 84ef91529c
Merge branch 'master' into index-lifecycle 2018-10-19 13:24:04 +01:00
David Roberts 0f8d05f2d0
[TEST] Reduce forecast disk space requirement for tests (#34552)
The setting that reduces the disk space requirement
for the forecasting integration tests was accidentally
removed in #31757 when files were moved around.  This
change simply adds back the setting that existed before
that.
2018-10-18 12:43:41 +01:00
Ryan Ernst d445785f1a
Scripting: Convert domainSplit function for ML to whitelist (#34426)
This commit moves the definition of domainSplit into java and exposes it
as a painless whitelist extension. The method also no longer needs
params, and version which ignores params is added and deprecated.
2018-10-17 15:54:21 -07:00
Colin Goodheart-Smithe 90f7cec7a5
Merge branch 'master' into index-lifecycle 2018-10-17 18:22:23 +01:00
Benjamin Trent fb579d2d9a
ML: Adding support for lazy nodes (#29991) (#34538) 2018-10-17 08:30:15 -05:00
Colin Goodheart-Smithe 0b42eda0e3
Merge branch 'master' into index-lifecycle 2018-10-15 16:03:37 +01:00
David Roberts 21c759af0e
[ML] Add an ingest pipeline definition to structure finder (#34350)
The ingest pipeline that is produced is very simple.  It
contains a grok processor if the format is semi-structured
text, a date processor if the format contains a timestamp,
and a remove processor if required to remove the interim
timestamp field parsed out of semi-structured text.

Eventually the UI should offer the option to customize the
pipeline with additional processors to perform other data
preparation steps before ingesting data to an index.
2018-10-12 07:56:35 +01:00
David Turner 7352f0da60
Handle pre-6.x time fields (#34373)
In ccb9ab5717 we changed how we deal with time
fields to support the `DateTime`-format fields added in 6.0, but dropped
support for pre-6.x `Long`-format fields. This change reinstates this support
for cases where pre-6.x data is made available to ML (e.g. in a mixed-version
CCS setup or after an upgrade).
2018-10-11 15:33:09 +01:00
Dimitris Athanasiou 4dacfa95d2
[ML] Allow asynchronous job deletion (#34058)
This changes the delete job API by adding
the choice to delete a job asynchronously.
The commit adds a `wait_for_completion` parameter
to the delete job request. When set to `false`,
the action returns immediately and the response
contains the task id.

This also changes the handling of subsequent
delete requests for a job that is already being
deleted. It now uses the task framework to check
if the job is being deleted instead of the cluster
state. This is a beneficial for it is going to also
be working once the job configs are moved out of the
cluster state and into an index. Also, force delete
requests that are waiting for the job to be deleted
will not proceed with the deletion if the first task
fails. This will prevent overloading the cluster. Instead,
the failure is communicated better via notifications
so that the user may retry.

Finally, this makes the `deleting` property of the job
visible (also it was renamed from `deleted`). This allows
a client to render a deleting job differently.

Closes #32836
2018-10-05 02:41:28 +03:00
David Kyle ef5007b6d8
[ML] Remove unused last_data_time member from Job (#34262) 2018-10-04 13:16:14 +01:00
Kazuhiro Sera d45fe43a68 Fix a variety of typos and misspelled words (#32792) 2018-10-03 18:11:38 +01:00
Lee Hinman 2d9cb21490 Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-10-01 14:10:09 -06:00
Benjamin Trent 96be057195
Removing unused ML parameters (#34159) 2018-10-01 08:09:46 -07:00
David Roberts a1d2ded98d
[ML] Fix unit test deadlock problem (#34174)
This change fixes a potential deadlock problem in the unit
test introduced in #34117.

It also removes a piece of debug code and corrects a docs
formatting problem that were both added in that same PR.
2018-10-01 15:35:37 +01:00
Lee Hinman 6ea396a476 Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-09-28 15:40:12 -06:00
David Roberts f709c2f694
[ML] Add a timeout option to file structure finder (#34117)
This can be used to restrict the amount of CPU a single
structure finder request can use.

The timeout is not implemented precisely, so requests
may run for slightly longer than the timeout before
aborting.

The default is 25 seconds, which is a little below
Kibana's default timeout of 30 seconds for calls to
Elasticsearch APIs.
2018-09-28 17:32:35 +01:00
Lee Hinman a26cc1a242 Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-09-27 11:00:37 -06:00
Christoph Büscher ba3ceeaccf
Clean up "unused variable" warnings (#31876)
This change cleans up "unused variable" warnings. There are several cases were we 
most likely want to suppress the warnings (especially in the client documentation test
where the snippets contain many unused variables). In a lot of cases the unused
variables can just be deleted though.
2018-09-26 14:09:32 +02:00
Ed Savage cc70352b3f
[ML] Modify thresholds for normalization triggers (#33663)
[ML] Modify thresholds for normalization triggers

The (arbitrary) threshold factors used to judge if scores have
changed significantly enough to trigger a look-back renormalization have
been changed to values that reduce the frequency of such
renormalizations.

Added a clause to treat changes in scores as a 'big change' if it would
result in a change of severity reported in the UI.

Also altered the clause affecting small scores so that a change should
be considered big if scores have changed by at least 1.5. 

Relates https://github.com/elastic/machine-learning-qa/issues/263
2018-09-25 15:30:10 +01:00
David Roberts dfe5af0411
[ML] Return both Joda and Java formats from structure finder (#33900)
Previously the timestamp_formats field in the response
from the find_file_structure endpoint contained Joda
timestamp formats.  This change makes that clear by
renaming the field to joda_timestamp_formats, and also
adds a java_timestamp_formats field containing the
equivalent Java time format strings.
2018-09-25 12:52:51 +01:00
Benjamin Trent 74d7be805a
Make certain ML node settings dynamic (#33565) (#33961)
* Make certain ML node settings dynamic (#33565)

* Changing to pull in updating settings and pass to constructor

* adding note about only newly opened jobs getting updated value
2018-09-24 12:54:32 -07:00
Lee Hinman 243e863f6e Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-09-24 10:33:51 -06:00
David Roberts b89551c452
[ML] Display integers without .0 in file structure field stats (#33947)
Previously numeric values in the field_stats created by the
find_file_structure endpoint were always output with a
decimal point.  This looked unfriendly and unnatural for
fields that clearly store integer values.  This change
converts integer values to type Integer before output in
the file structure field stats.
2018-09-22 15:48:59 +01:00
Benjamin Trent e17bd8e913
Removing poor randomization for node name (#33918) 2018-09-21 04:49:20 -07:00
Christoph Büscher b654d986d7
Add OneStatementPerLineCheck to Checkstyle rules (#33682)
This change adds the OneStatementPerLineCheck to our checkstyle precommit
checks. This rule restricts the number of statements per line to one. The
resoning behind this is that it is very difficult to read multiple statements on
one line. People seem to mostly use it in short lambdas and switch statements in
our code base, but just going through the changes already uncovered some actual
problems in randomization in test code, so I think its worth it.
2018-09-21 11:52:31 +02:00
Dimitris Athanasiou 8e3a0fad9d
[ML] Refactor job deletion logic into the transport action (#33891)
The job deletion logic was scattered around a few places:
the transport action, the job manager and the deletion task.
Overloading the task with deletion logic also meant extra
dependencies in the core package which should be unnecessary.

This commit consolidates all this logic into the transport action
and replaces the deletion task with a plain one that needs not be
aware of deletion logic.
2018-09-20 15:48:42 +01:00
Benjamin Trent 4767a016a5
Adding node_count to ML Usage (#33850) (#33863) 2018-09-19 13:35:09 -07:00
Lee Hinman 81e9150c7a Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-09-19 09:43:26 -06:00
Alan Woodward 5107949402
Allow TokenFilterFactories to rewrite themselves against their preceding chain (#33702)
We currently special-case SynonymFilterFactory and SynonymGraphFilterFactory, which need to 
know their predecessors in the analysis chain in order to correctly analyze their synonym lists. This
special-casing doesn't work with Referring filter factories, such as the Multiplexer or Conditional
filters. We also have a number of filters (eg the Multiplexer) that will break synonyms when they
appear before them in a chain, because they produce multiple tokens at the same position.

This commit adds two methods to the TokenFilterFactory interface.

* `getChainAwareTokenFilterFactory()` allows a filter factory to rewrite itself against its preceding
  filter chain, or to resolve references to other filters. It replaces `ReferringFilterFactory` and
  `CustomAnalyzerProvider.checkAndApplySynonymFilter`, and by default returns `this`.
* `getSynonymFilter()` defines whether or not a filter should be applied when building a synonym
  list `Analyzer`. By default it returns `true`.

Fixes #33609
2018-09-19 15:52:14 +01:00
Benjamin Trent 4190a9f1e9
Delete custom index if the only contained job is deleted (#33788)
* Delete custom index if the only contained job is deleted
2018-09-19 07:42:26 -07:00
Lee Hinman e6cbaa5a78 Merge remote-tracking branch 'origin/master' into index-lifecycle 2018-09-14 16:27:37 -06:00
David Roberts 568ac10ca6
[ML] Allow overrides for some file structure detection decisions (#33630)
This change modifies the file structure detection functionality
such that some of the decisions can be overridden with user
supplied values.

The fields that can be overridden are:

- charset
- format
- has_header_row
- column_names
- delimiter
- quote
- should_trim_fields
- grok_pattern
- timestamp_field
- timestamp_format

If an override makes finding the file structure impossible then
the endpoint will return an exception.
2018-09-14 09:29:11 +01:00
Benjamin Trent 7e51b960fb
Adding index refresh (#33647) 2018-09-13 10:44:33 -07:00
Colin Goodheart-Smithe 8e59de3eb2
Merge branch 'master' into index-lifecycle 2018-09-13 09:46:14 +01:00
Jay Modi 20c6c9c542
Address license state update/read thread safety (#33396)
This change addresses some issues regarding thread safety around
updates and method calls on the XPackLicenseState object. There exists
a possibility that there could be a concurrent update to the
XPackLicenseState when there is a scheduled check to see if the license
is expired and a cluster state update. In order to address this, the
update method now has a synchronized block where member variables are
updated. Each method that reads these variables is now also
synchronized.

Along with the above change, there was a consistency issue around
security calls to the license state. The majority of security checks
make two calls to the license state, which could result in incorrect
behavior due to the checks being made against different license states.
The majority of this behavior was introduced for 6.3 with the inclusion
of x-pack in the default distribution. In order to resolve the majority
of these cases, the `isSecurityEnabled` method is no longer public and
the logic is also included in individual methods about security such as
`isAuthAllowed`. There were a few cases where this did not remove
multiple calls on the license state, so a new method has been added
which creates a copy of the current license state that will not change.
Callers can use this copy of the license state to make decisions based
on a consistent view of the license state.
2018-09-12 13:08:09 -06:00
David Roberts 8e05ce567f
[ML] Rename input_fields to column_names in file structure (#33568)
This change tightens up the meaning of the "input_fields" field
in the file structure finder output.  Previously it was permitted
but not calculated for JSON and XML files.  Following this change
the field is called "column_names" and is only permitted for
delimited files.

Additionally the way the column names are set for headerless
delimited files is refactored to encapsulate the way they're
named to one line of the code rather than having the same
logic in two places.
2018-09-11 08:46:26 +01:00
Colin Goodheart-Smithe cdc4f57a77
Merge branch 'master' into index-lifecycle 2018-09-10 21:30:44 +01:00
Dimitris Athanasiou fcb15b0ce3
[ML] Get job stats request should filter non-ML job tasks (#33516)
When requesting job stats for `_all`, all ES tasks are accepted
resulting to loads of cluster traffic and a memory overhead.
This commit correctly filters out non ML job tasks.

Closes #33515
2018-09-09 22:53:03 +01:00
Nhat Nguyen 94e4cb64c2
Bootstrap a new history_uuid when force allocating a stale primary (#33432)
This commit ensures that we bootstrap a new history_uuid when force
allocating a stale primary. A stale primary should never be the source
of an operation-based recovery to another shard which exists before the
forced-allocation.

Closes #26712
2018-09-08 19:29:31 -04:00
David Roberts e42cc5cd8c
[ML] Add a file structure determination endpoint (#33471)
This endpoint accepts an arbitrary file in the request body and
attempts to determine the structure.  If successful it also
proposes mappings that could be used when indexing the file's
contents, and calculates simple statistics for each of the fields
that are useful in the data preparation step prior to configuring
machine learning jobs.
2018-09-07 17:41:57 +01:00
Colin Goodheart-Smithe 017ffe5d12
Merge branch 'master' into index-lifecycle 2018-09-07 10:59:10 +01:00
Jim Ferenczi 79cd6385fe
Collapse package structure for metrics aggs (#33463)
This change collapses all metrics aggregations classes into a single package `org.elasticsearch.aggregations.metrics`.
It also restricts the visibility of some classes (aggregators and factories) that should not be used outside of the package.

Relates #22868
2018-09-07 10:58:06 +02:00
David Roberts 0849b98f60
[ML] Rename log structure to file structure (#33421)
Many files supplied to the upcoming ML data preparation
functionality will not be "log" files.  For example,
CSV files are generally not "log" files.  Therefore it
makes sense to rename library that determines the
structure of these files.

Although "file structure" could be considered too broad,
as the library currently only works with a few text
formats, in the future it may be extended to work with
more formats.
2018-09-06 09:13:08 +01:00
Tal Levy b5f7fb6882 Merge branch 'master' into index-lifecycle 2018-09-05 12:56:58 -07:00
David Roberts a296829205
[ML] Add field stats to log structure finder (#33351)
The log structure endpoint will return these in addition to
pure structure information so that it can be used to drive
pre-import data visualizer functionality.

The statistics for every field are count, cardinality
(distinct count) and top hits (most common values).  Extra
statistics are calculated if the field is numeric: min, max,
mean and median.
2018-09-05 12:57:20 +01:00
Colin Goodheart-Smithe f00a28a909
Merge branch 'master' into index-lifecycle 2018-09-05 09:48:48 +01:00
Nik Everett ebd5eb6dc2 ML: Fix build after HLRC change
I recently merged a HLRC change that passed the PR builds but didn't
compile after merging. Sad time. This fixes the compilation.
2018-09-04 11:10:44 -04:00
Sohaib Iftikhar 761e8c461f HLRC: Add delete by query API (#32782)
Adds the delete-by-query API to the High Level REST Client.
2018-09-04 08:56:26 -04:00
Dimitris Athanasiou 1457b07a06
[ML] The sort field on get records should default to the record_score (#33358)
This is not changing the behaviour as when the sort field was set
to `influencer_score` the secondary sort would be used and that
was using the `record_score` at the highest priority.
2018-09-04 11:38:24 +01:00
David Roberts 84eaac79d7
[ML] Minor improvements to categorization Grok pattern creation (#33353)
1. The TOMCAT_DATESTAMP format needs to be checked before
   TIMESTAMP_ISO8601, otherwise TIMESTAMP_ISO8601 will
   match the start of the Tomcat datestamp.
2. Exclude more characters before and after numbers.  For
   example, in 1.2.3 we don't want to match 1.2 as a float.
2018-09-04 09:43:49 +01:00
Alpar Torok 7f7e8fd733
Disable assemble task instead of removing it (#33348) 2018-09-04 07:32:14 +03:00
Benjamin Trent 767d8e0801
[ML] Delete forecast API (#31134) (#33218)
* Delete forecast API (#31134)
2018-09-03 19:06:18 -05:00
Colin Goodheart-Smithe e2c1beb1be
Merge branch 'master' into index-lifecycle 2018-09-03 10:01:16 +01:00
Nhat Nguyen b93507608a Merge branch 'master' into ccr
* master:
  Mute test watcher usage stats output
  [Rollup] Fix FullClusterRestart test
  Adjust soft-deletes version after backport into 6.5
  completely drop `index.shard.check_on_startup: fix` for 7.0 (#33194)
  Fix AwaitsFix issue number
  Mute SmokeTestWatcherWithSecurityIT testsi
  drop `index.shard.check_on_startup: fix` (#32279)
  tracked at
  [DOCS] Moves ml folder from x-pack/docs to docs (#33248)
  [DOCS] Move rollup APIs to docs (#31450)
  [DOCS] Rename X-Pack Commands section (#33005)
  TEST: Disable soft-deletes in ParentChildTestCase
  Fixes SecurityIntegTestCase so it always adds at least one alias (#33296)
  Fix pom for build-tools (#33300)
  Lazy evaluate java9home (#33301)
  SQL: test coverage for JdbcResultSet (#32813)
  Work around to be able to generate eclipse projects (#33295)
  Highlight that index_phrases only works if no slop is used (#33303)
  Different handling for security specific errors in the CLI. Fix for https://github.com/elastic/elasticsearch/issues/33230 (#33255)
  [ML] Refactor delimited file structure detection (#33233)
  SQL: Support multi-index format as table identifier (#33278)
  MINOR: Remove Dead Code from PathTrie (#33280)
  Enable forbiddenapis server java9 (#33245)
2018-08-31 19:03:04 -04:00
Colin Goodheart-Smithe 3eef74d5d5
Merge branch 'master' into index-lifecycle 2018-08-31 14:45:22 +01:00
David Roberts 7345878d33
[ML] Refactor delimited file structure detection (#33233)
1. Use the term "delimited" rather than "separated values"
2. Use a single factory class with arguments to specify the
   delimiter and identification constraints

This change makes it easier to add support for other
delimiter characters.
2018-08-31 08:48:45 +01:00