The /_sql endpoint now returns the results in the text format by default. Structured formats are also supported using the format parameter or accept header similar to _cat endpoints.
Original commit: elastic/x-pack-elasticsearch@4353793b83
* [DOCS] Move migrate tool reference
* [DOCS] Fixed link to migration tool
* [DOCS] Small edits to the migrate tool parameters
* [DOCS] Fixed migrate tool example
Original commit: elastic/x-pack-elasticsearch@0ff40ebdcc
* [DOCS] Enabled code snippet testing for start datafeed API
* [DOCS] Added datafeed creation to build.gradle
Original commit: elastic/x-pack-elasticsearch@1acb452cf0
The chained input in watcher is a useful feature to
call several endpoints before execution a condition.
However it was pretty hard to modify data from a previous
input in order to be able to execute it in another input.
This commit adds a another input, called a `transform` input,
which allows you to do a transform as another input in a chained
input.
See this example
```
"input" : {
"chain" : {
"inputs" : [ <1>
{
"first" : {
"simple" : { "path" : "/_search" }
}
},
{
"second" : {
"transform" : {
"script" : "return [ 'path' : 'ctx.payload.first.path' + '/' ]"
}
}
},
{
"third" : {
"http" : {
"request" : {
"host" : "localhost",
"port" : 9200,
"path" : "{{ctx.payload.second.path}}" <2>
}
}
}
}
]
}
}
```
This allows for far more flexibility before executing the next input in a chained
one.
Original commit: elastic/x-pack-elasticsearch@3af9ba6e9b
* [DOCS] Added ML forecast API
* [DOCS] Added forecast API to build.gradle
* [DOCS] Added forecast API example
* [DOCS] Fixed forecast API intro
* [DOCS] Addressed feedback on forecast API
* [DOCS] Added duration to forecast API
* [DOCS] Removed end time from forecast API
* [DOCS] Fixed gradle errors for forecast API
Original commit: elastic/x-pack-elasticsearch@db79e3d5bb
The 6.x and 6.0 versions of the documentation show
```yml
transport.profiles.client.xpack.security.ssl.client_authentication: no
```
Which results in
```
2017-11-22T11:13:33,225][ERROR][org.elasticsearch.bootstrap.Bootstrap] Exception
java.lang.IllegalStateException: failed to load plugin class [org.elasticsearch.xpack.XPackPlugin]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:452) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.plugins.PluginsService.loadBundles(PluginsService.java:392) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.plugins.PluginsService.<init>(PluginsService.java:142) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.node.Node.<init>(Node.java:302) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.node.Node.<init>(Node.java:245) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap$5.<init>(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.setup(Bootstrap.java:212) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Bootstrap.init(Bootstrap.java:322) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.init(Elasticsearch.java:130) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.execute(Elasticsearch.java:121) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.EnvironmentAwareCommand.execute(EnvironmentAwareCommand.java:69) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.mainWithoutErrorHandling(Command.java:134) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.cli.Command.main(Command.java:90) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:92) [elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.bootstrap.Elasticsearch.main(Elasticsearch.java:85) [elasticsearch-6.0.0.jar:6.0.0]
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:443) ~[elasticsearch-6.0.0.jar:6.0.0]
... 14 more
Caused by: java.lang.IllegalArgumentException: could not resolve ssl client auth. unknown value [no]
at org.elasticsearch.xpack.ssl.SSLClientAuth.parse(SSLClientAuth.java:78) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLConfigurationSettings.lambda$null$27(SSLConfigurationSettings.java:183) ~[?:?]
at org.elasticsearch.common.settings.Setting.get(Setting.java:352) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.common.settings.Setting.get(Setting.java:346) ~[elasticsearch-6.0.0.jar:6.0.0]
at org.elasticsearch.xpack.ssl.SSLConfiguration.<init>(SSLConfiguration.java:80) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.lambda$loadSSLConfigurations$1(SSLService.java:462) ~[?:?]
at java.util.ArrayList.forEach(ArrayList.java:1249) ~[?:1.8.0_144]
at org.elasticsearch.xpack.ssl.SSLService.loadSSLConfigurations(SSLService.java:461) ~[?:?]
at org.elasticsearch.xpack.ssl.SSLService.<init>(SSLService.java:87) ~[?:?]
at org.elasticsearch.xpack.XPackPlugin.<init>(XPackPlugin.java:237) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) ~[?:?]
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62) ~[?:?]
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) ~[?:?]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423) ~[?:1.8.0_144]
at org.elasticsearch.plugins.PluginsService.loadPlugin(PluginsService.java:443) ~[elasticsearch-6.0.0.jar:6.0.0]
... 14 more
```
I think the valid value there is `none`, so
```yml
transport.profiles.client.xpack.security.ssl.client_authentication: none
```
The tests seem to confirm that https://github.com/elastic/x-pack-elasticsearch/blob/elastic/x-pack-elasticsearch@4860e92d906e046a23aa07b39ee6ef637f011dc1/plugin/src/test/java/org/elasticsearch/xpack/ssl/SSLServiceTests.java#L269
Original commit: elastic/x-pack-elasticsearch@a35b3ac8c9
Adds the option to specify an elasticsearch filter in addition to the SQL query by introducing a filter parameter in the REST query which would create a boolean filter if the SQL query generates an elasticsearch query or a constant score query if SQL if the SQL query doesn't generates an elasticsearch query. Usage:
{
"query": "SELECT * FROM index",
"filter" : { "term" : { "tag" : "tech" } }
}
relates elastic/x-pack-elasticsearch#2895
Original commit: elastic/x-pack-elasticsearch@9a73813c7f
This commit adds the manage_index_templates permission to the kibana_system role that is used by
the kibana system user. This is needed due to an upcoming feature in kibana where a index template
will be used to create the saved objects index.
relates elastic/x-pack-elasticsearch#2937
Original commit: elastic/x-pack-elasticsearch@85a67c73aa
This change modifies the way ML jobs are assigned to nodes to primarily
base the decision on the estimated memory footprint of the jobs. The
memory footprint comes from the model size stats if the job has been
running long enough, otherwise from the model memory limit. In addition,
an allowance for the program code and stack is added.
If insufficient information is available to base the allocation decision on
memory requirements then the decision falls back to using simple job
counts per node.
relates elastic/x-pack-elasticsearch#546
Original commit: elastic/x-pack-elasticsearch@b276aedf2f
1. decouple JdbcDriver from other classes to not trigger static
initialization (this happens through JDBC service discovery)
2. reduce visibility of JdbcDriver#close so only on jar unloading it
gets triggered
3. mark 3 methods introduced in Jdbc 4.1 as unsupported (their semantics
are somewhat weird)
4. Move versioning info in one class
5. Hook Version class in both JDBC entry points to perform cp sanity
checks
6. Remove JdbcDataSource#close (DebugLog are closed when the Driver gets
unloaded by the DriverManager) as there can be multiple instances of DS
but only one for Driver known by the DriverManager
Replace Strings with constants
Properly set TZ in security tests as well
JdbcDataSource is more defensive with its internal properties
JdbcConfiguration password parameter is aligned with JDBC DriverManager
Remove usage of JdbcConnection API
Removed JdbcConnection#setTimeZone - this encourages folks to use our
private API which would tie us down.
It is somewhat limiting for folks but it has less downsides overall and
does not trip debugging (which adds a proxy unaware of this method).
Update docs
Add JdbcDataSource into the Jdbc suite
Original commit: elastic/x-pack-elasticsearch@c713665d53
and add warning for Windows users not using
PowerShell (e.g. `cmd.exe`) to remove the `\` character and join
lines.
Also fix trailing whitespace character in link back to `docker.asciidoc`.
Relates elastic/x-pack-elasticsearch#2999
Original commit: elastic/x-pack-elasticsearch@fe1c5dbc11
* [DOCS] Add docker TLS configuration info
* [DOCS] Updated layout of TLS docker page
* [DOCS] Clean up docker TLS pages
* [DOCS] Changed nesting of TLS docker info
* [DOCS] More small updates to TLS docker page
Original commit: elastic/x-pack-elasticsearch@2b0504632a
The keywords inside SqlBase are now sorted alphabetically - much easier
to read and update the docs
Original commit: elastic/x-pack-elasticsearch@5aa89c5950
The action condition feature was carefully hidden in an example.
This commit creates an own paragraph to highlight this feature better.
Original commit: elastic/x-pack-elasticsearch@006318787b
* [DOCS] Add SSL info to setup-passwords
* [DOCS] Addressed feedback in setup-passwords
* [DOCS] Added link to setup-passwords troubleshooting page
Original commit: elastic/x-pack-elasticsearch@2bf820c303
Adds docs for the REST API, translate API, the CLI, and JDBC.
Next we need to add more example queries and documentation for our
extensions.
Original commit: elastic/x-pack-elasticsearch@ed6d1360d2
* [DOCS] Enabled code snippet testing for put datafeed API
* [DOCS] Addressed gradle errors in put datafeed API
* [DOCS] Added job creation test to build.gradle
Original commit: elastic/x-pack-elasticsearch@3548d920c7
For the purpose of getting this API consumed by our UI, returning
overall buckets that match the job's largest `bucket_span` can
result in too much data. The UI only ever displays a few buckets
in the swimlane. Their span depends on the time range selected and
the screen resolution, but it will only ever be a relatively
low number.
This PR adds the ability to aggregate overall buckets in a user
specified `bucket_span`. That `bucket_span` may be equal or
greater to the largest job's `bucket_span`. The `overall_score`
of the result overall buckets is the max score of the
corresponding overall buckets with a span equal to the job's
largest `bucket_span`.
The implementation is now chunking the bucket requests
as otherwise the aggregation would fail when too many buckets
are matching.
Original commit: elastic/x-pack-elasticsearch@981f7a40e5
* [DOCS] Added transport client info for X-Pack
* [DOCS] Relocated X-Pack java client info
* [DOCS] Added transport client deprecation info
Original commit: elastic/x-pack-elasticsearch@416aab1d76
Adds the GET overall_buckets API.
The REST end point is: GET
/_xpack/ml/anomaly_detectors/job_id/results/overall_buckets
The API returns overall bucket results. An overall bucket
is a summarized bucket result over multiple jobs.
It has the `bucket_span` of the longest job's `bucket_span`.
It also has an `overall_score` that is the `top_n` average of the
max anomaly scores per job.
relates elastic/x-pack-elasticsearch#2693
Original commit: elastic/x-pack-elasticsearch@ba6061482d
The execution state is kind of a global indicator if a watch has been
running successfully and is used by the watcher UI.
However this field is only stored in the watch history but not part of
the watch status, thus it is not available everywhere. In order to
simplify the watcher UI this commit also adds the field to the
watch status which is stored together with the watch.
It is stored under the `status.execution_state` field as `status.state`
is already taken. This is also reflects with the name of the java class.
The WatchStatus class does not contain serialization checks, as this is
intended to be backported to 6.x, where those checks will be added.
Once the backport is done, the old execution state field can be fully
deleted from the master branch in another commit (syncing with Kibana
folks required).
relates elastic/x-pack-elasticsearch#2385
* fix doc tests
Original commit: elastic/x-pack-elasticsearch@26e8f99571
The path of a JIRA endpoint used to be fixed. This commit allows the
path to be dynamic, so that users can deploy their JIRA instance under
an arbitrary prefix.
Original commit: elastic/x-pack-elasticsearch@7702505114
* [DOCS] Format Watcher APIs
* [DOCS] Removed master_timeout from Watcher APIs
* [DOCS] Added authority info to watcher APIs
Original commit: elastic/x-pack-elasticsearch@1e6de3b036
This change removes `xpack.security.authc.token.passphrase` entirely since from
6.0 onwards we use randomly generated keys by the master there is no need for
this setting anymore. This setting will be deprecated from 6.0 onwards.
Original commit: elastic/x-pack-elasticsearch@37ba90359e
As there are no master node operations anymore.
* TransportActions are regular Actions now
* Watcher requests are now ActionRequests, no MasterNodeRequests anymore
* REST spec does not contain master node timeout parameters anymore
* WatcherLifeCycleService does not have a check anymore if watcher is able to run distributed, this will be a given in 7.0
* Some serialization BWC checks against version 5 have been removed
Original commit: elastic/x-pack-elasticsearch@4607dd538c
Right now we'd like to be able to build the SQL docs without
modifying core even if they aren't quite how we're going to keep
them in the end. This removes the need for a change to core and
leaves a note that we'll have to make undo this commit before
release.
Original commit: elastic/x-pack-elasticsearch@9ca6a0e467
`authc.token.enabled` is true unless `http.ssl.enabled` is `false` and `http.enabled` is `true`.
* TokenService default enabled if HTTP_ENABLED == false
* Fixed tests that need TokenService explicitly enabled
* [DOC] Default value for `xpack.security.authc.token.enabled`
Original commit: elastic/x-pack-elasticsearch@bd154d16eb
* [DOCS] Describe severity threshold and interval in anomaly table
* [DOCS] Describe aggregation in anomaly table
* [DOCS] Fixed capitalization in ML getting started
Original commit: elastic/x-pack-elasticsearch@d4224c9fe8
* [DOCS] Add job groups to ML create/update job APIs
* [DOCS] Fix ML update job API example
* [DOCS] Address feedback for ML create/update job APIs
Original commit: elastic/x-pack-elasticsearch@0e7bb47342
* [DOCS] Add custom URL instructions
* [DOCS] Update custom URL examples
* [DOCS] Add example for custom URL
* [DOCS] Add ML category terms custom URL example
* [DOCS] Add tip for pop-up ML custom URLs
* [DOCS] Address feedback about ML custom URLs
* [DOCS] Clarify impact of interval on custom URL
Original commit: elastic/x-pack-elasticsearch@eee192226f
This change makes 2 improvements to the max_running_jobs setting:
1. Namespaces it by adding the xpack.ml. prefix
2. Renames "running" to "open", because the "running" terminology
is not used elsewhere
The old max_running_jobs setting is used as a fallback if the new
xpack.ml.max_open_jobs setting is not specified. max_running_jobs
is deprecated and (to ease backporting in the short term) will be
removed from 7.0 in a different PR closer to release of 7.0.
Relates elastic/x-pack-elasticsearch#2185
Original commit: elastic/x-pack-elasticsearch@18c539f9bb
* [DOCS] Add user setup to X-Pack install info
* [DOCS] Add TSL steps to X-Pack install
* [DOCS] Clarify SSL settings in Xpack install
Original commit: elastic/x-pack-elasticsearch@eee37729ff
* [DOCS] Update APIs for multiple jobs or datafeeds
* [DOCS] Fix syntax diagrams for ML stop/close APIs
* [DOCS] Removed TBD authorization for ML APIs
Original commit: elastic/x-pack-elasticsearch@1a9137a5a7
This commit adds detection of support for AES 256 bit ciphers and enables their use when the JVM
supports them. For OpenJDK, this is often the case without any changes but for the Oracle JVM, the
unlimited policy file needs to be installed. In order to simplify the work a user would need to do
we can detect this support and automatically enable the AES 256 bit versions of the ciphers we
already enable.
Original commit: elastic/x-pack-elasticsearch@5f23b18a1e
The deprecated handlers should have been removed earlier, but are now
going to to away finally.
Also the watcher restart action has been removed, mainly because users
should not blindly restart, but always make sure, that watcher is
stopped correctly before restarting. This had been removed from the
transport action previously.
Original commit: elastic/x-pack-elasticsearch@78a5ec3c05
- Fix typo `trustsore` -> `truststore` in several places
- Clarify that enabling TLS requires full restart
Original commit: elastic/x-pack-elasticsearch@0f430a1bea
Add an option to the ssl certificate generation tool (certgen) that generates PKCSelastic/x-pack-elasticsearch#12 (.p12) files in addition to the certificate (.crt) and key (.key) files.
A PKCSelastic/x-pack-elasticsearch#12 store is a container format for storing multiple crypto objects in a single file, which means we can put the cert and key into the same file.
These format is particularly useful for .NET environments, where .NET Core requires a single into file for PKI authentication.
Also adds documentation for all the command-line options in certgen.
Original commit: elastic/x-pack-elasticsearch@d10f88f12d
The active directory user principal name format typically takes the form user@domain, which is what
the current implementation expects. However, active directory also allows the definition of other
suffixes that are not actual domains. A user can still authenticate using this user principal name
but the behavior of our realm would cause it to fail as it parsed the suffix as a domain and used it
as the search base for the user. Instead, we should use the default user search base and only look
for entries that have this exact user principal name. In a scenario where a realm is configured for
multiple domains in the same forest, the search base should be the base for the entire forest.
relates elastic/x-pack-elasticsearch#1744
Original commit: elastic/x-pack-elasticsearch@de00c4817e
This commit adds support for a bind user when using the active directory realm. The addition of a
bind user also enables support for the user lookup mechanism, which is necessary to support the run
as functionality that we provide.
relates elastic/x-pack-elasticsearch#179
Original commit: elastic/x-pack-elasticsearch@40b07b3422
This commit adds a setting to allow changing the user search filter. Previously the filter was a
simple equality filter that mapped a given attribute to the value of the username. The default
behavior remains the same with this change but provides additional flexibility to users to who may
need more advanced LDAP searches. The user attribute setting has been deprecated due to the overlap
with the new filter setting.
relates elastic/x-pack-elasticsearch#1861
Original commit: elastic/x-pack-elasticsearch@e9d797e81c
* [DOCS] Update model_memory_limit
* [DOCS] Clarify minimum model_memory_limit value
* [DOCS] More updates to model_memory_limit
* [DOCS] Address feedback in jobresource.asciidoc
Original commit: elastic/x-pack-elasticsearch@3c62719037
- Document refresh interval for role mapping files
- Fix obsolete shield reference in transport profile example
- Clarify that AD & PKI don't support run_as
- Fix logstash conf examples
- Clarify interaction of SSL settings and PKI realm settings
- Document PKI DN format, and recommend use of pki_dn metadata
- Provide more details about action.auto_create_index during setup
Original commit: elastic/x-pack-elasticsearch@49ddb12a7e
This is step 2 of elastic/x-pack-elasticsearch#1604
This change stores `model_memory_limit` as a string with `mb` unit.
I considered using the `toString` method of `ByteSizeValue` but it
can lead to accuracy loss. Adding the fixed `mb` unit maintains
the accuracy, while making clear what unit the value is in.
Original commit: elastic/x-pack-elasticsearch@4dc48f0ce8
There are multiple references to this section in different areas of the
documentation. This commit brings back this section to fix the build.
A more extensive PR updating the documentation for "no default
password" work will follow up.
Original commit: elastic/x-pack-elasticsearch@0378e78c8a
This section has been removed from setting-up-authentication. This
commit removes a reference to this section that no longer exists.
Original commit: elastic/x-pack-elasticsearch@43aa0077f9
This commit removes the system key from master and changes watcher to use a secure setting instead
for the encryption key.
Original commit: elastic/x-pack-elasticsearch@5ac95c60ef
This is related to elastic/x-pack-elasticsearch#1217. This PR removes the default password of
"changeme" from the reserved users.
This PR adds special behavior for authenticating the reserved users. No
ReservedRealm user can be authenticated until its password is set. The
one exception to this is the elastic user. The elastic user can be
authenticated with an empty password if the action is a rest request
originating from localhost. In this scenario where an elastic user is
authenticated with a default password, it will have metadata indicating
that it is in setup mode. An elastic user in setup mode is only
authorized to execute a change password request.
Original commit: elastic/x-pack-elasticsearch@e1e101a237
* [DOCS] Move security APIs to Elasticsearch Ref
* [DOCS] Update links to security APIs
* [DOCS] Fix link to security APIs
Original commit: elastic/x-pack-elasticsearch@d7a9d3f1ab
* [DOCS] Model plot updates
Add to job create.
Remove terms from job resource.
* [DOCS] Describing terms as experimental
Original commit: elastic/x-pack-elasticsearch@815fa0ec37
* [DOCS] Getting started with multi-metric jobs
* [DOCS] More work on ML getting started with multi-metric jobs
* [DOCS] Add ML getting started with multi-metric jobs screenshot
* [DOCS] Add ML getting started information about influencers
* [DOCS] Getting started with multi-metric jobs
* [DOCS] Fix ML getting started build error
* [DOCS] Add ML getting started multi-metric snapshots
* [DOCS] Add screenshots and next steps to ML Getting Started tutorial
* [DOCS] Clarified anomaly scores in ML Getting Started pages
* [DOCS] Addressed ML getting started feedback
* [DOCS] Fix ML getting started links
Original commit: elastic/x-pack-elasticsearch@a7e80cfabf
Nik Everett
Lisa Cawley
Dimitrios Athanasiou
Igor Motov
Alexander Reelsen
Russ Cam
Igor Kupczyński
Jay Modi
javanna
David Roberts
Costin Leau
Dimitrios Liappis
Costin Leau
David Kyle
Martijn van Groningen
Tanguy Leroux
jaymode
Simon Willnauer
Deb Adair
Jim Ferenczi
Albert Zaharovits
Jason Tedor
Tim Vernum
Tal Levy
Tim Brooks
Drew Raines
Colin Goodheart-Smithe
Clinton Gormley
Sophie Chang