Adds the option to specify an elasticsearch filter in addition to the SQL query by introducing a filter parameter in the REST query which would create a boolean filter if the SQL query generates an elasticsearch query or a constant score query if SQL if the SQL query doesn't generates an elasticsearch query. Usage:
{
"query": "SELECT * FROM index",
"filter" : { "term" : { "tag" : "tech" } }
}
relates elastic/x-pack-elasticsearch#2895
Original commit: elastic/x-pack-elasticsearch@9a73813c7f
This commit adds the manage_index_templates permission to the kibana_system role that is used by
the kibana system user. This is needed due to an upcoming feature in kibana where a index template
will be used to create the saved objects index.
relates elastic/x-pack-elasticsearch#2937
Original commit: elastic/x-pack-elasticsearch@85a67c73aa
This change modifies the way ML jobs are assigned to nodes to primarily
base the decision on the estimated memory footprint of the jobs. The
memory footprint comes from the model size stats if the job has been
running long enough, otherwise from the model memory limit. In addition,
an allowance for the program code and stack is added.
If insufficient information is available to base the allocation decision on
memory requirements then the decision falls back to using simple job
counts per node.
relates elastic/x-pack-elasticsearch#546
Original commit: elastic/x-pack-elasticsearch@b276aedf2f
1. decouple JdbcDriver from other classes to not trigger static
initialization (this happens through JDBC service discovery)
2. reduce visibility of JdbcDriver#close so only on jar unloading it
gets triggered
3. mark 3 methods introduced in Jdbc 4.1 as unsupported (their semantics
are somewhat weird)
4. Move versioning info in one class
5. Hook Version class in both JDBC entry points to perform cp sanity
checks
6. Remove JdbcDataSource#close (DebugLog are closed when the Driver gets
unloaded by the DriverManager) as there can be multiple instances of DS
but only one for Driver known by the DriverManager
Replace Strings with constants
Properly set TZ in security tests as well
JdbcDataSource is more defensive with its internal properties
JdbcConfiguration password parameter is aligned with JDBC DriverManager
Remove usage of JdbcConnection API
Removed JdbcConnection#setTimeZone - this encourages folks to use our
private API which would tie us down.
It is somewhat limiting for folks but it has less downsides overall and
does not trip debugging (which adds a proxy unaware of this method).
Update docs
Add JdbcDataSource into the Jdbc suite
Original commit: elastic/x-pack-elasticsearch@c713665d53
and add warning for Windows users not using
PowerShell (e.g. `cmd.exe`) to remove the `\` character and join
lines.
Also fix trailing whitespace character in link back to `docker.asciidoc`.
Relates elastic/x-pack-elasticsearch#2999
Original commit: elastic/x-pack-elasticsearch@fe1c5dbc11
* [DOCS] Add docker TLS configuration info
* [DOCS] Updated layout of TLS docker page
* [DOCS] Clean up docker TLS pages
* [DOCS] Changed nesting of TLS docker info
* [DOCS] More small updates to TLS docker page
Original commit: elastic/x-pack-elasticsearch@2b0504632a
The keywords inside SqlBase are now sorted alphabetically - much easier
to read and update the docs
Original commit: elastic/x-pack-elasticsearch@5aa89c5950
The action condition feature was carefully hidden in an example.
This commit creates an own paragraph to highlight this feature better.
Original commit: elastic/x-pack-elasticsearch@006318787b
* [DOCS] Add SSL info to setup-passwords
* [DOCS] Addressed feedback in setup-passwords
* [DOCS] Added link to setup-passwords troubleshooting page
Original commit: elastic/x-pack-elasticsearch@2bf820c303
Adds docs for the REST API, translate API, the CLI, and JDBC.
Next we need to add more example queries and documentation for our
extensions.
Original commit: elastic/x-pack-elasticsearch@ed6d1360d2
* [DOCS] Enabled code snippet testing for put datafeed API
* [DOCS] Addressed gradle errors in put datafeed API
* [DOCS] Added job creation test to build.gradle
Original commit: elastic/x-pack-elasticsearch@3548d920c7
For the purpose of getting this API consumed by our UI, returning
overall buckets that match the job's largest `bucket_span` can
result in too much data. The UI only ever displays a few buckets
in the swimlane. Their span depends on the time range selected and
the screen resolution, but it will only ever be a relatively
low number.
This PR adds the ability to aggregate overall buckets in a user
specified `bucket_span`. That `bucket_span` may be equal or
greater to the largest job's `bucket_span`. The `overall_score`
of the result overall buckets is the max score of the
corresponding overall buckets with a span equal to the job's
largest `bucket_span`.
The implementation is now chunking the bucket requests
as otherwise the aggregation would fail when too many buckets
are matching.
Original commit: elastic/x-pack-elasticsearch@981f7a40e5
* [DOCS] Added transport client info for X-Pack
* [DOCS] Relocated X-Pack java client info
* [DOCS] Added transport client deprecation info
Original commit: elastic/x-pack-elasticsearch@416aab1d76
Adds the GET overall_buckets API.
The REST end point is: GET
/_xpack/ml/anomaly_detectors/job_id/results/overall_buckets
The API returns overall bucket results. An overall bucket
is a summarized bucket result over multiple jobs.
It has the `bucket_span` of the longest job's `bucket_span`.
It also has an `overall_score` that is the `top_n` average of the
max anomaly scores per job.
relates elastic/x-pack-elasticsearch#2693
Original commit: elastic/x-pack-elasticsearch@ba6061482d
The execution state is kind of a global indicator if a watch has been
running successfully and is used by the watcher UI.
However this field is only stored in the watch history but not part of
the watch status, thus it is not available everywhere. In order to
simplify the watcher UI this commit also adds the field to the
watch status which is stored together with the watch.
It is stored under the `status.execution_state` field as `status.state`
is already taken. This is also reflects with the name of the java class.
The WatchStatus class does not contain serialization checks, as this is
intended to be backported to 6.x, where those checks will be added.
Once the backport is done, the old execution state field can be fully
deleted from the master branch in another commit (syncing with Kibana
folks required).
relates elastic/x-pack-elasticsearch#2385
* fix doc tests
Original commit: elastic/x-pack-elasticsearch@26e8f99571
The path of a JIRA endpoint used to be fixed. This commit allows the
path to be dynamic, so that users can deploy their JIRA instance under
an arbitrary prefix.
Original commit: elastic/x-pack-elasticsearch@7702505114
* [DOCS] Format Watcher APIs
* [DOCS] Removed master_timeout from Watcher APIs
* [DOCS] Added authority info to watcher APIs
Original commit: elastic/x-pack-elasticsearch@1e6de3b036
This change removes `xpack.security.authc.token.passphrase` entirely since from
6.0 onwards we use randomly generated keys by the master there is no need for
this setting anymore. This setting will be deprecated from 6.0 onwards.
Original commit: elastic/x-pack-elasticsearch@37ba90359e
As there are no master node operations anymore.
* TransportActions are regular Actions now
* Watcher requests are now ActionRequests, no MasterNodeRequests anymore
* REST spec does not contain master node timeout parameters anymore
* WatcherLifeCycleService does not have a check anymore if watcher is able to run distributed, this will be a given in 7.0
* Some serialization BWC checks against version 5 have been removed
Original commit: elastic/x-pack-elasticsearch@4607dd538c
Right now we'd like to be able to build the SQL docs without
modifying core even if they aren't quite how we're going to keep
them in the end. This removes the need for a change to core and
leaves a note that we'll have to make undo this commit before
release.
Original commit: elastic/x-pack-elasticsearch@9ca6a0e467
`authc.token.enabled` is true unless `http.ssl.enabled` is `false` and `http.enabled` is `true`.
* TokenService default enabled if HTTP_ENABLED == false
* Fixed tests that need TokenService explicitly enabled
* [DOC] Default value for `xpack.security.authc.token.enabled`
Original commit: elastic/x-pack-elasticsearch@bd154d16eb
* [DOCS] Describe severity threshold and interval in anomaly table
* [DOCS] Describe aggregation in anomaly table
* [DOCS] Fixed capitalization in ML getting started
Original commit: elastic/x-pack-elasticsearch@d4224c9fe8
* [DOCS] Add job groups to ML create/update job APIs
* [DOCS] Fix ML update job API example
* [DOCS] Address feedback for ML create/update job APIs
Original commit: elastic/x-pack-elasticsearch@0e7bb47342
* [DOCS] Add custom URL instructions
* [DOCS] Update custom URL examples
* [DOCS] Add example for custom URL
* [DOCS] Add ML category terms custom URL example
* [DOCS] Add tip for pop-up ML custom URLs
* [DOCS] Address feedback about ML custom URLs
* [DOCS] Clarify impact of interval on custom URL
Original commit: elastic/x-pack-elasticsearch@eee192226f
This change makes 2 improvements to the max_running_jobs setting:
1. Namespaces it by adding the xpack.ml. prefix
2. Renames "running" to "open", because the "running" terminology
is not used elsewhere
The old max_running_jobs setting is used as a fallback if the new
xpack.ml.max_open_jobs setting is not specified. max_running_jobs
is deprecated and (to ease backporting in the short term) will be
removed from 7.0 in a different PR closer to release of 7.0.
Relates elastic/x-pack-elasticsearch#2185
Original commit: elastic/x-pack-elasticsearch@18c539f9bb
* [DOCS] Add user setup to X-Pack install info
* [DOCS] Add TSL steps to X-Pack install
* [DOCS] Clarify SSL settings in Xpack install
Original commit: elastic/x-pack-elasticsearch@eee37729ff
* [DOCS] Update APIs for multiple jobs or datafeeds
* [DOCS] Fix syntax diagrams for ML stop/close APIs
* [DOCS] Removed TBD authorization for ML APIs
Original commit: elastic/x-pack-elasticsearch@1a9137a5a7
In order to not restart watcher on every test, this checks
if this is a watcher test and only restarts watcher in that case.
In addition this also checks if watcher is not marked as started.
as otherwise restarting does not make sense. Lastly, this waits until
watcher is marked as started before proceeding.
Original commit: elastic/x-pack-elasticsearch@a8d72f3ebb
This commit adds detection of support for AES 256 bit ciphers and enables their use when the JVM
supports them. For OpenJDK, this is often the case without any changes but for the Oracle JVM, the
unlimited policy file needs to be installed. In order to simplify the work a user would need to do
we can detect this support and automatically enable the AES 256 bit versions of the ciphers we
already enable.
Original commit: elastic/x-pack-elasticsearch@5f23b18a1e
The deprecated handlers should have been removed earlier, but are now
going to to away finally.
Also the watcher restart action has been removed, mainly because users
should not blindly restart, but always make sure, that watcher is
stopped correctly before restarting. This had been removed from the
transport action previously.
Original commit: elastic/x-pack-elasticsearch@78a5ec3c05
- Fix typo `trustsore` -> `truststore` in several places
- Clarify that enabling TLS requires full restart
Original commit: elastic/x-pack-elasticsearch@0f430a1bea
Add an option to the ssl certificate generation tool (certgen) that generates PKCSelastic/x-pack-elasticsearch#12 (.p12) files in addition to the certificate (.crt) and key (.key) files.
A PKCSelastic/x-pack-elasticsearch#12 store is a container format for storing multiple crypto objects in a single file, which means we can put the cert and key into the same file.
These format is particularly useful for .NET environments, where .NET Core requires a single into file for PKI authentication.
Also adds documentation for all the command-line options in certgen.
Original commit: elastic/x-pack-elasticsearch@d10f88f12d
The active directory user principal name format typically takes the form user@domain, which is what
the current implementation expects. However, active directory also allows the definition of other
suffixes that are not actual domains. A user can still authenticate using this user principal name
but the behavior of our realm would cause it to fail as it parsed the suffix as a domain and used it
as the search base for the user. Instead, we should use the default user search base and only look
for entries that have this exact user principal name. In a scenario where a realm is configured for
multiple domains in the same forest, the search base should be the base for the entire forest.
relates elastic/x-pack-elasticsearch#1744
Original commit: elastic/x-pack-elasticsearch@de00c4817e
This commit adds support for a bind user when using the active directory realm. The addition of a
bind user also enables support for the user lookup mechanism, which is necessary to support the run
as functionality that we provide.
relates elastic/x-pack-elasticsearch#179
Original commit: elastic/x-pack-elasticsearch@40b07b3422
This commit adds a setting to allow changing the user search filter. Previously the filter was a
simple equality filter that mapped a given attribute to the value of the username. The default
behavior remains the same with this change but provides additional flexibility to users to who may
need more advanced LDAP searches. The user attribute setting has been deprecated due to the overlap
with the new filter setting.
relates elastic/x-pack-elasticsearch#1861
Original commit: elastic/x-pack-elasticsearch@e9d797e81c
* [DOCS] Update model_memory_limit
* [DOCS] Clarify minimum model_memory_limit value
* [DOCS] More updates to model_memory_limit
* [DOCS] Address feedback in jobresource.asciidoc
Original commit: elastic/x-pack-elasticsearch@3c62719037
- Document refresh interval for role mapping files
- Fix obsolete shield reference in transport profile example
- Clarify that AD & PKI don't support run_as
- Fix logstash conf examples
- Clarify interaction of SSL settings and PKI realm settings
- Document PKI DN format, and recommend use of pki_dn metadata
- Provide more details about action.auto_create_index during setup
Original commit: elastic/x-pack-elasticsearch@49ddb12a7e
This is step 2 of elastic/x-pack-elasticsearch#1604
This change stores `model_memory_limit` as a string with `mb` unit.
I considered using the `toString` method of `ByteSizeValue` but it
can lead to accuracy loss. Adding the fixed `mb` unit maintains
the accuracy, while making clear what unit the value is in.
Original commit: elastic/x-pack-elasticsearch@4dc48f0ce8
There are multiple references to this section in different areas of the
documentation. This commit brings back this section to fix the build.
A more extensive PR updating the documentation for "no default
password" work will follow up.
Original commit: elastic/x-pack-elasticsearch@0378e78c8a
This section has been removed from setting-up-authentication. This
commit removes a reference to this section that no longer exists.
Original commit: elastic/x-pack-elasticsearch@43aa0077f9
This commit removes the system key from master and changes watcher to use a secure setting instead
for the encryption key.
Original commit: elastic/x-pack-elasticsearch@5ac95c60ef
This is related to elastic/x-pack-elasticsearch#1217. This PR removes the default password of
"changeme" from the reserved users.
This PR adds special behavior for authenticating the reserved users. No
ReservedRealm user can be authenticated until its password is set. The
one exception to this is the elastic user. The elastic user can be
authenticated with an empty password if the action is a rest request
originating from localhost. In this scenario where an elastic user is
authenticated with a default password, it will have metadata indicating
that it is in setup mode. An elastic user in setup mode is only
authorized to execute a change password request.
Original commit: elastic/x-pack-elasticsearch@e1e101a237
* [DOCS] Move security APIs to Elasticsearch Ref
* [DOCS] Update links to security APIs
* [DOCS] Fix link to security APIs
Original commit: elastic/x-pack-elasticsearch@d7a9d3f1ab
* [DOCS] Model plot updates
Add to job create.
Remove terms from job resource.
* [DOCS] Describing terms as experimental
Original commit: elastic/x-pack-elasticsearch@815fa0ec37
* [DOCS] Getting started with multi-metric jobs
* [DOCS] More work on ML getting started with multi-metric jobs
* [DOCS] Add ML getting started with multi-metric jobs screenshot
* [DOCS] Add ML getting started information about influencers
* [DOCS] Getting started with multi-metric jobs
* [DOCS] Fix ML getting started build error
* [DOCS] Add ML getting started multi-metric snapshots
* [DOCS] Add screenshots and next steps to ML Getting Started tutorial
* [DOCS] Clarified anomaly scores in ML Getting Started pages
* [DOCS] Addressed ML getting started feedback
* [DOCS] Fix ML getting started links
Original commit: elastic/x-pack-elasticsearch@a7e80cfabf
* [DOCS] Add ML info about script fields
* [DOCS] Add links to ML script fields page
* [DOCS] Add ML API examples to transforms.asciidoc
* [DOCS] Addressed feedback in ML script field examples
* [DOCS] Add preview to ML script fields example
* [DOCS] Expanded code snippets in ML transform examples
* [DOCS] Add output for ML scripted fields example
* [DOCS] Add output for more ML scripted field examples
* [DOCS] Add output for final ML scripted field examples
* [DOC] Add Kibana details for ML script fields
* [DOCS] Remove example from ML transforms
Original commit: elastic/x-pack-elasticsearch@51057b029f
* [DOCS] Add configuration information for population analysis
* [DOCS] Add ML population analysis examples
* [DOCS] Address feedback for population analysis
* [DOCS] More feedback on population analysis
Original commit: elastic/x-pack-elasticsearch@ffa2bfeed9
This changes the validation criteria we use for user and role
names in the file realm, native realm, and the
realm-agnostic code in x-pack security. The new criteria is:
A valid username's length must be at least 1 and no more than 1024
characters. It may not contain leading or trailing whitespace. All
characters in the name must be be alphanumeric (`a-z`, `A-Z`, `0-9`),
printable punctuation or symbols in the https://en.wikipedia.org/wiki/Basic_Latin_(Unicode_block)[Basic Latin (ASCII) block],
or the space character.
Original commit: elastic/x-pack-elasticsearch@f77640f269
Igor Motov
Jay Modi
javanna
David Roberts
Lisa Cawley
David Kyle
Nik Everett
Costin Leau
Dimitrios Liappis
Alexander Reelsen
Costin Leau
Martijn van Groningen
Dimitris Athanasiou
Russ Cam
Tanguy Leroux
jaymode
Simon Willnauer
Deb Adair
Jim Ferenczi
Michael Basnight
Albert Zaharovits
Jason Tedor
Luca Cavanna
Tim Vernum
Tal Levy
Tim Brooks
Drew Raines
Colin Goodheart-Smithe
Clinton Gormley
Sophie Chang
Andy Bristol