Commit Graph

38723 Commits

Author SHA1 Message Date
Jason Tedor 2c3e71f116
Remove the suggest metric from stats APIs (#29635)
This metric previously existed for backwards compatibility reasons
although the suggest stats were folded into search stats. This metric
was deprecated in 6.3.0 and this commit removes them for 7.0.0.
2018-04-24 19:03:48 -04:00
Jason Tedor 25e45a765c
Fix byte size value equals/hash code test (#29643)
This commit fixes two issues with the byte size value equals/hash code
test.

The first problem is due to a test failure when the original instance is
zero bytes and we pick the mutation branch where we preserve the size
but change the unit. The mutation should result in a different byte size
value but changing the unit on zero bytes still leaves us with zero
bytes.

During the course of fixing this test I discovered another problem. When
we need to randomize size, we could randomly select a size that would
lead to an overflow of Long.MAX_VALUE.

This commit fixes both of these issues.
2018-04-24 19:01:27 -04:00
Jason Tedor 286f4ce8ba
Upgrade to Gradle 4.7 (#29644)
This commit gives us Gradle 4.7 which brings with it official JDK 10
support (which we are already using as our minimum compiler version) and
support for JDK 11 early-access releases (previous versions of Gradle
will not even start on JDK 11).
2018-04-24 19:00:29 -04:00
Jason Tedor bdf241347d
Add 6.4.0 version to master (#29684)
This commit adds the 6.4.0 version constant to the master branch.
2018-04-24 18:21:37 -04:00
Michael Basnight 72f57c8e72
Add comments inadvertently removed during migrate
A few files had their first comment removed even though it did not
contain a license. This re-adds those comments.
2018-04-24 13:41:09 -05:00
David Kyle cfc66a1fd5 [ML] Wait for updates to established memory usage
Tests need to wait for changes to the job's established memory usage to
propagate and an over enthusiastic optimisation meant jobs were updated
from stale state causing recent change to be lost.
2018-04-24 13:46:58 -04:00
Jason Tedor f1aedd9ae8
Add build time checks for package licenses
This commit adds some build time checks that the archive distributions
and package distributions contain the appropriate license and notice
files, and the package distributions contain the appropriate license
metadata.
2018-04-24 12:10:51 -04:00
Ryan Ernst 7abc55f905 Build: Fix License attribute to be written in deb control data
This commit uses the customFields setting of the Deb task in ospackage
to work around the fact it does not know anything about the License
attribute natively.
2018-04-23 17:24:23 -07:00
Ryan Ernst ea2a29530c Build: Use templated copyright file for deb distributions
THe deb distribution has a special copyright file instead of
LICENSE.txt, but the distributions were including the template file
instead of the rendered file (which includes the license name and text).
2018-04-23 09:43:23 -07:00
Jason Tedor 11244cb443
Fix the dashes in license names
For the Debian packages, the license names should contain a dash, but
not for the RPM packages. This commit fixes this.
2018-04-23 11:49:54 -04:00
Jason Tedor 3cadd5c40c Only enable modules to have native controllers
This commit removes the ability for a plugin to have a native controller
as leaves it as only modules can have a native controller.
2018-04-20 15:34:02 -07:00
Jason Tedor ab101976d6 Fix SQL CLI on Windows
This commit fixes the classpath for the SQL CLI tool on Windows. As the
x-pack bin folder was collapsed into the distribution bin folder, the
location of the classpath here needed to no longer contain the old
plugins directory.
2018-04-20 15:34:01 -07:00
Jason Tedor d99d0fa669 Add distribution type to startup scripts
This commit adds the distribution type to the startup scripts so that we
can discern from log output and the main response the type of the
distribution (deb/rpm/tar/zip).
2018-04-20 15:34:01 -07:00
Jason Tedor c7f6b35485 Fix classpath for X-Pack scripts on Windows
With the move of X-Pack to a module, the classpath for the scripts needs
to be adjusted. This was done on Unix, but not for Windows. This commit
addresses Windows.
2018-04-20 15:34:01 -07:00
Ryan Ernst d8636d0fe3 Rename syskeygen
This commit renames syskeygen to elasticsearch-syskeygen
2018-04-20 15:34:01 -07:00
Jason Tedor c7c0e330b8 Rename users
This commit renames users to elasticsearch-users.
2018-04-20 15:34:01 -07:00
Jason Tedor b05c8bf781 Rename migrate
This commit renames migrate to elasticsearch-migrate.
2018-04-20 15:34:00 -07:00
Jason Tedor 2d3fac307a Rename saml-metadata
This commit renames saml-metadata to elasticsearch-saml-metadata.
2018-04-20 15:34:00 -07:00
Jason Tedor 8a8a1dfc2e Rename sql-cli
This commit renames sql-cli to elasticsearch-sql-cli.
2018-04-20 15:34:00 -07:00
Ryan Ernst 6dbb7be067 Rename croneval
This commit renames croneval to elasticsearch-croneval
2018-04-20 15:34:00 -07:00
Jason Tedor 7ba2defdbf Rename setup-passwords
This commit renames setup-passwords to elasticsearch-setup-passwords.
2018-04-20 15:34:00 -07:00
Jason Tedor 03ffd16921 Rename certutil
This commit renames certutil to elasticsearch-certutil.
2018-04-20 15:33:59 -07:00
Jason Tedor a6b4682711 Rename certgen
This commit renames certgen to elasticsearch-certgen.
2018-04-20 15:33:59 -07:00
Ryan Ernst 0d8aa7527e Reorganize license files
This commit moves the apache and elastic license files into a new
root level `licenses` directory and rewrites the top level LICENSE.txt
to clarify the repository has a mix of apache and elastic licensed code.
2018-04-20 15:33:59 -07:00
Ryan Ernst 1fb2d08b31 Packaging: Add license specifications to packages
This commit adds license metadata to rpm and deb packages. Additionally,
it makes the copyright file for deb files follow the machine readable
specification, and sets the correct license text based on the oss vs
default deb packages.
2018-04-20 15:33:59 -07:00
Ryan Ernst 9a45662309 Convert license header check to use the new Elastic License
This commit adapts the license headers check to no longer look for the
ealsticsearch confidential license, but instead to look for the new
Elastic License header.
2018-04-20 15:33:59 -07:00
Jason Tedor 4fe1d64070 Add POM and license generation for X-Pack artifacts
With the switch to X-Pack as a module, we lost production of POMs for
the JARs that we publish, and did not have a license/notice file in the
zip archives nor the exploded module. This commit ensures that we
generate these POMs, and license/notice files.
2018-04-20 15:33:58 -07:00
Jason Tedor 2e869f2017 Handle installing X-Pack
X-Pack can no longer be installed as a plugin. This commit adds special
handling for when a user attempts to install X-Pack. This special
handling informs the user of the oss distribution that they should
download the default distribution and the user of the default
distribution that X-Pack does not require installation as it is included
by default.
2018-04-20 15:33:58 -07:00
Jason Tedor 0750f5d6f3 Add conflicts for default and oss packages
This commit adds conflicts between the default and oss packages so that
a user can not install both via a package manager (as they share paths).
2018-04-20 15:33:58 -07:00
Jason Tedor e64e6d8996 Add distribution flavor to startup scripts
This commit adds the distribution flavor (default versus oss) to the
build process which is passed through the startup scripts to
Elasticsearch. This change will be used to customize the message on
attempting to install/remove x-pack based on the distribution flavor.
2018-04-20 15:33:58 -07:00
Ryan Ernst fab5e21e7d Build: Split distributions into oss and default
This commit makes x-pack a module and adds it to the default
distrubtion. It also creates distributions for zip, tar, deb and rpm
which contain only oss code.
2018-04-20 15:33:57 -07:00
Ryan Ernst cb56bf49d5 Fixup build after opening x-pack 2018-04-20 15:33:46 -07:00
Ryan Ernst cf607e79d7 migrate master branch for xpack 2018-04-20 15:29:55 -07:00
Ryan Ernst 2efd22454a Migrate x-pack-elasticsearch source to elasticsearch 2018-04-20 15:29:54 -07:00
Jay Modi b57318c1a3 Execute watcher lifecycle changes in order (elastic/x-pack-elasticsearch#4437)
This change updates the WatcherLifecycleService to have its own single
thread executor that is used for lifecycle changes in order to have a
guarantee for the order that the changes are executed in.

Previously, a runnable would be submitted to the generic threadpool for
each lifecycle change that is needed. There was no guarantee of
ordering for these changes and no checks to see if a state change was
already in flight.

Relates elastic/x-pack-elasticsearch#4429

Original commit: elastic/x-pack-elasticsearch@14b73381db
2018-04-20 12:59:18 -06:00
Jay Modi 91ab88e86e Security: cache users in PKI realm (elastic/x-pack-elasticsearch#4428)
The PKI realm has never been a caching realm as the need had not
presented itself until now. The PKI realm relies on role mappings to
map the DN from a certificate to roles so that the users have the
appropriate access permissions. Without caching, this role mapping will
happen on every request. For file based role mappings, this is not an
issue as the mappings are based on equality checks for the DN.

However, the design of the API based role mappings allows for more
complex matches. These matches are implemented using automata, which
are built on every request that needs role mappings. Building automata
is an expensive operation and in combination with the PKI realm's lack
of caching leads to a significant performance impact.

The change in this commit makes the PkiRealm a caching realm using the
same pattern as other caching realms. The cache provided by
elasticsearch core is used to map the fingerprint of a certificate to
the user that was resolved from this certificate. The semantics of
modifications to this cache during iteration requires that we use a
read-write lock to protect access. There can be multiple concurrent
modifications and retrievals but iteration must be protected from any
attempts to modify the cache.

Additionally, some PKI tests were converted to single node tests as
part of this change. One test only used a single node and the other did
not require multiple nodes.

relates elastic/x-pack-elasticsearch#4406

Original commit: elastic/x-pack-elasticsearch@214772e1c1
2018-04-20 09:53:47 -06:00
Yannick Welsch 6a4c5f3e93
Abort early on finding duplicate snapshot name in internal structures (#29634)
Adds a check in BlobstoreRepository.snapshot(...) that prevents duplicate snapshot names and fails 
the snapshot before writing out the new index file. This ensures that you cannot end up in this
situation where the index file has duplicate names and cannot be read anymore .

Relates to #28906
2018-04-20 17:32:34 +02:00
Jay Modi cdf41cffd8 Watcher: use same value of watcher state (elastic/x-pack-elasticsearch#4430)
This commit updates the WatcherLifecycleService to use the same value
for the watcher service state when making comparisons to determine the
actions that need to be taken.

See elastic/x-pack-elasticsearch#4429

Original commit: elastic/x-pack-elasticsearch@c6c24fc6da
2018-04-20 08:05:58 -06:00
Jason Tedor 0045111ce2 Deprecate the suggest metrics (#29627)
The suggest stats were folded into the search stats as part of the
indices stats API in 5.0.0. However, the suggest metric remained as a
synonym for the search metric for BWC reasons. This commit deprecates
usage of the suggest metric on the indices stats API.

Similarly, due to the changes to fold the suggest stats into the search
stats, requesting the suggest index metric on the indices metric on the
nodes stats API has produced an empty object as the response since
5.0.0. This commit deprecates this index metric on the indices metric on
the nodes stats API.
2018-04-20 09:47:38 -04:00
Jay Modi dfc7ca7214
Implement Iterator#remove for Cache values iter (#29633)
This commit implements the ability to remove values from a Cache using
the values iterator. This brings the values iterator in line with the
keys iterator and adds support for removing items in the cache that are
not easily found by the key used for the cache.
2018-04-20 07:21:08 -06:00
Costin Leau 03ecbc0ccb SQL: implement COT, RANDOM, SIGN math functions (elastic/x-pack-elasticsearch#4394)
Add a number of missing math functions

Original commit: elastic/x-pack-elasticsearch@a26d9d2454
2018-04-20 15:07:14 +03:00
David Kyle 043a877a31 [ML] Prevent unnecessary job updates. (elastic/x-pack-elasticsearch#4424)
Original commit: elastic/x-pack-elasticsearch@8e0629789e
2018-04-20 12:42:31 +01:00
Ioannis Kakavas 14ad96ea8b Fix link to certificate settings in PKI realm (elastic/x-pack-elasticsearch#4402)
Remove indentation so that the two lines are not interpreted as
code block and the link is rendered correctly

Original commit: elastic/x-pack-elasticsearch@a73d52dfba
2018-04-20 12:43:55 +03:00
Costin Leau bc2ef139a8 SQL: Add Atan2 and Power functions (elastic/x-pack-elasticsearch#4412)
Add missing Atan2 & Power(and introduce BinaryMath operations), similar
to MathOperation.
Also align arithmetic package with binary math for code reuse.

Original commit: elastic/x-pack-elasticsearch@311961815e
2018-04-20 12:23:42 +03:00
Costin Leau e6f69ee269 SQL: Fix version loading in JDBC (elastic/x-pack-elasticsearch#4425)
When deployed, JDBC version needs to look at jars inside the classpath
not on the file system. Failing to do that, causes the version
information to be missing

Original commit: elastic/x-pack-elasticsearch@d35e8abb29
2018-04-20 12:19:45 +03:00
Ryan Ernst 935df40f70 Docs: Fix reference to legacy tribe section
Original commit: elastic/x-pack-elasticsearch@5261193fc1
2018-04-19 18:40:32 -07:00
Nhat Nguyen 42d81a2945 TEST: Unmute testPrimaryRelocationWhileIndexing
Previously we did not put an indexing to a version map if that map does
not require safe access but removed the existing delete tombstone only
if assertion enabled. In #29585, we removed the side-effect caused by
assertion then this test started failing. This failure can be explained
as follows:

- Step 1: Index a doc then delete that doc

- Step 2: The version map can switch to unsafe mode because of
  concurrent refreshes (implicitly called by flushes)

- Step 3: Index a document - the version map won't add this version
  value and won't prune the tombstone (previously it did)

- Step 4: Delete a document - this will return NOT_FOUND instead of
  DELETED because of the stale delete tombstone

This failure is actually fixed by #29619 in which we never leave stale
delete tombstones

Closes #29626
2018-04-19 21:35:21 -04:00
Ryan Ernst ba9c13302a Remove remaining references to tribe node (elastic/x-pack-elasticsearch#4407)
This commit review remaining docs and build references from the docs.

Original commit: elastic/x-pack-elasticsearch@33830b63ed
2018-04-19 18:02:09 -07:00
Ryan Ernst 7975280383
Remove remaining tribe node references (#29574)
While tribe node was removed in
https://github.com/elastic/elasticsearch/pull/28443, there remained a
couple lingering references to it in docs and code. This commit removes
those remaining references.
2018-04-19 18:02:01 -07:00
Nhat Nguyen 9cf8b01fc4
Never leave stale delete tombstones in version map (#29619)
Today the VersionMap does not clean up a stale delete tombstone if it
does not require safe access. However, in a very rare situation due to
concurrent refreshes, the safe-access flag may be flipped over then an
engine accidentally consult that stale delete tombstone.

This commit ensures to never leave stale delete tombstones in a version
map by always pruning delete tombstones when putting a new index entry
regardless of the value of the safe-access flag.
2018-04-19 20:49:56 -04:00