Commit Graph

50 Commits

Author SHA1 Message Date
Tanguy Leroux bf58660482
Remove all unused imports and fix CRLF (#31207)
The X-Pack opening and the recent other refactorings left a lot of 
unused imports in the codebase. This commit removes them all.
2018-06-11 15:12:12 +02:00
Hendrik Muhs 6577f5b0d1 silence InstallPluginCommandTests, see https://github.com/elastic/elasticsearch/issues/30900 2018-05-28 11:37:11 +02:00
Jason Tedor 35ffb8c65a
Add public key header/footer (#30877)
A previous commit added the public key used for signing artifacts to the
plugin CLI. This commit is an iteration on that to add the header and
footer to the key so that it is clear what the key is. Instead, we strip
the header/footer on read. With this change we simplify our test where
keys already in this format are generated and we had to strip on the
test side.
2018-05-25 18:06:59 -04:00
Jason Tedor d31e10a87d
Verify signatures on official plugins (#30800)
We sign our official plugins yet this is not well-advertised and not at
all consumed during plugin installation. For plugins that are installed
over the intertubes, verifying that the downloaded artifact is signed by
our signing key would establish both integrity and validity of the
downloaded artifact. The chain of trust here is simple: our installable
artifacts (archive and package distributions) so that if a user trusts
our packages via their signatures, and our plugin installer (which would
be executing trusted code) verifies the downloaded plugin, then the user
can trust the downloaded plugin too. This commit adds verification of
official plugins downloaded during installation. We do not add
verification for offline plugin installs; a user can download our
signatures and verify the artifacts themselves.

This commit also needs to solve a few interesting challenges. One of
these is that we want the bouncy castle JARs on the classpath only for
the plugin installer, but not for the runtime
Elasticsearch. Additionally, we want these JARs to not be present for
the JAR hell checks. To address this, we shift these JARs into a
sub-directory of lib (lib/tools/plugin-cli) that is only loaded for the
plugin installer, and in the plugin installer we filter any JARs in this
directory from the JAR hell check.
2018-05-25 07:56:35 -04:00
Jason Tedor 2844bcbcc8
Enable installing plugins from snapshots.elastic.co (#30765)
We post snapshot builds to snapshots.elastic.co yet the official plugin
installer will not let you install such plugins without manually
downloading them and installing them from a file URL. This commit adds
the ability for the plugin installer to use snapshots.elastic.co for
installing official plugins if a es.plugins.staging is set and the
current build is also a snapshot build. Otherwise, we continue to use
staging.elastic.co if the current build is a release build and
es.plugins.staging is set and, of course, use the release artifacts at
artifacts.elastic.co for release builds with es.plugins.staging unset.
2018-05-22 01:09:31 -04:00
Ryan Ernst b3f3a4312b
Plugins: Remove meta plugins (#30670)
Meta plugins existed only for a short time, in order to enable breaking
up x-pack into multiple plugins. However, now that x-pack is no longer
installed as a plugin, the need for them has disappeared. This commit
removes the meta plugins infrastructure.
2018-05-18 10:56:08 -07:00
Jason Tedor 3cadd5c40c Only enable modules to have native controllers
This commit removes the ability for a plugin to have a native controller
as leaves it as only modules can have a native controller.
2018-04-20 15:34:02 -07:00
Jason Tedor 2e869f2017 Handle installing X-Pack
X-Pack can no longer be installed as a plugin. This commit adds special
handling for when a user attempts to install X-Pack. This special
handling informs the user of the oss distribution that they should
download the default distribution and the user of the default
distribution that X-Pack does not require installation as it is included
by default.
2018-04-20 15:33:58 -07:00
Jason Tedor e64e6d8996 Add distribution flavor to startup scripts
This commit adds the distribution flavor (default versus oss) to the
build process which is passed through the startup scripts to
Elasticsearch. This change will be used to customize the message on
attempting to install/remove x-pack based on the distribution flavor.
2018-04-20 15:33:58 -07:00
Ryan Ernst fab5e21e7d Build: Split distributions into oss and default
This commit makes x-pack a module and adds it to the default
distrubtion. It also creates distributions for zip, tar, deb and rpm
which contain only oss code.
2018-04-20 15:33:57 -07:00
Ryan Ernst e3d954c6a5
Plugins: Fix native controller confirmation for non-meta plugin (#29434)
This commit fixes plugin warning confirmation to include native
controller confirmation when no security policy exists. The case was
already covered for meta plugins, but not for normal plugins. Tests are
also added for all cases.
2018-04-16 12:27:34 -07:00
Jason Tedor 5cdd831a31
Remove silent batch mode from install plugin (#29359)
Today we have a silent batch mode in the install plugin command when
standard input is closed or there is no tty. It appears that
historically this was useful when running tests where we want to accept
plugin permissions without having to acknowledge them. Now that we have
an explicit batch mode flag, this use-case is removed. The motivation
for removing this now is that there is another place where silent batch
mode arises and that is when a user attempts to install a plugin inside
a Docker container without keeping standard input open and attaching a
tty. In this case, the install plugin command will treat the situation
as a silent batch mode and therefore the user will never have the chance
to acknowledge the additional permissions required by a plugin. This
commit removes this silent batch mode in favor of using the --batch flag
when running tests and requiring the user to take explicit action to
acknowledge the additional permissions (either by leaving standard input
open and attaching a tty, or by passing the --batch flags themselves).

Note that with this change the user will now see a null pointer
exception when they try to install a plugin in a Docker container
without keeping standard input open and attaching a tty. This will be
addressed in an immediate follow-up, but because the implications of
that change are larger, they should be handled separately from this one.
2018-04-03 21:23:01 -04:00
Ryan Ernst 7112d4368f
Plugins: Fix module name conflict check for meta plugins (#29146)
This commit moves the check for plugin names conflicting with builtin
modules to a location that also applies to meta plugins.
2018-03-19 19:58:03 -07:00
Mayya Sharipova 8cb3d18eac Revert "Improve error message for installing plugin (#28298)"
This reverts commit 0cc1ffdf20

The reason is that Windows test are failing,
because of the incorrect path for the plugin
2018-03-15 10:47:50 -07:00
Mayya Sharipova 0cc1ffdf20
Improve error message for installing plugin (#28298)
Provide more actionable error message when installing an offline plugin
in the plugins directory, and the `plugins` directory for the node
contains plugin distribution.

Closes #27401
2018-03-14 16:19:04 -07:00
Jason Tedor 5904d936fa
Copy Lucene IOUtils (#29012)
As we have factored Elasticsearch into smaller libraries, we have ended
up in a situation that some of the dependencies of Elasticsearch are not
available to code that depends on these smaller libraries but not server
Elasticsearch. This is a good thing, this was one of the goals of
separating Elasticsearch into smaller libraries, to shed some of the
dependencies from other components of the system. However, this now
means that simple utility methods from Lucene that we rely on are no
longer available everywhere. This commit copies IOUtils (with some small
formatting changes for our codebase) into the fold so that other
components of the system can rely on these methods where they no longer
depend on Lucene.
2018-03-13 12:49:33 -04:00
Jason Tedor 6331bcaf76
Create keystore on package install (#28928)
This commit removes the ability to specify that a plugin requires the
keystore and instead creates the keystore on package installation or
when Elasticsearch is started for the first time. The reason that we opt
to create the keystore on package installation is to ensure that the
keystore has the correct permissions (the package installation scripts
run as root as opposed to Elasticsearch running as the elasticsearch
user) and to enable removing the keystore on package removal if the
keystore is not modified.
2018-03-12 12:48:00 -04:00
Ryan Ernst ea381969be
Plugins: Separate plugin semantic validation from properties format validation (#28581)
This commit moves the semantic validation (like which version a plugin
was built for or which java version it is compatible with) from reading
a plugin descriptor, leaving the checks on the format of the descriptor
intact.

relates #28540
2018-02-12 21:30:11 -08:00
Ryan Ernst 65f1dd424e
Plugins: Remove intermediate "elasticsearch" directory within plugin zips (#28589)
This commit removes the extra layer of all plugin files existing under
"elasticsearch" within plugin zips. This simplifies building plugin zips
and removes the need for special logic of modules vs plugins.
2018-02-12 14:27:30 -08:00
Ryan Ernst 20c37efea2
Build: Replace provided configuration with compileOnly (#28564)
When elasticsearch was originally moved to gradle, the "provided" equivalent in maven had to be done through a plugin. Since then, gradle added the "compileOnly" configuration. This commit removes the provided plugin and replaces all uses with compileOnly.
2018-02-09 11:30:24 -08:00
Ryan Ernst a55eda626f
Plugins: Store elasticsearch and java versions in PluginInfo (#28556)
Plugin descriptors currently contain an elasticsearch version,
which the plugin was built against, and a java version, which the plugin
was built with. These versions are read and validated, but not stored.
This commit keeps them in PluginInfo so they can be used later.
While seeing the elasticsearch version is less interesting (since it is
enforced to match that of the running elasticsearc node), the java
version is interesting since we only validate the format, not the actual
version. This also makes PluginInfo have full parity with the plugin
properties file.
2018-02-08 08:31:39 -08:00
Jason Tedor c2fcf15d9d
Fix the ability to remove old plugin
We now read the plugin descriptor when removing an old plugin. This is
to check if we are removing a plugin that is extended by another
plugin. However, when reading the descriptor we enforce that it is of
the same version that we are. This is not the case when a user has
upgraded Elasticsearch and is now trying to remove an old plugin. This
commit fixes this by skipping the version enforcement when reading the
plugin descriptor only when removing a plugin.

Relates #28540
2018-02-06 17:38:26 -05:00
Christoph Büscher 202d28be86
[Test] Fix InstallPluginCommandTests failure on Windows (#28447)
The `testMetaPluginPolicyConfirmation` needs to close the file streams it is
iterating over, otherwise some OSes (like Windows) might not be able to delete
all temporary folders, which in turn leads to test failures.

Closes #28415
2018-01-31 09:20:51 +01:00
Christoph Büscher f6a7ee91c9 Add @AwaitsFix for failing InstallPluginCommandTests
`testMetaPluginPolicyConfirmation` fails consistently on windows (#28415).
2018-01-30 11:59:05 +01:00
Ryan Ernst b47b399f00
Settings: Reimplement keystore format to use FIPS compliant algorithms (#28255)
This commit switches the internal format of the elasticsearch keystore
to no longer use java's KeyStore class, but instead encrypt the binary
data of the secrets using AES-GCM. The cipher key is generated using
PBKDF2WithHmacSHA512. Tests are also added for backcompat reading the v1
and v2 formats.
2018-01-26 15:51:07 -08:00
Ryan Ernst 3dd833ca0a
Plugins: Use one confirmation of all meta plugin permissions (#28366)
Currently meta plugins will ask for confirmation of security policy
exceptions for each bundled plugin. This commit collects the necessary
permissions of each bundled plugin, and asks for confirmation of all of
them at the same time.
2018-01-26 15:44:44 -08:00
Ryan Ernst de9d903b1e
Plugins: Fix meta plugins to install bundled plugins with their real name (#28285)
Meta plugins move the unzipped plugin as is, but the inner plugins may
have a different directory name than their corresponding plugin
properties file specifies. This commit fixes installation to rename the
directory if necessary.
2018-01-18 12:01:20 -08:00
Jim Ferenczi fcf4114adc
Make sure that we don't detect files as maven coordinate when installing a plugin (#28163)
* This change makes sure that we don't detect a file path containing a ':' as
a maven coordinate (e.g.: `file:C:\path\to\zip`)

* restore test muted on master
2018-01-10 14:59:37 +01:00
Jim Ferenczi 5cac7eac2b
meta-plugin should install bin and config at the top level (#28162)
This change modifies the installation for a meta plugin,
the content of the config and bin directory inside each bundled plugins are now moved in the meta plugin directory.
So instead of `$configDir/meta-plugin-name/bundled_plugin/name/` the content of the config
for a bundled plugin is now in `$configDir/meta-plugin-name`. Same applies for the bin directory.
2018-01-10 02:49:52 +01:00
Jim Ferenczi 36729d1c46
Add the ability to bundle multiple plugins into a meta plugin (#28022)
This commit adds the ability to package multiple plugins in a single zip.
The zip file for a meta plugin must contains the following structure:

|____elasticsearch/
| |____   <plugin1> <-- The plugin files for plugin1 (the content of the elastisearch directory)
| |____   <plugin2>  <-- The plugin files for plugin2
| |____   meta-plugin-descriptor.properties <-- example contents below
The meta plugin properties descriptor is mandatory and must contain the following properties:

description: simple summary of the meta plugin.
name: the meta plugin name
The installation process installs each plugin in a sub-folder inside the meta plugin directory.
The example above would create the following structure in the plugins directory:

|_____ plugins
| |____   <name_of_the_meta_plugin>
| | |____   meta-plugin-descriptor.properties
| | |____   <plugin1>
| | |____   <plugin2>
If the sub plugins contain a config or a bin directory, they are copied in a sub folder inside the meta plugin config/bin directory.

|_____ config
| |____   <name_of_the_meta_plugin>
| | |____   <plugin1>
| | |____   <plugin2>

|_____ bin
| |____   <name_of_the_meta_plugin>
| | |____   <plugin1>
| | |____   <plugin2>
The sub-plugins are loaded at startup like normal plugins with the same restrictions; they have a separate class loader and a sub-plugin
cannot have the same name than another plugin (or a sub-plugin inside another meta plugin).

It is also not possible to remove a sub-plugin inside a meta plugin, only full removal of the meta plugin is allowed.

Closes #27316
2018-01-09 18:28:43 +01:00
Ryan Ernst d36ec18029
Plugins: Add plugin extension capabilities (#27881)
This commit adds the infrastructure to plugin building and loading to
allow one plugin to extend another. That is, one plugin may extend
another by the "parent" plugin allowing itself to be extended through
java SPI. When all plugins extending a plugin are finished loading, the
"parent" plugin has a callback (through the ExtensiblePlugin interface)
allowing it to reload SPI.

This commit also adds an example plugin which uses as-yet implemented
extensibility (adding to the painless whitelist).
2018-01-03 11:12:43 -08:00
Jason Tedor 0519fa223c
Ensure logging is configured for CLI commands
Any CLI commands that depend on core Elasticsearch might touch classes
(directly or indirectly) that depends on logging. If they do this and
logging is not configured, Log4j will dump status error messages to the
console. As such, we need to ensure that any such CLI command configures
logging (with a trivial configuration that dumps log messages to the
console). Previously we did this in the base CLI command but with the
refactoring of this class out of core Elasticsearch, we no longer
configure logging there (since we did not want this class to depend on
settings and logging). However, this meant for some CLI commands (like
the plugin CLI) we were no longer configuring logging. This commit adds
base classes between the low-level command and multi-command classes
that ensure that logging is configured. Any CLI command that depends on
core Elasticsearch should use this infrastructure to ensure logging is
configured. There is one exception to this: Elasticsearch itself because
it takes reponsibility into its own hands for configuring logging from
Elasticsearch settings and log4j2.properties. We preserve this special
status.

Relates #27523
2017-11-25 11:40:08 -05:00
Michael Basnight 2949c53174
Remove config prompting for secrets and text (#27216)
This commit removes the ability to use ${prompt.secret} and
${prompt.text} as valid config settings. Secure settings has obsoleted
the need for this, and it cleans up some of the code in Bootstrap.
2017-11-19 22:33:17 -06:00
Michael Basnight cb3e8f4763
Move the CLI into its own subproject (#27114)
Projects the depend on the CLI currently depend on core. This should not
always be the case. The EnvironmentAwareCommand will remain in :core,
but the rest of the CLI components have been moved into their own
subproject of :core, :core:cli.
2017-11-18 21:42:57 -06:00
David Roberts 749c3ec716
Remove the single argument Environment constructor (#27235)
Only tests should use the single argument Environment constructor.  To
enforce this the single arg Environment constructor has been replaced with
a test framework factory method.

Production code (beyond initial Bootstrap) should always use the same
Environment object that Node.getEnvironment() returns.  This Environment
is also available via dependency injection.
2017-11-04 13:25:09 +00:00
Jason Tedor cfa4646161 Adjust SHA-512 supported format on plugin install
This commit adjusts the format of the SHA-512 checksum files supported
by the plugin installer. In particular, we now require that the SHA-512
format be a single-line file containing the checksum followed by two
spaces followed by the filename. We continue to support the legacy
format for SHA-1.

Relates #27093
2017-10-25 07:53:33 -04:00
Simon Willnauer a506ba8602 Remove `Settings,put(Map<String,String>)` (#26785)
`Map<String,String>` is basically erasing the type while other methods on
the `Settings.Builder` are type safe and have corresponding `get` methods.
2017-09-26 12:15:20 +02:00
Ryan Ernst 5b711c283d Plugins: Add backcompat for sha1 checksums (#26748)
With 6.0 rc1 we now publish sha512 checksums for official plugins.
However, in order to ease the pain for plugin authors, this commit adds
backcompat to still allow sha1 checksums.  Also added tests for
checksums.

Closes #26746
2017-09-22 11:26:32 +02:00
Ryan Ernst 5202e7e93b Settings: Move keystore creation to plugin installation (#26329)
This commit removes the keystore creation on elasticsearch startup, and
instead adds a plugin property which indicates the plugin needs the
keystore to exist. It does still make sure the keystore.seed exists on
ES startup, but through an "upgrade" method that loading the keystore in
Bootstrap calls.

closes #26309
2017-08-24 12:12:47 -07:00
Jason Tedor 9f12f3c338 Fix checkstyle violation in remove plugin tests
This commit fixes a redundant modifier checkstyle violation in
RemovePluginCommandTests.java.
2017-07-28 14:52:11 +09:00
Jason Tedor 1492ccd7ae Fix environment-aware command tests
This commit fixes tests for environment-aware commands. A previous
change added a check that es.path.conf is not null. The problem is that
this system property is not being set in tests so this check trips every
single time. To fix this, we move the check into a method that can be
overridden, and then override this method in relevant places in tests to
avoid having to set the property in tests. We also add a test that this
check works as expected.
2017-07-28 14:37:04 +09:00
Jason Tedor 5961e1d6ab Remove import from InstallPluginCommandTests
This commit removes an unused import from
InstallPluginCommandTests.java.
2017-06-01 08:56:25 -04:00
Jason Tedor 9b4a189147 Add purge option to remove plugin CLI
By default, the remove plugin CLI command preserves configuration
files. This is so that if a user is upgrading the plugin (which is done
by first removing the old version and then installing the new version)
they do not lose their configuration file. Yet, there are circumstances
where preserving the configuration file is not desired. This commit adds
a purge option to the remove plugin CLI command.

Relates #24981
2017-06-01 08:53:39 -04:00
Ryan Ernst 2a65bed243 Tests: Change rest test extension from .yaml to .yml (#24659)
This commit renames all rest test files to use the .yml extension
instead of .yaml. This way the extension used within all of
elasticsearch for yaml is consistent.
2017-05-16 17:24:35 -07:00
Jason Tedor 3e485c2ca5 Fix plugin installation permissions
When installing plugin permissions, we try to set the permissions on all
installed files ourselves because a umask from the user could violate
everything needed to get the permissions right. Sadly, directories were
not handled correctly at all and so we were still left with broken
installations with umasks like 0077. This commit fixes this issue, adds
a thorough unit test for the situation, and most importantly, adds a
test that sets the umask before installing the plugin.

Relates #24527
2017-05-06 13:25:10 -04:00
Ryan Ernst cdcc75dd2a Plugins: Add support for platform specific plugins (#24265)
This commit adds support for plugins having a platform specific variant.
It also adds unit tests for all official and maven urls.
2017-04-27 11:27:29 -07:00
Jason Tedor 08f52d1a18 Remove extra newline in RemovePluginCommand.java
This commit removes an extra newline in RemovePluginCommand.java because
it drives the OCD among us insane.
2017-04-21 22:18:33 -04:00
Jason Tedor 9912650641 Fix delete of plugin directory on remove plugin
This commit fixes an issue when deleting the plugin directory while
executing the remove plugin command. Namely, we take out a file
descriptor on the plugin directory to traverse its contents to obtain
the list of files to delete. We leaked this file descriptor. On
Unix-based filesystems, this is not a problem, deleting the plugin
directory deletes the plugin directory. On Windows though, a delete is
not executed until the last file descriptor is closed. Since we leaked
this file descriptor, the plugin was not actually deleted. This led to
test failures that tried to cleanup left behind temporary directories
but these test failures were just exposing this bug. This commit fixes
this issue by ensuring that we close the file descriptor to the plugin
directory when we are finished with it.

Relates #24266
2017-04-21 22:15:29 -04:00
Jason Tedor fe91c72151 Use a marker file when removing a plugin
Today when removing a plugin, we attempt to move the plugin directory to
a temporary directory and then delete that directory from the
filesystem. We do this to avoid a plugin being in a half-removed
state. We previously tried an atomic move, and fell back to a non-atomic
move if that failed. Atomic moves can fail on union filesystems when the
plugin directory is not in the top layer of the
filesystem. Interestingly, the regular move can fail as well. This is
because when the JDK is executing such a move, it first tries to rename
the source directory to the target directory and if this fails with
EXDEV (as in the case of an atomic move failing), it falls back to
copying the source to the target, and then attempts to rmdir the
source. The bug here is that the JDK never deleted the contents of the
source so the rmdir will always fail (except in the case of an empty
directory).

Given all this silliness, we were inspired to find a different
strategy. The strategy is simple. We will add a marker file to the
plugin directory that indicates the plugin is in a state of
removal. This file will be the last file out the door during removal. If
this file exists during startup, we fail startup.

Relates #24252
2017-04-21 15:50:44 -04:00
Ryan Ernst ba48674695 Build: Move plugin cli and tests to distribution tool (#24220)
The plugin cli currently resides inside the elasticsearch jar. This
commit moves it into a plugin-cli jar. This is change alone is a no-op;
it does not change anything about what is loaded at runtime. But it will
allow easier testing (with fixtures in the future to test ES or maven
installation), as well as eventually not loading these classes when
starting elasticsearch.
2017-04-21 09:25:58 -07:00