Commit Graph

541 Commits

Author SHA1 Message Date
Ryan Ernst 3aec8c7725
Packaging: Set elasticsearch user to have non-existent homedir ()
This commit adds setting the homedir for the elasticsearch user to the
adduser command in the packaging preinstall script. While the
elasticsearch user is a system user, it is sometimes conventient to have
an existing homedir (even if it is not writeable). For example, running
cron as the elasticsearch user will try to change dir to the homedir.

closes 
2018-05-02 14:33:34 -07:00
Ryan Ernst fba2f00a73
Packaging: Unmark systemd service file as a config file ()
Systemd overrides should happen through /etc/systemd/system, not
directly editing the service file. This commit removes marking the
service file as configuration for rpm and deb packages.
2018-05-02 09:48:49 -07:00
David Roberts ea35a16645
Create default ES_TMPDIR on Windows ()
If the elasticsearch-env bash script chooses $ES_TMPDIR
then it also creates the directory.  This change makes
elasticsearch-env.bat do the same thing: if %ES_TMPDIR%
is chosen by the script then the script will ensure it
exists, but if %ES_TMPDIR% is already set then the user
is responsible for creating it.

Relates 
Relates 
2018-05-02 12:11:52 +01:00
David Roberts 225f7093a9
[ML] Include 3rd party C++ component notices ()
The overall NOTICE file for the ML X-Pack module should
include the notices from the 3rd party C++ components as
well as the 3rd party Java components.
2018-04-30 20:05:27 +01:00
Ryan Ernst 4cfca2fbd0
Build: Fix deb version to use tilde with prerelease versions ()
This commit converts the deb package to use tildes in place of dash in
the internal package version. This is only relevant for prerelease
versions of elasticsearch. Previously, this was not possible due to
problems with the underlying library used by the ospackage plugin, but
since a recent upgrade, it now works.

closes 
2018-04-26 11:51:48 -07:00
Nik Everett a8f40b3e04 Build: Assert jar LICENSE and NOTICE files match
Adds tasks that check that the all jars that we build have LICENSE.txt
and NOTICE.txt files and that the files are correct. Sets check to
depend on these task.

This is mostly there for extra parnoia because we automatically
configure all Jar tasks to include the LICENSE.txt and NOTICE.txt
files anyway. But it is quite possible to add configuration to those
tasks that would override either file.

This causes check to depend on several more things than it used to.
Take, for example, javadoc:

check depends on the new verifyJavadocJarNotice which depends on
extractJavadocJar which depends on javadocJar which depends on
javadoc, this check now depends on javadoc.
2018-04-25 19:53:24 -04:00
Jason Tedor f1aedd9ae8
Add build time checks for package licenses
This commit adds some build time checks that the archive distributions
and package distributions contain the appropriate license and notice
files, and the package distributions contain the appropriate license
metadata.
2018-04-24 12:10:51 -04:00
Ryan Ernst 7abc55f905 Build: Fix License attribute to be written in deb control data
This commit uses the customFields setting of the Deb task in ospackage
to work around the fact it does not know anything about the License
attribute natively.
2018-04-23 17:24:23 -07:00
Ryan Ernst ea2a29530c Build: Use templated copyright file for deb distributions
THe deb distribution has a special copyright file instead of
LICENSE.txt, but the distributions were including the template file
instead of the rendered file (which includes the license name and text).
2018-04-23 09:43:23 -07:00
Jason Tedor 11244cb443
Fix the dashes in license names
For the Debian packages, the license names should contain a dash, but
not for the RPM packages. This commit fixes this.
2018-04-23 11:49:54 -04:00
Jason Tedor 3cadd5c40c Only enable modules to have native controllers
This commit removes the ability for a plugin to have a native controller
as leaves it as only modules can have a native controller.
2018-04-20 15:34:02 -07:00
Jason Tedor d99d0fa669 Add distribution type to startup scripts
This commit adds the distribution type to the startup scripts so that we
can discern from log output and the main response the type of the
distribution (deb/rpm/tar/zip).
2018-04-20 15:34:01 -07:00
Ryan Ernst 0d8aa7527e Reorganize license files
This commit moves the apache and elastic license files into a new
root level `licenses` directory and rewrites the top level LICENSE.txt
to clarify the repository has a mix of apache and elastic licensed code.
2018-04-20 15:33:59 -07:00
Ryan Ernst 1fb2d08b31 Packaging: Add license specifications to packages
This commit adds license metadata to rpm and deb packages. Additionally,
it makes the copyright file for deb files follow the machine readable
specification, and sets the correct license text based on the oss vs
default deb packages.
2018-04-20 15:33:59 -07:00
Jason Tedor 2e869f2017 Handle installing X-Pack
X-Pack can no longer be installed as a plugin. This commit adds special
handling for when a user attempts to install X-Pack. This special
handling informs the user of the oss distribution that they should
download the default distribution and the user of the default
distribution that X-Pack does not require installation as it is included
by default.
2018-04-20 15:33:58 -07:00
Jason Tedor 0750f5d6f3 Add conflicts for default and oss packages
This commit adds conflicts between the default and oss packages so that
a user can not install both via a package manager (as they share paths).
2018-04-20 15:33:58 -07:00
Jason Tedor e64e6d8996 Add distribution flavor to startup scripts
This commit adds the distribution flavor (default versus oss) to the
build process which is passed through the startup scripts to
Elasticsearch. This change will be used to customize the message on
attempting to install/remove x-pack based on the distribution flavor.
2018-04-20 15:33:58 -07:00
Ryan Ernst fab5e21e7d Build: Split distributions into oss and default
This commit makes x-pack a module and adds it to the default
distrubtion. It also creates distributions for zip, tar, deb and rpm
which contain only oss code.
2018-04-20 15:33:57 -07:00
Ryan Ernst 4f282e9e32
Build: Move java home checks to pre-execution phase ()
This commit moves the checks on JAVAX_HOME (where X is the java version
number) existing to the end of gradle's configuration phase, and based
on whether the tasks needing the java home are configured to execute.

relates 
2018-04-19 09:51:52 -07:00
Ryan Ernst e3d954c6a5
Plugins: Fix native controller confirmation for non-meta plugin ()
This commit fixes plugin warning confirmation to include native
controller confirmation when no security policy exists. The case was
already covered for meta plugins, but not for normal plugins. Tests are
also added for all cases.
2018-04-16 12:27:34 -07:00
Jason Tedor b883e1217f
Lazy configure build tasks that require older JDKs ()
Some build tasks require older JDKs. For example, the BWC build tasks
for older versions of Elasticsearch require older JDKs. It is onerous to
require these be configured when merely compiling Elasticsearch, the
requirement that they be strictly set to appropriate values should only
be enforced if these tasks are going to be executed. To address this, we
lazy configure these tasks.
2018-04-14 15:44:43 -04:00
Jason Tedor 27fafa24f5
Use proper Java version for BWC builds ()
Today we have JAVA_HOME for the compiler Java home and RUNTIME_JAVA_HOME
for the test Java home. However, when we compile BWC nodes and run them,
neither of these Java homes might be the version that was suitable for
that BWC node (e.g., 5.6 requires JDK 8 to compile and to run). This
commit adds support for the environment variables JAVA\d+_HOME and uses
the appropriate Java home based on the version of the node being
started. We even do this for reindex-from-old which requires JDK 7 for
these very old nodes. Note that these environment variables are not
required if not running BWC tests, and they are strictly required if
running BWC tests.
2018-04-13 12:41:18 -04:00
Jason Tedor 03ce3dd4a4 Enable skipping fetching latest for BWC builds ()
The BWC builds always fetch the latest from the elastic/elasticsearch
repository for the BWC branches. Yet, there are use-cases for using the
local checkout without fetching the latest. This commit enables these
use-cases by adding a tests.bwc.git.fetch.latest property to skip the
fetches.
2018-04-13 09:31:06 -04:00
Van0SS 4310ef0be6 [Docs] Update Copyright notices to 2018 () 2018-04-06 16:21:20 +02:00
Jason Tedor 5cdd831a31
Remove silent batch mode from install plugin ()
Today we have a silent batch mode in the install plugin command when
standard input is closed or there is no tty. It appears that
historically this was useful when running tests where we want to accept
plugin permissions without having to acknowledge them. Now that we have
an explicit batch mode flag, this use-case is removed. The motivation
for removing this now is that there is another place where silent batch
mode arises and that is when a user attempts to install a plugin inside
a Docker container without keeping standard input open and attaching a
tty. In this case, the install plugin command will treat the situation
as a silent batch mode and therefore the user will never have the chance
to acknowledge the additional permissions required by a plugin. This
commit removes this silent batch mode in favor of using the --batch flag
when running tests and requiring the user to take explicit action to
acknowledge the additional permissions (either by leaving standard input
open and attaching a tty, or by passing the --batch flags themselves).

Note that with this change the user will now see a null pointer
exception when they try to install a plugin in a Docker container
without keeping standard input open and attaching a tty. This will be
addressed in an immediate follow-up, but because the implications of
that change are larger, they should be handled separately from this one.
2018-04-03 21:23:01 -04:00
Ryan Ernst a75a7d22b2
Build: Use branch specific refspec sysprop for bwc builds ()
This commit changes the sysprop for overriding the branch bwc builds use
to be branch specific. There are 3 different bwc branches built, but all
of them currently read the exact same sysprop. For example, with this change
and current branches, you can now specify eg `-Dtests.bwc.refspec.6.x=my_6x`
and it will build only next-minor-snapshot with that branch, while
next-bugfix-snapshot will continue to use 5.6.
2018-03-29 11:59:52 -07:00
Ryan Ernst 7112d4368f
Plugins: Fix module name conflict check for meta plugins ()
This commit moves the check for plugin names conflicting with builtin
modules to a location that also applies to meta plugins.
2018-03-19 19:58:03 -07:00
Jason Tedor 0abf51af3d
Configure error file for archive packages ()
This is a follow up to a previous change which set the error file path
for the package distributions. The observation here is that we always
set the working directory of Elasticsearch to the root of the
installation (i.e., Elasticsearch home). Therefore, we can specify the
error file path relative to this directory and default it to the logs
directory, similar to the package distributions.
2018-03-18 15:33:17 -04:00
Jason Tedor 29fedb2669
Configure heap dump path for archive packages ()
This is a follow up to a previous change which set the heap dump path
for the package distributions. The observation here is that we always
set the working directory of Elasticsearch to to the root of
installation (i.e., Elasticsearch home). Therefore, we can specify the
heap dump path relative to this directory and default it to the data
directory, similar to the package distributions.
2018-03-18 15:32:37 -04:00
Jason Tedor b56afebad1
Fix creating keystore when upgrading ()
When upgrading via the RPM package, we can run into a problem where
the keystore fails to be created. This arises because the %post script
on RPM runs after the new package files are installed but before the
removal of the old package files. This means that the contents of the
lib folder can contain files from the old package and the new package
and thus running the create keystore tool can encounter JAR hell
issues and fail. To solve this, we move creating the keystore to the
%posttrans script which runs after the old package files are
removed. We only need to do this on the RPM package, so we add a
switch in the shared post-install script.
2018-03-17 07:48:40 -04:00
Martijn Laarman 42c7c75298 Fix starting on Windows from another drive ()
The cd command on Windows has an oddity regarding changing
directories. If the drive of the current directory is a different drive
than than of the directory that was passed to the cd command, cd acts in
query mode and does not change the current directory. Instead, a flag is
needed to put the cd command into set mode so that the directory
actually changes. This causes a problem when starting Elasticsearch from
a directory different than the one where it is installed and this commit
fixes the issue.
2018-03-16 07:28:24 -04:00
Jason Tedor 4897e0034b
Allow overriding JVM options in Windows service ()
Today we allow any other method of starting Elastisearch to override
jvm.options via ES_JAVA_OPTS. Yet, for some settings in the Windows
service, we do not allow this. This commit removes this in favor of
being consistent with other packaging choices.
2018-03-15 21:35:40 -04:00
Mayya Sharipova 8cb3d18eac Revert "Improve error message for installing plugin ()"
This reverts commit 0cc1ffdf20

The reason is that Windows test are failing,
because of the incorrect path for the plugin
2018-03-15 10:47:50 -07:00
Mayya Sharipova 0cc1ffdf20
Improve error message for installing plugin ()
Provide more actionable error message when installing an offline plugin
in the plugins directory, and the `plugins` directory for the node
contains plugin distribution.

Closes 
2018-03-14 16:19:04 -07:00
Jason Tedor 98ad2596d8
Put JVM crash logs in the default log directory ()
This commit adds a JVM flag to ensure that the JVM fatal error logs land
in the default log directory. Users that wish to use an alternative
location should change the path configured here.
2018-03-13 16:32:16 -04:00
Jason Tedor 5904d936fa
Copy Lucene IOUtils ()
As we have factored Elasticsearch into smaller libraries, we have ended
up in a situation that some of the dependencies of Elasticsearch are not
available to code that depends on these smaller libraries but not server
Elasticsearch. This is a good thing, this was one of the goals of
separating Elasticsearch into smaller libraries, to shed some of the
dependencies from other components of the system. However, this now
means that simple utility methods from Lucene that we rely on are no
longer available everywhere. This commit copies IOUtils (with some small
formatting changes for our codebase) into the fold so that other
components of the system can rely on these methods where they no longer
depend on Lucene.
2018-03-13 12:49:33 -04:00
Jason Tedor 46e16b68fe Fix packaging scripts references to /etc/elasticsearch
We no longer source the environment file in the packaging scripts yet we
had leftover references to variables defined by those environment
files. This commit cleans these up.
2018-03-13 10:07:02 -04:00
Jason Tedor f0164cc954
Stop sourcing scripts during installation/removal ()
Previously we allowed a lot of customization of Elasticsearch during
package installation (e.g., the username and group). This customization
was achieved by sourcing the env script (e.g.,
/etc/sysconfig/elasticsearch) during installation. Since we no longer
allow such flexibility, we do not need to source these env scripts
during package installation and removal.
2018-03-12 13:44:47 -04:00
Jason Tedor 6331bcaf76
Create keystore on package install ()
This commit removes the ability to specify that a plugin requires the
keystore and instead creates the keystore on package installation or
when Elasticsearch is started for the first time. The reason that we opt
to create the keystore on package installation is to ensure that the
keystore has the correct permissions (the package installation scripts
run as root as opposed to Elasticsearch running as the elasticsearch
user) and to enable removing the keystore on package removal if the
keystore is not modified.
2018-03-12 12:48:00 -04:00
Ryan Ernst 0f95636a91
Build: Remove rest tests on archive distribution projects ()
This commit removes running rest tests on the full zip and tar
distributions in favor of doing a simple extraction check like is done
for rpm and deb files. The rest tests are still run on the integ test
zip, at least for now (this should eventually be moved out to a different
location).
2018-03-08 19:46:54 -08:00
Ryan Ernst b29ba25c86 Build: Fix packages distributions inclusion of empty directories
This was accidentally broken in .
2018-02-22 11:30:49 -08:00
Ryan Ernst 014e90d903
Build: Consolidate archives and packages configuration ()
This commit moves the distribution specific tasks into the respective
archives and packages builds. The collocation of common and distribution
specific tasks make it much easier to reason about what is expected in a
particular distribution.
2018-02-21 17:46:40 -08:00
Jason Tedor ed2bbc6e64
Delay path expansion on Windows
There is a bug in the for statement where we execute the JVM options
parser. The bug manfiests in the handling of paths with ) in the
name. The problem is this: we use a for statement to capture the output
of the JVM options parser. A for statement that executes a command
defers execution to cmd. There is this gem from the help:

  1.  If all of the following conditions are met, then quote characters
      on the command line are preserved:

      - no /S switch
      - exactly two quote characters
      - no special characters between the two quote characters,
        where special is one of: &<>()@^|
      - there are one or more whitespace characters between the
        two quote characters
      - the string between the two quote characters is the name
        of an executable file.

  2.  Otherwise, old behavior is to see if the first character is
      a quote character and if so, strip the leading character and
      remove the last quote character on the command line, preserving
      any text after the last quote character.

This means that the ) causes the quotes to be stripped which ruins
everything. This commit fixes this by delaying expansion of the paths.

Relates 
2018-02-21 10:58:49 -05:00
Jason Tedor 1fa701c18d
Fix using relative custom config path
Previously a user could set a custom config path to a relative directory
using ES_PATH_CONF. In a previous change related to enabling GC logging
by default, we forced the working directory for Elasticsearch to be
ES_HOME. This had the impact of causing all relative paths to be
relative to ES_HOME, against the intent of the user. This commit
addresses this by making ES_PATH_CONF absolute before we switch the
working directory to ES_HOME.

Relates 
2018-02-16 06:08:54 -05:00
Ryan Ernst 53c38cc8fe
Build: Group archive and package distribution projects ()
This commit adds intermediate gradle projects for archive based
distributions (zip, tar) and package based distributions (rpm, deb). The
grouping allows the common distribution build file to be considerably
shorter and clearly separated from the common zip/tar and rpm/deb
configuration.
2018-02-13 22:49:53 -08:00
Michael Basnight 04fe9a9ab5
Relax remote check for bwc project checkouts ()
The remote check previously validated both the remote name and the
repository as well, meaning that if someone passed in a repository that
was not a github URL, it would fail. This meant that it was not possible
to fully test bwc out with multiple branches without first pushing to a
remote. Removing the full check allows a user to pass in the origin
remote as its remote, which is already added as a file based remote to
each bwc snapshot build. This will allow changes to be made locally
across all bwc branches, tested, and then pushed simultaneously.
2018-02-13 14:54:11 -06:00
Michael Basnight 3cd4da792c
Remove snapshot conditional for bwc snapshots ()
The build.snapshot flag used by the main build was being propagated down
into the bwc snapshot builds, which is not correct. The bwc subprojects
are always meant to be snapshot builds, or null if they do not
exist. Marking these builds as non snapshots threw the release off as it
was looking for -SNAPSHOT builds.

Relates 
2018-02-13 10:35:20 -06:00
Ryan Ernst ea381969be
Plugins: Separate plugin semantic validation from properties format validation ()
This commit moves the semantic validation (like which version a plugin
was built for or which java version it is compatible with) from reading
a plugin descriptor, leaving the checks on the format of the descriptor
intact.

relates 
2018-02-12 21:30:11 -08:00
Ryan Ernst 65f1dd424e
Plugins: Remove intermediate "elasticsearch" directory within plugin zips ()
This commit removes the extra layer of all plugin files existing under
"elasticsearch" within plugin zips. This simplifies building plugin zips
and removes the need for special logic of modules vs plugins.
2018-02-12 14:27:30 -08:00
Jason Tedor 69313ffef3
Disable console logging in the Windows service
When Elasticsearch is run as a service we should not use the console
logger otherwise we end up duplicating logging (to the Elasticsearch
logs and whereever standard output is captured). Previously we disabled
the console logger when started as a service using systemd (otherwise
the console logs are duplicated to the journal). This commit does the
same for the Windows service, starting Elasticsearch with the --quiet
flag to avoid standard output being written to the service stdout logs.

Relates 
2018-02-11 11:10:40 -05:00