Commit Graph

95 Commits

Author SHA1 Message Date
Lee Hinman ac53c90ff4 Add 'elasticsearch-translog' CLI tool with 'translog' command
This adds the `bin/elasticsearch-translate` bin file that will be used
for CLI tasks pertaining to Elasticsearch. Currently it implements only
a single sub-command, `truncate-translog`, that creates a truncated
translog for a given folder.

Here's what running the tool looks like:

```
λ bin/elasticsearch-translog truncate -d data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/
Checking existing translog files
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
!   WARNING: Elasticsearch MUST be stopped before running this tool   !
!                                                                     !
!   WARNING:    Documents inside of translog files will be lost       !
!                                                                     !
!   WARNING:          The following files will be DELETED!            !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-10.tlog
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-18.tlog
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-21.tlog
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-12.ckp
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-25.ckp
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-29.tlog
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-2.tlog
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-5.tlog
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-41.ckp
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-6.ckp
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-37.ckp
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-24.ckp
--> data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-11.ckp

Continue and DELETE files? [y/N] y
Reading translog UUID information from Lucene commit from shard at [data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/index]
Translog Generation: 3
Translog UUID      : AxqC4rocTC6e0fwsljAh-Q
Removing existing translog files
Creating new empty checkpoint at [data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog.ckp]
Creating new empty translog at [data/nodes/0/indices/P45vf_YQRhqjfwLMUvSqDw/0/translog/translog-3.tlog]
Done.
```

It also includes a `-b` batch operation that can be used to skip the
confirmation diaglog.

Resolves #19123
2016-07-26 08:34:07 -06:00
Jason Tedor d29d2f8793 Disable service in pre-uninstall
Today in the packaging removal scripts, we disable the service in
post-uninstall. Yet, this happens after service files have been
erased. On some systems, this can cause the service disable to fail
leaving behind state causing the service to be enabled on subsequent
installs. This commit moves the service disabling to the pre-uninstall
script to prevent this issue.

Relates #19328
2016-07-08 07:58:52 -04:00
Jason Tedor 370b198f9f Remove explicit parallel new GC flag
Unless explicitly disabled, the parallel new collector is enabled
automatically as soon as the CMS collector is enabled:

  void Arguments::set_cms_and_parnew_gc_flags() {
    assert(
      !UseSerialGC && !UseParallelOldGC && !UseParallelGC,
      "Error");
    assert(UseConcMarkSweepGC, "CMS is expected to be on here");

    // If we are using CMS, we prefer to UseParNewGC,
    // unless explicitly forbidden.
    if (FLAG_IS_DEFAULT(UseParNewGC)) {
      FLAG_SET_ERGO(bool, UseParNewGC, true);
    }

While it's fine to be explicit, the UseParNewGC flag is deprecatd in JDK
8 and produces warning messages in JDK 9:

  Java HotSpot(TM) 64-Bit Server VM warning: Option UseParNewGC was
  deprecated in version 9.0 and will likely be removed in a future
  release.

Thus, we can and should just remove this flag from the default JVM
options.

Relates #18767
2016-06-07 12:05:38 -04:00
Stefan Scherer 92c6d78d4a Use java from path if JAVA_HOME is not set
This commit adds support to the Windows scripts for finding java.exe via
the path if JAVA_HOME is not set.

Relates #18685
2016-06-03 15:30:18 -04:00
Jason Tedor 8e2a7d0fe1 Rename boostrap.mlockall to bootstrap.memory_lock
The setting bootstrap.mlockall is useful on both POSIX-like systems
(POSIX mlockall) and Windows (Win32 VirtualLock). But mlockall is really
a POSIX only thing so the name should not be tied POSIX. This commit
renames the setting to "bootstrap.memory_lock".

Relates #18669
2016-06-01 16:25:51 -04:00
Ryan Ernst 4735b5a5fa Build: Use license, notice and readme files from root for distributions
The distributions had their own copies of these extra files, which was a
carry over from maven. This change removes the duplicate files and
copies them from the root of the project.

closes #18597
2016-05-26 09:19:38 -07:00
Jason Tedor 452faa220c Filter client/server VM options from jvm.options
The default jvm.options file ships with the -server flag to force the
server VM on systems where the server VM is not the default. However,
the method of starting the JVM via the Windows service does not support
the command-line flags for selecting the VM because it starts from a
DLL that is specific to the server or client VM. Thus, we need to
filter these options from the jvm.options configuration file when
installing the Windows service.

Relates #18473
2016-05-20 15:25:45 -04:00
Jason Tedor c257e2c51f Remove settings and system properties entanglement
Today when parsing settings during bootstrap, we add a system property
for every Elasticsearch setting. Additionally, settings can be set via
system properties. This commit simplifies this situation.
 - settings are no longer propogated to system properties
 - system properties can not be used to set settings
 - the "es." prefix on settings is no longer required (nor permitted)
 - test logging has a dedicated system property (tests.logger.level)

Relates #18198
2016-05-19 14:08:08 -04:00
Jason Tedor db4809d906 Remove last vestigates of /bin/sh shebangs
This commit removes the remaining /bin/sh shebangs in favor of
/bin/bash.

Relates #18448
2016-05-18 11:03:00 -04:00
Jason Tedor 1dfbe8b0f1 Fix handling of default path conf in plugin script
In the plugin script we set the setting default.path.conf to control the
path to the configuration file. But the default directory might not
exist in which case we can run into exceptions. This commit restores a
guard against this directory not existing.
2016-05-10 15:32:23 -04:00
Jason Tedor ef6a7631cf Switch scripts to use bash
This commit switches the command-line scripts to use bash instead of sh
so that we can take advantage of features that bash provides like
arrays.

Relates #18251
2016-05-10 15:06:51 -04:00
Jason Tedor 557fc8b437 Further simplifications of plugin script
In 7d1fd17172 the parsing of command-line
properties in the plugin script was removed. That commit missed
additional parsing of the properties for es.default.path.conf. This
commit removes that parsing and also replaces the use of eval with exec.

Relates #18239
2016-05-10 10:08:58 -04:00
Jason Tedor 7d1fd17172 Remove plugin script parsing of system properties
The plugin script parses command-line options looking for Java system
properties and extracts these arguments to pass to the java command when
starting the JVM. Since elasticsearch-plugin allows arbitrary user
arguments to the JVM via ES_JAVA_OPTS, this parsing is unnecessary. This
commit removes this unnecessary 

Relates #18207
2016-05-09 13:06:18 -04:00
Jason Tedor 0eaa831f48 Preserve config files from RPM install
This commit modifies the packaging for the RPM package so that edits to
config files will not get lost during removal and upgrade.

Relates #18188
2016-05-06 13:24:54 -04:00
Radovan Ondas 443976d03c Fix typo in message for variable setup ES_MAX_MEM
Small typo fix in startup script.

Relates #18168
2016-05-05 19:17:31 -04:00
Jason Tedor f1fb6a37c0 Fix error message on ES_MAX_MEM set
This commit fixes a typo in the error message when the now unsupported
environment variable ES_MAX_MEM is set on Windows.
2016-05-05 19:16:57 -04:00
Jason Tedor e11b96ca9c Default to server VM and add client VM check
Today we softly warn about running with the client VM. However, we
should really refuse to start in production mode if running with the
client VM as the performance of the client VM is too devastating for a
server application. This commit adds an option to jvm.options to ensure
that we are starting with the server VM (on all 32-bit non-Windows
platforms on server-class machines (2+ CPUs, 2+ GB physical RAM) this is
the default and on all 64-bit platforms this is the only option) and
adds a bootstrap check for the client VM.

Relates #18155
2016-05-05 10:36:21 -04:00
Jason Tedor 124e8e5a6d Remove client option for JVM for plugins script
Today we specify the client option for the JVM when executing plugin
commands. Yet, this option does nothing on a 64-bit capable JDK as such
JDKs always select the Hotspot server VM. And for 32-bit JDKs, running
plugin commands with the server VM is okay. Thus, we should just remove
this unnecessary flag and just let the default VM be selected.

Relates #18142
2016-05-04 14:11:29 -04:00
Jason Tedor 4c6cf7ee88 Pass ES_JAVA_OPTS to JVM for plugins script
This commit adds support for ES_JAVA_OPTS to the elasticsearch-plugin
script.

Relates #18140
2016-05-04 12:48:39 -04:00
Lee Hinman 4fca5f734a Explicitly set packaging permissions
This changes our packaging to be explicit about the permissions of files
and directories in the tar.gz, rpm, and deb packages. This is to protect
against a user having an incorrectly set umask when installing.

Additionally, plugins that are installed now have their permissions set
by the plugin installation so that plugins that may have been packaged
with incorrect permissions are secured.

Resolves #17634
2016-04-21 12:30:56 -06:00
Lee Hinman b8899cdb78 Merge remote-tracking branch 'dakrone/allow-bad-json' 2016-04-19 10:02:53 -06:00
Lee Hinman a1e8fb794c Allow JSON with unquoted field names by enabling system property
In Elasticsearch 5.0.0, by default unquoted field names in JSON will be
rejected. This can cause issues, however, for documents that were
already indexed with unquoted field names. To alleviate this, a system
property has been added that can be enabled so migration can occur.

This system property will be removed in Elasticsearch 6.0.0

Resolves #17674
2016-04-19 09:14:13 -06:00
Danilo Vaz 2e2d8c1442 Updated copyright years to include 2016 (#17808) 2016-04-18 12:39:23 +02:00
Jason Tedor 7297580170 Simplify JVM options
This commit simplifies the default JVM options that ship with
Elasticsearch. In particular, expert settings that were previously
configurable via environment variables have been removed from the
default configuration file. Further, the heap size settings have been
moved to the top of the file with a clearer message that is in
concordance with their importance.

Closes #17714
2016-04-13 17:59:59 -04:00
Jason Tedor f4ccf9bd4f Set variable in Windows scripts to default value
This commit sets the bad_env_var to a default value of 0 so that in case
no bad environment variables are encountered the variable is still
defined and does not later cause a parse error in the execution of these
batch files.

Relates #17675
2016-04-13 12:17:36 -04:00
Nik Everett bf2483e4b3 Setup jvm opts correctly for deb tests
Also stops warning about JAVA_OPTS when it is an empty string.
2016-04-13 11:18:19 -04:00
Jason Tedor 5662a14d22 Rewrite if-statement for shell compatibility
This commit rewrites an if-statement in bin/elasticsearch for
compatibility with some shells (e.g., dash on Ubuntu).
2016-04-12 22:27:08 -04:00
Jason Tedor 27867c729d Additional handling of ES_GC_LOG_FILE
This commit adds handling of the old ES_GC_LOG_FILE environment variable
to treat it like the other removed environment variables.
2016-04-12 22:26:43 -04:00
Jason Tedor bdfa063946 Add migration messages when rejecting env. vars
This commit adds migration messages to the startup scripts when
rejecting the old unsupported environment variables.
2016-04-12 15:32:31 -04:00
Jason Tedor d4cf0f021a Fail if unsupported environment variables are set
This commit adds a hard failure on startup if any unsupported
environment variables from older versions of Elasticsearch are still
set.
2016-04-12 13:29:09 -04:00
Jason Tedor 2b1c191b12 Add comments to JVM options parsing for Windows
This commit adds some comments to the portion of bin/elasticsearch.bat
that is responsible for parsing the JVM options.
2016-04-12 12:37:15 -04:00
Jason Tedor 7743c422db Warn in bin/elasticsearch if JAVA_OPTS is set
JAVA_OPTS is not a built-in mechanism for passing options to the JVM but
many people think that it is. We do not respect JAVA_OPTS, but this
commit adds a warning if it is set in case the end-user thinks that it
will affect Elasticsearch.
2016-04-12 11:51:30 -04:00
Jason Tedor 9460289bee Rename function that parses JVM options
This commit renames the function that parses JVM options in
bin/elasticsearch from jvm_options to parse_jvm_options. The reason for
the rename is because a for-loop variable was shadowing the name of this
function and changing the function name further clarifies the purpose of
the function.
2016-04-12 11:42:06 -04:00
Jason Tedor 00033b3da3 Fix placement of comment in bin/elasticsearch
This commit fixes the placement of a comment in bin/elasticsearch. The
comment was made out of place by the addition of a function definition
but order is restored.
2016-04-12 11:39:33 -04:00
Jason Tedor 3879aa2a98 Add JVM options configuration file
This commit adds a new configuration file jvm.options to centralize and
simplify management of JVM options. This separates the configuration of
the JVM from the packaging scripts (bin/elasticsearch*, bin/service.bat,
and init.d/elasticsearch) simplifying end-user operational management of
custom JVM options.
2016-04-12 11:19:16 -04:00
Daniil Penkin 696fa18c61 Support for configuring Elasticsearch service display name, description and user on Windows 2016-04-07 10:39:57 +10:00
Lee Hinman 6e44ddaf0e Merge branch 'pr/11338' 2016-04-06 15:06:30 -06:00
Lee Hinman 42745c8c53 Merge pull request #14596 from girirajsharma/ES-RPM
rpm uses non-portable `--system` flag to `useradd`
2016-04-06 14:46:08 -06:00
Jason Tedor cea3c3c92f Quote path to java binary
This commit quotes the variable that contains the path to the java
binary. Without these quotes, when the arguments to eval are evaluated
the existing quotes will be removed leading to unquoted use of the path
to the java binary. If this path contains spaces, evaluation will fail.
2016-04-03 23:23:25 -04:00
Joe Hillenbrand 81b6326891 Set MAX_OPEN_FILES to 65536
Relates to #17430
2016-03-30 16:44:37 -07:00
Mpdreamz e8776799cc Service command still had positional start command
Which caused the service to fail to start
2016-03-29 23:25:03 +02:00
Jason Tedor a0634b2442 Merge pull request #17197 from jasontedor/chown
Elasticsearch ownership for data, logs, and configs
2016-03-24 21:21:11 -04:00
Jason Tedor d22b5977de Ownership for data, logs, and configs for packages
This commit ensures that the data, logs, and config directories have the
proper ownership after the packages are installed. Additionally, this
commit ensures that the configs in /etc/elasticsearch are preserved
after removal of the RPM package.
2016-03-24 20:56:02 -04:00
Simon Willnauer 63b754aa22 Remove positional parameter from bat file 2016-03-24 11:19:25 +01:00
Yannick Welsch e32da555aa Fix Windows start script to pass parameters in last position
Relates to #15320
2016-03-16 13:43:32 +01:00
Jason Tedor d2db9cf95f Fix es.path.home on Windows 2016-03-16 04:46:25 -04:00
Jason Tedor 4ee90db13d Remove path.home command-line setting 2016-03-15 16:29:29 -04:00
Jason Tedor 5994e91b08 Fix systemd pidfile setting
This commit fixes the pidfile setting on systems that used systemd. The
issue is that the pidfile can only be set via the command line arguments
-p or --pidfile, and is no longer settable via a setting.
2016-03-15 15:57:37 -04:00
Jason Tedor 8a05c2a2be Bootstrap does not set system properties
Today, certain bootstrap properties are set and read via system
properties. This action-at-distance way of managing these properties is
rather confusing, and completely unnecessary. But another problem exists
with setting these as system properties. Namely, these system properties
are interpreted as Elasticsearch settings, not all of which are
registered. This leads to Elasticsearch failing to startup if any of
these special properties are set. Instead, these properties should be
kept as local as possible, and passed around as method parameters where
needed. This eliminates the action-at-distance way of handling these
properties, and eliminates the need to register these non-setting
properties. This commit does exactly that.

Additionally, today we use the "-D" command line flag to set the
properties, but this is confusing because "-D" is a special flag to the
JVM for setting system properties. This creates confusion because some
"-D" properties should be passed via arguments to the JVM (so via
ES_JAVA_OPTS), and some should be passed as arguments to
Elasticsearch. This commit changes the "-D" flag for Elasticsearch
settings to "-E".
2016-03-13 20:09:15 -04:00
Lee Hinman 22e716551b Add -XX+AlwaysPreTouch JVM flag
Enables the touching of all memory pages used by the JVM heap spaces
during initialization of the HotSpot VM, which commits all memory pages
at initialization time. By default, pages are committed only as they are
needed.
2016-03-10 10:11:32 -07:00