honeymoose
/
OpenSearch
mirror of https://github.com/honeymoose/OpenSearch.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge pull request elastic/elasticsearch#4 from elastic/master 

update from upstream

Original commit: elastic/x-pack-elasticsearch@d8bdb10cd5
This commit is contained in:
Konrad Beiske 2015-09-16 13:01:35 +02:00
parent 28af1e275a a0557e23a6
commit d56df6e04c
486 changed files with 20597 additions and 3569 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
marvel/.settings/org.eclipse.jdt.core.prefs
View File

@ -3,8 +3,8 @@ org.eclipse.jdt.core.compiler.annotation.inheritNullAnnotations=enabled
org.eclipse.jdt.core.compiler.annotation.missingNonNullByDefaultAnnotation=ignore
org.eclipse.jdt.core.compiler.annotation.nullable=org.elasticsearch.common.Nullable
org.eclipse.jdt.core.compiler.annotation.nullanalysis=enabled
org.eclipse.jdt.core.compiler.codegen.targetPlatform=1.7
org.eclipse.jdt.core.compiler.compliance=1.7
org.eclipse.jdt.core.compiler.codegen.targetPlatform=1.8
org.eclipse.jdt.core.compiler.compliance=1.8
org.eclipse.jdt.core.compiler.problem.forbiddenReference=warning
org.eclipse.jdt.core.compiler.problem.nonnullParameterAnnotationDropped=warning
org.eclipse.jdt.core.compiler.problem.nullAnnotationInferenceConflict=warning
@ -12,7 +12,7 @@ org.eclipse.jdt.core.compiler.problem.nullReference=warning
org.eclipse.jdt.core.compiler.problem.nullSpecViolation=warning
org.eclipse.jdt.core.compiler.problem.nullUncheckedConversion=warning
org.eclipse.jdt.core.compiler.problem.potentialNullReference=warning
org.eclipse.jdt.core.compiler.source=1.7
org.eclipse.jdt.core.compiler.source=1.8
org.eclipse.jdt.core.formatter.lineSplit=140
org.eclipse.jdt.core.formatter.tabulation.char=space
org.eclipse.jdt.core.formatter.tabulation.size=4

172
marvel/docs/configuring-marvel.asciidoc Normal file
View File

@ -0,0 +1,172 @@
[[configuration]]
== Configuring Marvel
[float]
[[stats-export]]
=== Controlling Marvel Data Collection
You can set `marvel.agent` options in a node's `elasticsearch.yml` file to control how Marvel data
is collected from the node.
`marvel.agent.enabled`::
Controls whether or not data is collected from the node. Enabled by default. Set
`marvel.agent.enabled` to `false` to disable data collection. Use to disable data collection
on the monitoring nodes when you use a separate monitoring cluster.
`marvel.agent.exporter.es.hosts`::
Specifies where the collected Marvel data should be stored. Defaults to the address of the
local node (typically `localhost:9200`). To send data to a monitoring cluster,
set to a comma-seprarated list of nodes in `hostname:port` format . Marvel attempts to send data
to the hosts in the order they are listed--if the first node is unreachable, the second node is
tried, and so on. added[1.3.0, before the default was `localhost:9200`]
+
You can update this setting through the Cluster Update Settings API.
+
added[1.0.2] - HTTP Basic authentication credentials can be specified as part of the host name,
i.e., ["user:pwd@host:9200"]
+
added[1.3.0] - next to the host and port, you can specify a protocol to use,
i.e., ["https://host:9200"] (defaults to "http")
+
added[1.3.0] - the `hostname:port` part can be extended with a base path
i.e., ["host:9200/monitor1"]
`marvel.agent.indices`::
Controls which indices Marvel collects data for. Defaults to all indices. Specify the index names
as a comma-separated list, for example `test1,test2,test3`. Names can include wildcards, for
example `test*`. You can explicitly include or exclude indices by prepending
`+` to include the index, or `-` to exclude the index. For example, to include all indices that
start with `test` except `test3`, you could specify `+test*,-test3`.
+
You can update this setting through the Cluster Update Settings API.
`marvel.agent.interval`::
Controls how often data samples are collected from the node. Defaults to `10s`. Set to
`-1` to temporarily disable data collection.
+
You can update this setting through the Cluster Update Settings API.
`marvel.agent.exporter.es.index.timeformat`::
Controls the time component in the index name to which data is exported.
Defaults to `"YYYY.MM.dd"`, which produces index names such as
`.marvel-2015.08.23`. Supports date formats as explained
http://joda-time.sourceforge.net/api-release/org/joda/time/format/DateTimeFormat.html[here].
`marvel.agent.exporter.es.timeout`::
Sets the connection timeout for sending data. Defaults to `6s`.
+
You can update this setting through the Cluster Update Settings API.
`marvel.agent.stats.timeout`::
added[1.3.1]
Sets the timeout when collecting statistics from the master node. Defaults to `10m`.
`marvel.agent.exporter.es.ssl.truststore.path`::
added[1.3.0]
The location of the truststore to use for HTTPS connections. Specified as an absolute path.
`marvel.agent.exporter.es.ssl.truststore.password`::
The password to use to access the truststore.
`marvel.agent.exporter.es.ssl.truststore.algorithm`::
The truststore format. Defaults to SunX509.
`marvel.agent.exporter.es.ssl.hostname_verification`::
+
added[1.3.1]
Controls whether or not the hostname is verified when using HTTPS. Set to `false` to disable
hostname verification when sending data to a remote host.
+
You can update this setting through the Cluster Update Settings API.
[float]
[[marvel-indices]]
=== Configuring Marvel Indices
Marvel stores it's data using time-based indices. By default, Marvel generates
an index per day, with one shard and one replica. We expect this to be a good
default for most use cases. For very large clusters, you might need to make changes
by overriding the settings in the default Marvel index template.
[[config-marvel-indices]]
Marvel uses an {ref}/indices-templates.html[index template] to preconfigure newly created indices.
You can retrieve it with:
[source,sh]
----------------------------------
GET /_template/marvel
----------------------------------
You can override the default settings by adding your own template. Make sure your template uses
the `.marvel-*` matching pattern and has an order of 1 or higher. For example, the following
template increases the number of shards to 5:
[source,json]
----------------------------------
PUT /_template/custom_marvel
{
"template": ".marvel*",
"order": 1,
"settings": {
"number_of_shards": 5
}
}
----------------------------------
IMPORTANT: We recommend only changing the `settings` section. Other sections are
important for the correct operation of the dashboards.
For reference, the following snippet shows the `settings` section of the default template.
[source,json]
----------------------------------
{
"template": ".marvel*",
"settings": {
"number_of_shards": 1,
"number_of_replicas": 1,
"analysis": {
"analyzer": {
"default": {
"type": "standard",
"stopwords": "_none_"
}
}
},
"mapper.dynamic": true,
"marvel.index_format": 1
}
.....
}
----------------------------------
[float]
[[relevant-settings]]
=== Enabling Automatic Index Creation
Marvel relies on Elasticsearch's ability to automatically create new indices
when indexing documents. If you have disabled automatic index creation, you
need to configure the `action.auto_create_index` setting to allow the
creation of Marvel indices:
[source,yaml]
----------------------
action.auto_create_index: .marvel-*
----------------------
For more information see {ref}/docs-index_.html#index-creation[Index Creation] in the Elasticsearch
Reference.

47
marvel/docs/getting-started.asciidoc Normal file
View File

@ -0,0 +1,47 @@
[[getting-started]]
== Getting Started
This getting started guide walks you through installing Marvel
and using the Marvel Kibana app to monitor your Elastisearch cluster.
To install Marvel:
. Install the Marvel plugin on each node in your cluster:
.. Run `bin/plugin install` from `ES_HOME` to install the License plugin:
+
[source,shell]
----------------------------------------------------------
bin/plugin install elasticsearch/license/latest
----------------------------------------------------------
.. Run `bin/plugin install` to install the Marvel plugin.
+
[source,shell]
----------------------------------------------------------
bin/plugin install elasticsearch/marvel/latest
----------------------------------------------------------
+
NOTE: If you are using a <<package-installation, DEB/RPM distribution>> of Elasticsearch, run the installation with superuser permissions. To perform an offline installation, <<offline-installation, download the Marvel binaries>>.
.. Start Elasticsearch.
+
[source,shell]
----------------------------------------------------------
bin/elasticsearch
----------------------------------------------------------
. Install the Marvel app into Kibana.
. Run Kibana and open the Marvel app to verify the installation. You should
see an overview of your cluster's status:
+
// image:images/overview_thumb.png["Overview Dashboard",link="images/overview.png"]
Now you're ready to use Marvel to monitor and analyze your cluster! For example, you can:
*
*

17
marvel/docs/index.asciidoc Normal file
View File

@ -0,0 +1,17 @@
= Marvel Documentation
:ref: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current
:esversion: 2.0
:licenseversion: 2.0
:shieldversion: 2.0
:kibanaversion: 4.2+
include::introduction.asciidoc[]
include::getting-started.asciidoc[]
include::installing-marvel.asciidoc[]
include::configuring-marvel.asciidoc[]
include::release-notes.asciidoc[]

141
marvel/docs/installing-marvel.asciidoc Normal file
View File

@ -0,0 +1,141 @@
[[installing-marvel]]
== Installing Marvel
To use Marvel, you need to install two components:
* An Elasticsearch plugin that collects data from each node in your cluster.
This plugin must be installed on every node.
* A Kibana app that provides the Marvel monitoring and management UI.
By default, the Marvel plugin stores data in the same Elasticsearch cluster
where it is installed. If you are monitoring a production cluster we recommend
that you store the Marvel data in a separate monitoring cluster. Sending the Marvel
data to a monitoring cluster helps ensure that you can continue to monitor your
production cluster if it's in an unhealthy state.
For basic installation instructions, see <<getting-started, Getting Started with Marvel>>. For
information about storing Marvel data in a separate monitoring cluster, see <<monitoring-cluster,
Setting up a Separate Monitoring Cluster>>.
[float]
[[marvel-prequisites]]
=== Marvel Installation Prequisites
* Elasticsearch {esversion} or later.
* Kibana {kibanaversion} or later.
* A modern web browser - http://www.elastic.co/subscriptions/matrix#matrix_browsers[Supported
Browsers].
[float]
[[package-installation]]
=== Installing Marvel on a DEB/RPM Package Installation
If you use the DEB/RPM packages to install Elasticsearch, by default Elasticsearch is installed in
`/usr/share/elasticsearch` and the configuration files are stored in `/etc/elasticsearch`. (For the
complete list of default paths, see {ref}/setup-dir-layout.html#_deb_and_rpm[Directory Layout] in
the Elasticsearch Reference.)
To install the Marvel and License plugins on a DEB/RPM package installation, you need to run
`bin/plugin -i` from the `/usr/share/elasticsearch` directory with superuser permissions, and
specify the location of the configuration files by setting `-Des.path.conf`. For example:
[source,shell]
----------------------------------------------------------
cd /usr/share/elasticsearch
sudo bin/plugin -i elasticsearch/license/latest -Des.path.conf=/etc/elasticsearch
sudo bin/plugin -i elasticsearch/marvel/latest -Des.path.conf=/etc/elasticsearch
----------------------------------------------------------
[float]
[[offline-installation]]
=== Installing Marvel on Offline Machines
Elasticsearch’s `bin/plugin` script requires direct Internet access to download and install the
License and Marvel plugins. If your server doesn’t have Internet access, you can manually
download and install the plugins.
To install Marvel on a machine that doesn't have Internet access:
. Manually download the License and Marvel binaries:
+
** https://download.elastic.co/elasticsearch/license/license-latest.zip[
https://download.elastic.co/elasticsearch/license/license-latest.zip]
** https://download.elastic.co/elasticsearch/marvel/marvel-latest.zip[
https://download.elastic.co/elasticsearch/marvel/marvel-latest.zip]
. Transfer the zip files to the offline machine.
. Run `bin/plugin` with the `-u` option to install the plugins using the zip files. For example:
+
[source,shell]
----------------------------------------------------------
bin/plugin -i license -u file:///path/to/file/license-latest.zip <1>
bin/plugin -i marvel -u file:///path/to/file/marvel-latest.zip
----------------------------------------------------------
<1> Note that you must specify an absolute path to the zip file after the `file://` protocol.
[float]
[[monitoring-cluster]]
=== Setting up a Separate Monitoring Cluster
To store Marvel data in a separate monitoring cluster:
. Set up the Elasticsearch cluster you want to use for monitoring. For example, a two host cluster
with the nodes `es-mon-1` and `es-mon-2`.
. Disable Marvel data collection for the nodes in your monitoring cluster by configuring the
`marvel.agent.enabled` setting in their `elasticsearch.yml` configuration files.
+
[source,yaml]
------------------------
marvel.agent.enabled: false
------------------------
. Install the Marvel and License plugins on the nodes in your monitoring cluster:
+
[source,sh]
----------------
bin/plugin -i elasticsearch/license/latest
bin/plugin -i elasticsearch/marvel/latest
----------------
. Start Elasticsearch on the nodes in your monitoring cluster:
+
[source,shell]
----------------------------------------------------------
bin/elasticsearch
----------------------------------------------------------
. Configure the nodes in your production cluster to send Marvel data to your monitoring cluster by
configuring the `marvel.agent.exporter.es.hosts` setting in their `elasticsearch.yml`
configuration files:
+
[source,yaml]
------------------------
marvel.agent.exporter.es.hosts: ["es-mon-1:9200","es-mon-2:9200"]
------------------------
. Install the Marvel and License plugins on the nodes in your production cluster:
+
[source,sh]
----------------
bin/plugin -i elasticsearch/license/latest
bin/plugin -i elasticsearch/marvel/latest
----------------
. Restart Elasticsearch on the nodes in your production cluster
+
[source,shell]
----------------------------------------------------------
bin/elasticsearch
---------------------------------------------------------
+
TIP: You may want to temporarily {ref}/modules-cluster.html[disable shard
allocation] before you restart your nodes to avoid unnecessary shard
reallocation during the install process.
. Install the Marvel app into Kibana.
. Run Kibana and open the Marvel app to verify the installation. You should
see an overview of your cluster's status:

23
marvel/docs/introduction.asciidoc Normal file
View File

@ -0,0 +1,23 @@
[[introduction]]
== Introduction
_Marvel_ is a plugin for Elasticsearch that enables you to easily monitor and manage
your Elasticsearch cluster. Marvel aggregates cluster-wide statistics and events and provides a
Kibana app that makes it easy to view and analyze the data.
When you open the Marvel app in Kibana, you see the key metrics that indicate the health of your
cluster.
From there, you can dive into the details for particular nodes and indices.
[float]
=== Where to Go Next
* <<getting-started, Getting Started>> steps through how to install and start using Marvel to
monitor Elasticsearch.
[float]
=== Have Comments, Questions, or Feedback?
Head over to our https://discuss.elastic.co/c/marvel[Marvel Discussion Forum] to share your
experience, questions, and suggestions.

252
marvel/docs/release-notes.asciidoc Normal file
View File

@ -0,0 +1,252 @@
[[release-notes]]
== Release Notes
[float]
[[version-compatibility]]
=== Version Compatibility
Marvel {marvelversion} is compatible with:
* Elasticsearch {esversion}
* License {licenseversion}
* Shield {shieldversion}
* Kibana {kibanaversion}
[float]
[[upgrading]]
=== Upgrading Marvel
When upgrading Marvel, you must upgrade *every node in the cluster*. If you're using a monitoring
cluster, upgrade the nodes in the monitoring cluster before upgrading your production cluster. You
do not need to fully shut down your production or monitoring clusters to perform the upgrade, you
can perform a rolling upgrade.
To perform a rolling upgrade of Marvel:
. Disable shard reallocation. While this is optional, it enables a faster startup after cluster
shutdown. If you don't disable shard reallocation, the nodes immediately start trying to
replicate shards to each other on startup and spend a lot of time on wasted I/O. With shard
reallocation disabled, the nodes join the cluster with their indices intact and do not attempt to
rebalance. After startup is complete, you can turn reallocation back on.
+
[source,sh]
--------------------------------------------------
curl -XPUT localhost:9200/_cluster/settings -d '{
"transient" : {
"cluster.routing.allocation.enable" : "none"
}
}'
--------------------------------------------------
. Upgrade each node, one at a time:
.. Stop Elasticsearch.
.. Remove the old version of the Marvel plugin:
+
[source,sh]
--------------------------------------------------
bin/plugin -r marvel
--------------------------------------------------
.. Install the latest version of the Marvel plugin:
+
[source,sh]
--------------------------------------------------
bin/plugin -i elasticsearch/marvel/latest
--------------------------------------------------
.. Start Elasticsearch and confirm that the node rejoins the cluster and that there are no errors
in the logs.
. When you've upgraded all of the nodes in the cluster, reenable shard allocation:
+
[source,sh]
--------------------------------------------------
curl -XPUT localhost:9200/_cluster/settings -d '{
"transient" : {
"cluster.routing.allocation.enable" : "all"
}
}'
--------------------------------------------------
[float]
[[change_list]]
=== Change List
[float]
==== 2.0.0-beta2
Marvel versions are now aligned with Elasticsearch version.
- Agent:
* Added: add license expiration support to stop collecting data 7 days
after license expiration
* Added: add integration tests
* Added: add support to index cluster license information
* Added: add support to collect node statistics
* Added: add support to collect index recoveries
* Added: add support to collect index level statistics
* Added: add support to collect cluster statistics
* Added: add support to collect cluster state
* Improved: update project to work with Elasticsearch 2.x
- Monitoring UI:
* TODO
- Sense:
* Sense has been removed from Marvel starting version 2.x and is now shipped as
a Kibana application
[float]
==== 1.3.1
- Agent:
* Added: add timeouts to better deal with unresponsive ES nodes / hiccups.
* Added: Allow SSL hostname verification to be disabled.
* Fixed: Node failed to start if HTTP is disabled.
* Fixed: Potential NPE if HTTP server didn't start fast enough.
* Fixed: `marvel.agent.indices` wasn't dynamically updatable when using a single value or a
comma separated list.
* Fixed: unused shield SSL settings caused errors during start up.
- Monitoring UI:
* Fixed: Upgraded Kibana3 to latest version, fixing a wrap around issue in Safari.
- Sense:
* Added: Cluster health's level url parameter.
* Added: _recovery API.
* Fixed: trailing space after URL broke request parsing.
* Added: _search_shards API.
[float]
==== 1.3.0
- Agent:
* Added: support for shipping over https.
* Removed: support for optional shard level stats due to an incompatible change in ES 1.4.
* Fixed: an issue causing a tribe node (ES 1.4.0) not to initialize when Marvel is installed.
* Improved: resiliency and error checking around the marvel index template (both checking for it and adding it).
* Improved: logging upon error, supressing repreated logs.
* Added: Automcally detect the local node's port when using not default Marvel settings (previously was always 9200)
* Improved: Change _bulk export command to set the index name in the url param. This is usefull when `rest.action.multi.allow_explicit_index` is set to false.
- Monitoring UI
* Added: charts for new circuit breakers introduce with ES 1.4.0
* Added: a chart to plot circuit break limit.
* Added: a charts for query cache.
* Added: charts for index throttling.
* Added: charts to expose memory usage of the index writer and version map.
* Fixed: Network Transport Bytes Received chart actually shows bytes sent
* Fixed: Node Stats dashboard missed some thread pools.
- Sense:
* Added: a settings to allow disabling mappings and/or indices autocomplete. This is usefull for extremly large deployments where parsing by the browser is unrealistic.
* Added: Custer Reroute API.
* Added: Get Field Mappings API,
* Fixed: Url auto complete failed with completing fully qualified urls (i.e. with protocol and host).
* Added: Query Cache parmaters to the Search API.
* Added: Analyze API.
* Added: Validate Query API.
* Fixed: include_in_parent and include_in_root is missing for nested type mapping.
* Added: Put Percolator API.
* Fixed: Range filter template to use gt, gte, lt and lte.
* Added: cluster.routing.allocation.* settings
* Added: weight param to the Function Score query.
* Added: Flush API.
* Added: show_term_doc_count_error parameter to the Terms Aggregation.
* Added: Update API
* Added: _geo_distance as a sort option.
* Added: Updated the Significant Terms aggregation to 1.4.0 features.
* Added: metadata fields to the Mapping API.
* Added: Get Index API.
* Added: Scripted Metric Aggregation.
* Added: simple_query_string query.
* Added: Updated the More Like This query to 1.4.0 features.
* Added: min_childeren, max_children options to the has_child query and dilter.
* Added: Updated execution hint options in terms and significant terms aggs.
* Added: transform section of Mappings API.
* Added: indexed scripts and templates.
* Added: Geo Bounds aggregation.
* Added: Top Hits aggregation.
* Added: collect_mode option the Terms aggregation.
* Added: Percentiles Rank aggregation.
* Added: Disk Threshold Allocator settings.
* Fixed: Exists filter auto complete.
* Fixed: Snapshot and Restore API failed to autocomplete repository settings.
[float]
==== 1.2.1
- Fix a cluster state data shipping for cluster states larger than 16K (in `SMILE` format and without meta data).
[float]
==== 1.2.0
- New Shard Allocation Dashboard.
- Simplified navigation and dashboard customization.
- Sense:
* Update the KB to the ES 1.2.0 API, adding the following:
* `_cat/plugins`
* `_cat/segments`
* `_search/template`
* `_count`
* `_snapshot`
* Alias support for index creation.
* Significant terms aggregation.
* Percentiles aggregation.
* Cardinality aggregation.
* Time_zone keyword to date_histogram facet/aggregation.
* Removed deprecated `custom_score` & `custom_boost_factor` from the 1.0 API.
* Fixed a bug causing the query panel to loose focus after running a command.
- Charts and Dashboards changes:
* Added an information icon next to the status information of Cluster Summary panel. Hovering on it will show a
short explanation of current status.
* The indices stats table in the Overview dashboard now shows an information icon next to red and yellow indices.
Hovering on it will show a short shard level summary.
* Marvel's index template will now be automatically updated upon upgrade.
* Added field data & filter cache eviction charts to Node Stats dashboard and Index Stats dashboard.
* Added field data circuit breaker charts to Node Stats dashboard.
* Added a registration & purchasing form.
* Hidden indexes are now shown by default.
* Default cluster pulse default time span to 7 days.
* Fixed: Split brain detection algorithm didn't fire in some configurations.
- `marvel.agent.exporter.es.host` configuration option now defaults to port 9200.
[float]
==== 1.1.1
- Fixed: agent did not interpret timeout settings correctly, causing potential connectivity errors when shipping data.
[float]
==== 1.1.0
- Improved Sense's autocomplete suggestions:
* Added Snapshot & Restore
* Added Aggregations
* Added support for url query string parameters
* Updated for breaking changes in Elasticsearch 1.0
- Updated welcome splash screen.
- Sense now uses the last used server when opened (previously used the hostname used to access it).
- The agent's keep-alive thread is now stopped upon errors to reduce log noise. It will be restarted
upon successful connection.
- Improved error reporting for failures of items in the agent's bulk requests.
- Index Statistics Dashboard: Indexing Rate Primaries chart was based on the wrong field.
- Introduced `marvel.agent.shard_stats.enabled` to control exporting of shard level statistics. Defaults to `false`.
- Changed agent's default sampling rate to 10s (was 5s)
- Added a visual indication for the master node at the Nodes section of the Overview Dashboard
- Node and Indices tables visually indicate stale data
- Added error reporting to nodes and indices tables
- Made the following agent settings changeable via the Cluster Update Settings API:
* marvel.agent.interval (also supports setting to -1 to disable exporting)
* marvel.agent.exporter.es.hosts
* marvel.agent.exporter.es.timeout
* marvel.agent.shard_stats.enabled
[float]
==== 1.0.2
- Kibana uses `window.location.protocol` (http or https) to make ES calls.
- Added support for basic authentication when sending data from agent. See <<configuration>>.
- Reduced DEBUG logging verbosity.
[float]
==== 1.0.1
- fixed an issue with usage statistics report.
- improve logging message when running on old Elasticsearch versions.

4
marvel/pom.xml
View File

@ -12,7 +12,7 @@
<parent>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>x-plugins</artifactId>
<version>2.1.0-SNAPSHOT</version>
<version>3.0.0-SNAPSHOT</version>
</parent>
<properties>
@ -22,6 +22,8 @@
<elasticsearch.integ.antfile>dev-tools/integration-tests.xml</elasticsearch.integ.antfile>
<tests.rest.suite>marvel</tests.rest.suite>
<tests.rest.load_packaged>false</tests.rest.load_packaged>
<deploy.skip>true</deploy.skip>
<xlint.options>-Xlint:-rawtypes,-unchecked</xlint.options>
</properties>
<dependencies>

8
marvel/src/main/java/org/elasticsearch/marvel/MarvelPlugin.java
View File

@ -5,7 +5,6 @@
*/
package org.elasticsearch.marvel;
import com.google.common.collect.ImmutableList;
import org.elasticsearch.client.Client;
import org.elasticsearch.cluster.ClusterModule;
import org.elasticsearch.cluster.settings.Validator;
@ -20,9 +19,9 @@ import org.elasticsearch.marvel.agent.exporter.ExporterModule;
import org.elasticsearch.marvel.agent.exporter.HttpESExporter;
import org.elasticsearch.marvel.agent.renderer.RendererModule;
import org.elasticsearch.marvel.agent.settings.MarvelModule;
import org.elasticsearch.marvel.license.LicenseModule;
import org.elasticsearch.marvel.agent.settings.MarvelSetting;
import org.elasticsearch.marvel.agent.settings.MarvelSettings;
import org.elasticsearch.marvel.license.LicenseModule;
import org.elasticsearch.marvel.license.LicenseService;
import org.elasticsearch.plugins.Plugin;
import org.elasticsearch.tribe.TribeService;
@ -74,9 +73,9 @@ public class MarvelPlugin extends Plugin {
@Override
public Collection<Class<? extends LifecycleComponent>> nodeServices() {
if (!enabled) {
return ImmutableList.of();
return Collections.emptyList();
}
return ImmutableList.<Class<? extends LifecycleComponent>>of(LicenseService.class, AgentService.class);
return Arrays.<Class<? extends LifecycleComponent>>asList(LicenseService.class, AgentService.class);
}
public static boolean marvelEnabled(Settings settings) {
@ -95,6 +94,7 @@ public class MarvelPlugin extends Plugin {
public void onModule(ClusterModule module) {
// HttpESExporter
module.registerClusterDynamicSetting(HttpESExporter.SETTINGS_INDEX_TIME_FORMAT, Validator.EMPTY);
module.registerClusterDynamicSetting(HttpESExporter.SETTINGS_HOSTS, Validator.EMPTY);
module.registerClusterDynamicSetting(HttpESExporter.SETTINGS_HOSTS + ".*", Validator.EMPTY);
module.registerClusterDynamicSetting(HttpESExporter.SETTINGS_TIMEOUT, Validator.EMPTY);

4
marvel/src/main/java/org/elasticsearch/marvel/agent/AgentService.java
View File

@ -13,7 +13,7 @@ import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.common.util.CollectionUtils;
import org.elasticsearch.common.util.concurrent.EsExecutors;
import org.elasticsearch.marvel.agent.collector.Collector;
import org.elasticsearch.marvel.agent.collector.licenses.LicensesCollector;
import org.elasticsearch.marvel.agent.collector.cluster.ClusterInfoCollector;
import org.elasticsearch.marvel.agent.exporter.Exporter;
import org.elasticsearch.marvel.agent.exporter.MarvelDoc;
import org.elasticsearch.marvel.agent.settings.MarvelSettings;
@ -58,7 +58,7 @@ public class AgentService extends AbstractLifecycleComponent<AgentService> imple
for (Collector collector : collectors) {
if (Regex.simpleMatch(filters, collector.name().toLowerCase(Locale.ROOT))) {
list.add(collector);
} else if (collector instanceof LicensesCollector) {
} else if (collector instanceof ClusterInfoCollector) {
list.add(collector);
}
}

6
marvel/src/main/java/org/elasticsearch/marvel/agent/collector/CollectorModule.java
View File

@ -7,11 +7,12 @@ package org.elasticsearch.marvel.agent.collector;
import org.elasticsearch.common.inject.AbstractModule;
import org.elasticsearch.common.inject.multibindings.Multibinder;
import org.elasticsearch.marvel.agent.collector.cluster.ClusterInfoCollector;
import org.elasticsearch.marvel.agent.collector.cluster.ClusterStateCollector;
import org.elasticsearch.marvel.agent.collector.cluster.ClusterStatsCollector;
import org.elasticsearch.marvel.agent.collector.indices.IndexRecoveryCollector;
import org.elasticsearch.marvel.agent.collector.indices.IndexStatsCollector;
import org.elasticsearch.marvel.agent.collector.licenses.LicensesCollector;
import org.elasticsearch.marvel.agent.collector.indices.IndicesStatsCollector;
import org.elasticsearch.marvel.agent.collector.node.NodeStatsCollector;
import java.util.HashSet;
@ -23,7 +24,8 @@ public class CollectorModule extends AbstractModule {
public CollectorModule() {
// Registers default collectors
registerCollector(LicensesCollector.class);
registerCollector(ClusterInfoCollector.class);
registerCollector(IndicesStatsCollector.class);
registerCollector(IndexStatsCollector.class);
registerCollector(ClusterStatsCollector.class);
registerCollector(ClusterStateCollector.class);

72
marvel/src/main/java/org/elasticsearch/marvel/agent/collector/cluster/ClusterInfoCollector.java Normal file
View File

@ -0,0 +1,72 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.marvel.agent.collector.cluster;
import org.elasticsearch.Version;
import org.elasticsearch.action.admin.cluster.stats.ClusterStatsResponse;
import org.elasticsearch.client.Client;
import org.elasticsearch.cluster.ClusterName;
import org.elasticsearch.cluster.ClusterService;
import org.elasticsearch.common.inject.Inject;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.license.core.License;
import org.elasticsearch.marvel.agent.collector.AbstractCollector;
import org.elasticsearch.marvel.agent.exporter.MarvelDoc;
import org.elasticsearch.marvel.agent.settings.MarvelSettings;
import org.elasticsearch.marvel.license.LicenseService;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
/**
* Collector for registered licenses and additional cluster information.
* <p/>
* This collector runs on the master node and collect data about all
* known licenses that are currently registered. It also ships some stats
* about the cluster (to be used in Phone Home feature).
*/
public class ClusterInfoCollector extends AbstractCollector<ClusterInfoMarvelDoc> {
public static final String NAME = "cluster-info-collector";
public static final String TYPE = "cluster_info";
private final ClusterName clusterName;
private final LicenseService licenseService;
private final Client client;
@Inject
public ClusterInfoCollector(Settings settings, ClusterService clusterService, MarvelSettings marvelSettings, LicenseService licenseService,
ClusterName clusterName, Client client) {
super(settings, NAME, clusterService, marvelSettings, licenseService);
this.clusterName = clusterName;
this.licenseService = licenseService;
this.client = client;
}
@Override
protected boolean canCollect() {
// This collector can always collect data on the master node
return isLocalNodeMaster();
}
@Override
protected Collection<MarvelDoc> doCollect() throws Exception {
List<MarvelDoc> results = new ArrayList<>(1);
// Retrieves all licenses
List<License> licenses = licenseService.licenses();
// Retrieves additional cluster stats
ClusterStatsResponse clusterStats = client.admin().cluster().prepareClusterStats().get(marvelSettings.clusterStatsTimeout());
String clusterUUID = clusterUUID();
results.add(new ClusterInfoMarvelDoc(MarvelSettings.MARVEL_DATA_INDEX_NAME, TYPE, clusterUUID, clusterUUID, System.currentTimeMillis(),
clusterName.value(), Version.CURRENT.toString(), licenses, clusterStats));
return Collections.unmodifiableCollection(results);
}
}

14
marvel/src/main/java/org/elasticsearch/marvel/agent/collector/licenses/LicensesMarvelDoc.java → marvel/src/main/java/org/elasticsearch/marvel/agent/collector/cluster/ClusterInfoMarvelDoc.java
View File

@ -3,25 +3,28 @@
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.marvel.agent.collector.licenses;
package org.elasticsearch.marvel.agent.collector.cluster;
import org.elasticsearch.action.admin.cluster.stats.ClusterStatsResponse;
import org.elasticsearch.license.core.License;
import org.elasticsearch.marvel.agent.exporter.MarvelDoc;
import java.util.List;
public class LicensesMarvelDoc extends MarvelDoc {
public class ClusterInfoMarvelDoc extends MarvelDoc {
private final String clusterName;
private final String version;
private final List<License> licenses;
private final ClusterStatsResponse clusterStats;
LicensesMarvelDoc(String index, String type, String id, String clusterUUID, long timestamp,
String clusterName, String version, List<License> licenses) {
ClusterInfoMarvelDoc(String index, String type, String id, String clusterUUID, long timestamp,
String clusterName, String version, List<License> licenses, ClusterStatsResponse clusterStats) {
super(index, type, id, clusterUUID, timestamp);
this.clusterName = clusterName;
this.version = version;
this.licenses = licenses;
this.clusterStats = clusterStats;
}
public String getClusterName() {
@ -36,4 +39,7 @@ public class LicensesMarvelDoc extends MarvelDoc {
return licenses;
}
public ClusterStatsResponse getClusterStats() {
return clusterStats;
}
}

8
marvel/src/main/java/org/elasticsearch/marvel/agent/collector/cluster/ClusterStateCollector.java
View File

@ -5,7 +5,6 @@
*/
package org.elasticsearch.marvel.agent.collector.cluster;
import com.google.common.collect.ImmutableList;
import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
import org.elasticsearch.client.Client;
import org.elasticsearch.cluster.ClusterService;
@ -17,7 +16,10 @@ import org.elasticsearch.marvel.agent.exporter.MarvelDoc;
import org.elasticsearch.marvel.agent.settings.MarvelSettings;
import org.elasticsearch.marvel.license.LicenseService;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
/**
* Collector for cluster state.
@ -46,12 +48,12 @@ public class ClusterStateCollector extends AbstractCollector<ClusterStateCollect
@Override
protected Collection<MarvelDoc> doCollect() throws Exception {
ImmutableList.Builder<MarvelDoc> results = ImmutableList.builder();
List<MarvelDoc> results = new ArrayList<>(1);
ClusterState clusterState = clusterService.state();
ClusterHealthResponse clusterHealth = client.admin().cluster().prepareHealth().get(marvelSettings.clusterStateTimeout());
results.add(new ClusterStateMarvelDoc(clusterUUID(), TYPE, System.currentTimeMillis(), clusterState, clusterHealth.getStatus()));
return results.build();
return Collections.unmodifiableCollection(results);
}
}

8
marvel/src/main/java/org/elasticsearch/marvel/agent/collector/cluster/ClusterStatsCollector.java
View File

@ -5,7 +5,6 @@
*/
package org.elasticsearch.marvel.agent.collector.cluster;
import com.google.common.collect.ImmutableList;
import org.elasticsearch.action.admin.cluster.stats.ClusterStatsResponse;
import org.elasticsearch.client.Client;
import org.elasticsearch.cluster.ClusterService;
@ -16,7 +15,10 @@ import org.elasticsearch.marvel.agent.exporter.MarvelDoc;
import org.elasticsearch.marvel.agent.settings.MarvelSettings;
import org.elasticsearch.marvel.license.LicenseService;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
/**
* Collector for cluster stats.
@ -45,10 +47,10 @@ public class ClusterStatsCollector extends AbstractCollector<ClusterStatsCollect
@Override
protected Collection<MarvelDoc> doCollect() throws Exception {
ImmutableList.Builder<MarvelDoc> results = ImmutableList.builder();
List<MarvelDoc> results = new ArrayList<>(1);
ClusterStatsResponse clusterStatsResponse = client.admin().cluster().prepareClusterStats().get(marvelSettings.clusterStatsTimeout());
results.add(new ClusterStatsMarvelDoc(clusterUUID(), TYPE, System.currentTimeMillis(), clusterStatsResponse));
return results.build();
return Collections.unmodifiableCollection(results);
}
}

8
marvel/src/main/java/org/elasticsearch/marvel/agent/collector/indices/IndexRecoveryCollector.java
View File

@ -5,7 +5,6 @@
*/
package org.elasticsearch.marvel.agent.collector.indices;
import com.google.common.collect.ImmutableList;
import org.elasticsearch.action.admin.indices.recovery.RecoveryResponse;
import org.elasticsearch.action.support.IndicesOptions;
import org.elasticsearch.client.Client;
@ -17,7 +16,10 @@ import org.elasticsearch.marvel.agent.exporter.MarvelDoc;
import org.elasticsearch.marvel.agent.settings.MarvelSettings;
import org.elasticsearch.marvel.license.LicenseService;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
/**
* Collector for the Recovery API.
@ -46,7 +48,7 @@ public class IndexRecoveryCollector extends AbstractCollector<IndexRecoveryColle
@Override
protected Collection<MarvelDoc> doCollect() throws Exception {
ImmutableList.Builder<MarvelDoc> results = ImmutableList.builder();
List<MarvelDoc> results = new ArrayList<>(1);
RecoveryResponse recoveryResponse = client.admin().indices().prepareRecoveries()
.setIndices(marvelSettings.indices())
@ -57,6 +59,6 @@ public class IndexRecoveryCollector extends AbstractCollector<IndexRecoveryColle
if (recoveryResponse.hasRecoveries()) {
results.add(new IndexRecoveryMarvelDoc(clusterUUID(), TYPE, System.currentTimeMillis(), recoveryResponse));
}
return results.build();
return Collections.unmodifiableCollection(results);
}
}

8
marvel/src/main/java/org/elasticsearch/marvel/agent/collector/indices/IndexStatsCollector.java
View File

@ -5,7 +5,6 @@
*/
package org.elasticsearch.marvel.agent.collector.indices;
import com.google.common.collect.ImmutableList;
import org.elasticsearch.action.admin.indices.stats.IndexStats;
import org.elasticsearch.action.admin.indices.stats.IndicesStatsResponse;
import org.elasticsearch.action.support.IndicesOptions;
@ -18,7 +17,10 @@ import org.elasticsearch.marvel.agent.exporter.MarvelDoc;
import org.elasticsearch.marvel.agent.settings.MarvelSettings;
import org.elasticsearch.marvel.license.LicenseService;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
/**
* Collector for indices statistics.
@ -47,7 +49,7 @@ public class IndexStatsCollector extends AbstractCollector<IndexStatsCollector>
@Override
protected Collection<MarvelDoc> doCollect() throws Exception {
ImmutableList.Builder<MarvelDoc> results = ImmutableList.builder();
List<MarvelDoc> results = new ArrayList<>(1);
IndicesStatsResponse indicesStats = client.admin().indices().prepareStats()
.setRefresh(true)
@ -60,6 +62,6 @@ public class IndexStatsCollector extends AbstractCollector<IndexStatsCollector>
for (IndexStats indexStats : indicesStats.getIndices().values()) {
results.add(new IndexStatsMarvelDoc(clusterUUID, TYPE, timestamp, indexStats));
}
return results.build();
return Collections.unmodifiableCollection(results);
}
}

54
marvel/src/main/java/org/elasticsearch/marvel/agent/collector/indices/IndicesStatsCollector.java Normal file
View File

@ -0,0 +1,54 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.marvel.agent.collector.indices;
import org.elasticsearch.action.admin.indices.stats.IndicesStatsResponse;
import org.elasticsearch.client.Client;
import org.elasticsearch.cluster.ClusterService;
import org.elasticsearch.common.inject.Inject;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.marvel.agent.collector.AbstractCollector;
import org.elasticsearch.marvel.agent.exporter.MarvelDoc;
import org.elasticsearch.marvel.agent.settings.MarvelSettings;
import org.elasticsearch.marvel.license.LicenseService;
import java.util.Collection;
import java.util.Collections;
/**
* Collector for indices statistics.
* <p/>
* This collector runs on the master node only and collect one {@link IndicesStatsMarvelDoc} document.
*/
public class IndicesStatsCollector extends AbstractCollector<IndicesStatsCollector> {
public static final String NAME = "indices-stats-collector";
public static final String TYPE = "marvel_indices_stats";
private final Client client;
@Inject
public IndicesStatsCollector(Settings settings, ClusterService clusterService, MarvelSettings marvelSettings, LicenseService licenseService,
Client client) {
super(settings, NAME, clusterService, marvelSettings, licenseService);
this.client = client;
}
@Override
protected boolean canCollect() {
return super.canCollect() && isLocalNodeMaster();
}
@Override
protected Collection<MarvelDoc> doCollect() throws Exception {
IndicesStatsResponse indicesStats = client.admin().indices().prepareStats()
.setRefresh(true)
.get(marvelSettings.indicesStatsTimeout());
MarvelDoc result = new IndicesStatsMarvelDoc(clusterUUID(), TYPE, System.currentTimeMillis(), indicesStats);
return Collections.singletonList(result);
}
}

23
marvel/src/main/java/org/elasticsearch/marvel/agent/collector/indices/IndicesStatsMarvelDoc.java Normal file
View File

@ -0,0 +1,23 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.marvel.agent.collector.indices;
import org.elasticsearch.action.admin.indices.stats.IndicesStatsResponse;
import org.elasticsearch.marvel.agent.exporter.MarvelDoc;
public class IndicesStatsMarvelDoc extends MarvelDoc {
private final IndicesStatsResponse indicesStats;
public IndicesStatsMarvelDoc(String clusterUUID, String type, long timestamp, IndicesStatsResponse indicesStats) {
super(clusterUUID, type, timestamp);
this.indicesStats = indicesStats;
}
public IndicesStatsResponse getIndicesStats() {
return indicesStats;
}
}

64
marvel/src/main/java/org/elasticsearch/marvel/agent/collector/licenses/LicensesCollector.java
View File

@ -1,64 +0,0 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.marvel.agent.collector.licenses;
import com.google.common.collect.ImmutableList;
import org.elasticsearch.Version;
import org.elasticsearch.cluster.ClusterName;
import org.elasticsearch.cluster.ClusterService;
import org.elasticsearch.common.inject.Inject;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.license.core.License;
import org.elasticsearch.marvel.agent.collector.AbstractCollector;
import org.elasticsearch.marvel.agent.exporter.MarvelDoc;
import org.elasticsearch.marvel.agent.settings.MarvelSettings;
import org.elasticsearch.marvel.license.LicenseService;
import java.util.Collection;
import java.util.List;
/**
* Collector for registered licenses.
* <p/>
* This collector runs on the master node and collect data about all
* known licenses that are currently registered. Each license is
* collected as a {@link LicensesMarvelDoc} document.
*/
public class LicensesCollector extends AbstractCollector<LicensesMarvelDoc> {
public static final String NAME = "licenses-collector";
public static final String TYPE = "cluster_licenses";
private final ClusterName clusterName;
private final LicenseService licenseService;
@Inject
public LicensesCollector(Settings settings, ClusterService clusterService, MarvelSettings marvelSettings, LicenseService licenseService,
ClusterName clusterName) {
super(settings, NAME, clusterService, marvelSettings, licenseService);
this.clusterName = clusterName;
this.licenseService = licenseService;
}
@Override
protected boolean canCollect() {
// This collector can always collect data on the master node
return isLocalNodeMaster();
}
@Override
protected Collection<MarvelDoc> doCollect() throws Exception {
ImmutableList.Builder<MarvelDoc> results = ImmutableList.builder();
List<License> licenses = licenseService.licenses();
if (licenses != null) {
String clusterUUID = clusterUUID();
results.add(new LicensesMarvelDoc(MarvelSettings.MARVEL_DATA_INDEX_NAME, TYPE, clusterUUID, clusterUUID, System.currentTimeMillis(),
clusterName.value(), Version.CURRENT.toString(), licenses));
}
return results.build();
}
}

8
marvel/src/main/java/org/elasticsearch/marvel/agent/collector/node/NodeStatsCollector.java
View File

@ -6,7 +6,6 @@
package org.elasticsearch.marvel.agent.collector.node;
import com.google.common.collect.ImmutableList;
import org.elasticsearch.action.admin.cluster.node.stats.NodeStats;
import org.elasticsearch.bootstrap.BootstrapInfo;
import org.elasticsearch.cluster.ClusterService;
@ -23,7 +22,10 @@ import org.elasticsearch.marvel.agent.settings.MarvelSettings;
import org.elasticsearch.marvel.license.LicenseService;
import org.elasticsearch.node.service.NodeService;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
/**
* Collector for nodes statistics.
@ -55,7 +57,7 @@ public class NodeStatsCollector extends AbstractCollector<NodeStatsCollector> {
@Override
protected Collection<MarvelDoc> doCollect() throws Exception {
ImmutableList.Builder<MarvelDoc> results = ImmutableList.builder();
List<MarvelDoc> results = new ArrayList<>(1);
NodeStats nodeStats = nodeService.stats();
@ -75,6 +77,6 @@ public class NodeStatsCollector extends AbstractCollector<NodeStatsCollector> {
discoveryService.localNode().id(), isLocalNodeMaster(), nodeStats,
BootstrapInfo.isMemoryLocked(), diskThresholdWatermarkHigh, diskThresholdDeciderEnabled));
return results.build();
return Collections.unmodifiableCollection(results);
}
}

50
marvel/src/main/java/org/elasticsearch/marvel/agent/exporter/HttpESExporter.java
View File

@ -5,7 +5,6 @@
*/
package org.elasticsearch.marvel.agent.exporter;
import com.google.common.io.ByteStreams;
import org.elasticsearch.ExceptionsHelper;
import org.elasticsearch.cluster.ClusterName;
import org.elasticsearch.cluster.ClusterService;
@ -27,6 +26,7 @@ import org.elasticsearch.env.Environment;
import org.elasticsearch.http.HttpServer;
import org.elasticsearch.marvel.agent.renderer.Renderer;
import org.elasticsearch.marvel.agent.renderer.RendererRegistry;
import org.elasticsearch.marvel.agent.settings.MarvelSettings;
import org.elasticsearch.marvel.agent.support.AgentUtils;
import org.elasticsearch.node.Node;
import org.elasticsearch.node.service.NodeService;
@ -35,10 +35,7 @@ import org.joda.time.format.DateTimeFormat;
import org.joda.time.format.DateTimeFormatter;
import javax.net.ssl.*;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStream;
import java.io.OutputStream;
import java.io.*;
import java.net.HttpURLConnection;
import java.net.MalformedURLException;
import java.net.URISyntaxException;
@ -56,7 +53,6 @@ public class HttpESExporter extends AbstractExporter<HttpESExporter> implements
private static final String SETTINGS_PREFIX = "marvel.agent.exporter.es.";
public static final String SETTINGS_HOSTS = SETTINGS_PREFIX + "hosts";
public static final String SETTINGS_INDEX_PREFIX = SETTINGS_PREFIX + "index.prefix";
public static final String SETTINGS_INDEX_TIME_FORMAT = SETTINGS_PREFIX + "index.timeformat";
public static final String SETTINGS_TIMEOUT = SETTINGS_PREFIX + "timeout";
public static final String SETTINGS_READ_TIMEOUT = SETTINGS_PREFIX + "read_timeout";
@ -67,10 +63,11 @@ public class HttpESExporter extends AbstractExporter<HttpESExporter> implements
// es level timeout used for bulk indexing (used to speed up tests)
public static final String SETTINGS_BULK_TIMEOUT = SETTINGS_PREFIX + ".bulk.timeout";
public static final String DEFAULT_INDEX_TIME_FORMAT = "YYYY.MM.dd";
volatile String[] hosts;
volatile boolean boundToLocalNode = false;
final String indexPrefix;
final DateTimeFormatter indexTimeFormatter;
volatile DateTimeFormatter indexTimeFormatter;
volatile int timeoutInMillis;
volatile int readTimeoutInMillis;
@ -118,15 +115,19 @@ public class HttpESExporter extends AbstractExporter<HttpESExporter> implements
validateHosts(hosts);
indexPrefix = settings.get(SETTINGS_INDEX_PREFIX, ".marvel");
String indexTimeFormat = settings.get(SETTINGS_INDEX_TIME_FORMAT, "YYYY.MM.dd");
indexTimeFormatter = DateTimeFormat.forPattern(indexTimeFormat).withZoneUTC();
String indexTimeFormat = settings.get(SETTINGS_INDEX_TIME_FORMAT, DEFAULT_INDEX_TIME_FORMAT);
try {
logger.debug("checking that index time format [{}] is correct", indexTimeFormat);
indexTimeFormatter = DateTimeFormat.forPattern(indexTimeFormat).withZoneUTC();
} catch (IllegalArgumentException e) {
throw new IllegalArgumentException("Invalid marvel index time format [" + indexTimeFormat + "] configured in setting [" + SETTINGS_INDEX_TIME_FORMAT + "]", e);
}
timeoutInMillis = (int) settings.getAsTime(SETTINGS_TIMEOUT, new TimeValue(6000)).millis();
readTimeoutInMillis = (int) settings.getAsTime(SETTINGS_READ_TIMEOUT, new TimeValue(timeoutInMillis * 10)).millis();
templateCheckTimeout = settings.getAsTime(SETTINGS_CHECK_TEMPLATE_TIMEOUT, null);
bulkTimeout = settings.getAsTime(SETTINGS_CHECK_TEMPLATE_TIMEOUT, null);
bulkTimeout = settings.getAsTime(SETTINGS_BULK_TIMEOUT, null);
keepAliveWorker = new ConnectionKeepAliveWorker();
nodeSettingsService.addListener(this);
@ -140,7 +141,7 @@ public class HttpESExporter extends AbstractExporter<HttpESExporter> implements
hostnameVerification = settings.getAsBoolean(SETTINGS_SSL_HOSTNAME_VERIFICATION, true);
logger.debug("initialized with targets: {}, index prefix [{}], index time format [{}]",
AgentUtils.santizeUrlPwds(Strings.arrayToCommaDelimitedString(hosts)), indexPrefix, indexTimeFormat);
AgentUtils.santizeUrlPwds(Strings.arrayToCommaDelimitedString(hosts)), MarvelSettings.MARVEL_INDICES_PREFIX, indexTimeFormat);
}
static private void validateHosts(String[] hosts) {
@ -314,8 +315,8 @@ public class HttpESExporter extends AbstractExporter<HttpESExporter> implements
return hosts;
}
private String getIndexName() {
return indexPrefix + "-" + indexTimeFormatter.print(System.currentTimeMillis());
String getIndexName() {
return MarvelSettings.MARVEL_INDICES_PREFIX + indexTimeFormatter.print(System.currentTimeMillis());
}
@ -359,7 +360,7 @@ public class HttpESExporter extends AbstractExporter<HttpESExporter> implements
return null;
}
hosts = extractedHosts;
logger.trace("auto-resolved hosts to {}", extractedHosts);
logger.trace("auto-resolved hosts to {}", (Object)extractedHosts);
boundToLocalNode = true;
}
@ -462,7 +463,9 @@ public class HttpESExporter extends AbstractExporter<HttpESExporter> implements
private boolean checkAndUploadIndexTemplate(final String host) {
byte[] template;
try (InputStream is = getClass().getResourceAsStream("/marvel_index_template.json")) {
template = ByteStreams.toByteArray(is);
ByteArrayOutputStream out = new ByteArrayOutputStream();
Streams.copy(is, out);
template = out.toByteArray();
} catch (IOException e) {
// throwing an exception to stop exporting process - we don't want to send data unless
// we put in the template for it.
@ -488,7 +491,9 @@ public class HttpESExporter extends AbstractExporter<HttpESExporter> implements
if (conn.getResponseCode() == 200) {
// verify content.
InputStream is = conn.getInputStream();
byte[] existingTemplate = ByteStreams.toByteArray(is);
ByteArrayOutputStream out = new ByteArrayOutputStream();
Streams.copy(is, out);
byte[] existingTemplate = out.toByteArray();
is.close();
int foundVersion = AgentUtils.parseIndexVersionFromTemplate(existingTemplate);
if (foundVersion < 0) {
@ -567,6 +572,15 @@ public class HttpESExporter extends AbstractExporter<HttpESExporter> implements
logger.info("hostname verification set to [{}]", newHostnameVerification);
this.hostnameVerification = newHostnameVerification;
}
String newIndexTimeFormat = settings.get(SETTINGS_INDEX_TIME_FORMAT, null);
if (newIndexTimeFormat != null) {
try {
indexTimeFormatter = DateTimeFormat.forPattern(newIndexTimeFormat).withZoneUTC();
} catch (IllegalArgumentException e) {
logger.error("Unable to update marvel index time format: format [" + newIndexTimeFormat + "] is invalid", e);
}
}
}
protected void initKeepAliveThread() {

13
marvel/src/main/java/org/elasticsearch/marvel/agent/renderer/RendererModule.java
View File

@ -7,17 +7,19 @@ package org.elasticsearch.marvel.agent.renderer;
import org.elasticsearch.common.inject.AbstractModule;
import org.elasticsearch.common.inject.multibindings.MapBinder;
import org.elasticsearch.marvel.agent.collector.cluster.ClusterInfoCollector;
import org.elasticsearch.marvel.agent.collector.cluster.ClusterStateCollector;
import org.elasticsearch.marvel.agent.collector.cluster.ClusterStatsCollector;
import org.elasticsearch.marvel.agent.collector.indices.IndexRecoveryCollector;
import org.elasticsearch.marvel.agent.collector.indices.IndexStatsCollector;
import org.elasticsearch.marvel.agent.collector.licenses.LicensesCollector;
import org.elasticsearch.marvel.agent.collector.indices.IndicesStatsCollector;
import org.elasticsearch.marvel.agent.collector.node.NodeStatsCollector;
import org.elasticsearch.marvel.agent.renderer.cluster.ClusterInfoRenderer;
import org.elasticsearch.marvel.agent.renderer.cluster.ClusterStateRenderer;
import org.elasticsearch.marvel.agent.renderer.cluster.ClusterStatsRenderer;
import org.elasticsearch.marvel.agent.renderer.indices.IndexRecoveryRenderer;
import org.elasticsearch.marvel.agent.renderer.indices.IndexStatsRenderer;
import org.elasticsearch.marvel.agent.renderer.licenses.LicensesRenderer;
import org.elasticsearch.marvel.agent.renderer.indices.IndicesStatsRenderer;
import org.elasticsearch.marvel.agent.renderer.node.NodeStatsRenderer;
import java.util.HashMap;
@ -36,8 +38,11 @@ public class RendererModule extends AbstractModule {
MapBinder<String, Renderer> mbinder = MapBinder.newMapBinder(binder(), String.class, Renderer.class);
// Bind default renderers
bind(LicensesRenderer.class).asEagerSingleton();
mbinder.addBinding(LicensesCollector.TYPE).to(LicensesRenderer.class);
bind(ClusterInfoRenderer.class).asEagerSingleton();
mbinder.addBinding(ClusterInfoCollector.TYPE).to(ClusterInfoRenderer.class);
bind(IndicesStatsRenderer.class).asEagerSingleton();
mbinder.addBinding(IndicesStatsCollector.TYPE).to(IndicesStatsRenderer.class);
bind(IndexStatsRenderer.class).asEagerSingleton();
mbinder.addBinding(IndexStatsCollector.TYPE).to(IndexStatsRenderer.class);

4
marvel/src/main/java/org/elasticsearch/marvel/agent/renderer/RendererRegistry.java
View File

@ -5,9 +5,9 @@
*/
package org.elasticsearch.marvel.agent.renderer;
import com.google.common.collect.ImmutableMap;
import org.elasticsearch.common.inject.Inject;
import java.util.Collections;
import java.util.Map;
public class RendererRegistry {
@ -16,7 +16,7 @@ public class RendererRegistry {
@Inject
public RendererRegistry(Map<String, Renderer> renderers) {
this.renderers = ImmutableMap.copyOf(renderers);
this.renderers = Collections.unmodifiableMap(renderers);
}
public Renderer renderer(String type) {

29
marvel/src/main/java/org/elasticsearch/marvel/agent/renderer/licenses/LicensesRenderer.java → marvel/src/main/java/org/elasticsearch/marvel/agent/renderer/cluster/ClusterInfoRenderer.java
View File

@ -3,34 +3,33 @@
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.marvel.agent.renderer.licenses;
package org.elasticsearch.marvel.agent.renderer.cluster;
import com.google.common.base.Charsets;
import com.google.common.hash.Hashing;
import org.elasticsearch.common.Strings;
import org.elasticsearch.action.admin.cluster.stats.ClusterStatsResponse;
import org.elasticsearch.common.xcontent.ToXContent;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.XContentBuilderString;
import org.elasticsearch.license.core.License;
import org.elasticsearch.marvel.agent.collector.licenses.LicensesMarvelDoc;
import org.elasticsearch.marvel.agent.collector.cluster.ClusterInfoMarvelDoc;
import org.elasticsearch.marvel.agent.renderer.AbstractRenderer;
import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.List;
public class LicensesRenderer extends AbstractRenderer<LicensesMarvelDoc> {
public class ClusterInfoRenderer extends AbstractRenderer<ClusterInfoMarvelDoc> {
public LicensesRenderer() {
super(Strings.EMPTY_ARRAY, false);
public ClusterInfoRenderer() {
super(null, false);
}
@Override
protected void doRender(LicensesMarvelDoc marvelDoc, XContentBuilder builder, ToXContent.Params params) throws IOException {
protected void doRender(ClusterInfoMarvelDoc marvelDoc, XContentBuilder builder, ToXContent.Params params) throws IOException {
builder.field(Fields.CLUSTER_NAME, marvelDoc.getClusterName());
builder.field(Fields.VERSION, marvelDoc.getVersion());
builder.startArray(Fields.LICENSES);
builder.startArray(Fields.LICENSES);
List<License> licenses = marvelDoc.getLicenses();
if (licenses != null) {
for (License license : licenses) {
@ -50,6 +49,13 @@ public class LicensesRenderer extends AbstractRenderer<LicensesMarvelDoc> {
}
}
builder.endArray();
builder.startObject(Fields.CLUSTER_STATS);
ClusterStatsResponse clusterStats = marvelDoc.getClusterStats();
if (clusterStats != null) {
clusterStats.toXContent(builder, params);
}
builder.endObject();
}
public static String hash(License license, String clusterName) {
@ -58,13 +64,14 @@ public class LicensesRenderer extends AbstractRenderer<LicensesMarvelDoc> {
public static String hash(String licenseStatus, String licenseUid, String licenseType, String licenseExpiryDate, String clusterUUID) {
String toHash = licenseStatus + licenseUid + licenseType + licenseExpiryDate + clusterUUID;
return Hashing.sha256().hashString(toHash, Charsets.UTF_8).toString();
return Hashing.sha256().hashString(toHash, StandardCharsets.UTF_8).toString();
}
static final class Fields {
static final XContentBuilderString CLUSTER_NAME = new XContentBuilderString("cluster_name");
static final XContentBuilderString LICENSES = new XContentBuilderString("licenses");
static final XContentBuilderString VERSION = new XContentBuilderString("version");
static final XContentBuilderString CLUSTER_STATS = new XContentBuilderString("cluster_stats");
static final XContentBuilderString HKEY = new XContentBuilderString("hkey");

1
marvel/src/main/java/org/elasticsearch/marvel/agent/renderer/cluster/ClusterStatsRenderer.java
View File

@ -18,6 +18,7 @@ public class ClusterStatsRenderer extends AbstractRenderer<ClusterStatsMarvelDoc
public static final String[] FILTERS = {
"cluster_stats.nodes.count.total",
"cluster_stats.indices.count",
"cluster_stats.indices.shards.total",
"cluster_stats.indices.shards.index.replication.min",
"cluster_stats.indices.docs.count",

10
marvel/src/main/java/org/elasticsearch/marvel/agent/renderer/indices/IndexRecoveryRenderer.java
View File

@ -6,10 +6,10 @@
package org.elasticsearch.marvel.agent.renderer.indices;
import org.elasticsearch.action.admin.indices.recovery.RecoveryResponse;
import org.elasticsearch.action.admin.indices.recovery.ShardRecoveryResponse;
import org.elasticsearch.common.xcontent.ToXContent;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.XContentBuilderString;
import org.elasticsearch.indices.recovery.RecoveryState;
import org.elasticsearch.marvel.agent.collector.indices.IndexRecoveryMarvelDoc;
import org.elasticsearch.marvel.agent.renderer.AbstractRenderer;
@ -30,13 +30,13 @@ public class IndexRecoveryRenderer extends AbstractRenderer<IndexRecoveryMarvelD
RecoveryResponse recovery = marvelDoc.getRecoveryResponse();
if (recovery != null) {
builder.startArray(Fields.SHARDS);
Map<String, List<ShardRecoveryResponse>> shards = recovery.shardResponses();
Map<String, List<RecoveryState>> shards = recovery.shardRecoveryStates();
if (shards != null) {
for (Map.Entry<String, List<ShardRecoveryResponse>> shard : shards.entrySet()) {
for (Map.Entry<String, List<RecoveryState>> shard : shards.entrySet()) {
List<ShardRecoveryResponse> indexShards = shard.getValue();
List<RecoveryState> indexShards = shard.getValue();
if (indexShards != null) {
for (ShardRecoveryResponse indexShard : indexShards) {
for (RecoveryState indexShard : indexShards) {
builder.startObject();
builder.field(Fields.INDEX_NAME, shard.getKey());
indexShard.toXContent(builder, params);

57
marvel/src/main/java/org/elasticsearch/marvel/agent/renderer/indices/IndicesStatsRenderer.java Normal file
View File

@ -0,0 +1,57 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.marvel.agent.renderer.indices;
import org.elasticsearch.action.admin.indices.stats.IndicesStatsResponse;
import org.elasticsearch.common.xcontent.ToXContent;
import org.elasticsearch.common.xcontent.XContentBuilder;
import org.elasticsearch.common.xcontent.XContentBuilderString;
import org.elasticsearch.marvel.agent.collector.indices.IndicesStatsMarvelDoc;
import org.elasticsearch.marvel.agent.renderer.AbstractRenderer;
import java.io.IOException;
public class IndicesStatsRenderer extends AbstractRenderer<IndicesStatsMarvelDoc> {
public static final String[] FILTERS = {
"indices_stats._all.primaries.docs.count",
"indices_stats._all.primaries.indexing.index_time_in_millis",
"indices_stats._all.primaries.indexing.index_total",
"indices_stats._all.primaries.indexing.is_throttled",
"indices_stats._all.primaries.indexing.throttle_time_in_millis",
"indices_stats._all.primaries.search.query_time_in_millis",
"indices_stats._all.primaries.search.query_total",
"indices_stats._all.primaries.store.size_in_bytes",
"indices_stats._all.total.docs.count",
"indices_stats._all.total.indexing.index_time_in_millis",
"indices_stats._all.total.indexing.index_total",
"indices_stats._all.total.indexing.is_throttled",
"indices_stats._all.total.indexing.throttle_time_in_millis",
"indices_stats._all.total.search.query_time_in_millis",
"indices_stats._all.total.search.query_total",
"indices_stats._all.total.store.size_in_bytes",
};
public IndicesStatsRenderer() {
super(FILTERS, true);
}
@Override
protected void doRender(IndicesStatsMarvelDoc marvelDoc, XContentBuilder builder, ToXContent.Params params) throws IOException {
builder.startObject(Fields.INDICES_STATS);
IndicesStatsResponse indicesStats = marvelDoc.getIndicesStats();
if (indicesStats != null) {
indicesStats.toXContent(builder, params);
}
builder.endObject();
}
static final class Fields {
static final XContentBuilderString INDICES_STATS = new XContentBuilderString("indices_stats");
}
}

14
marvel/src/main/java/org/elasticsearch/marvel/agent/settings/MarvelSettings.java
View File

@ -21,12 +21,14 @@ public class MarvelSettings extends AbstractComponent implements NodeSettingsSer
private static final String PREFIX = MarvelPlugin.NAME + ".agent.";
public static final String MARVEL_DATA_INDEX_NAME = ".marvel-data";
public static final String MARVEL_INDICES_PREFIX = ".marvel-es-";
public static final String MARVEL_DATA_INDEX_NAME = MARVEL_INDICES_PREFIX + "data";
public static final TimeValue MAX_LICENSE_GRACE_PERIOD = TimeValue.timeValueHours(7 * 24);
public static final String INTERVAL = PREFIX + "interval";
public static final String STARTUP_DELAY = PREFIX + "startup.delay";
public static final String INDEX_STATS_TIMEOUT = PREFIX + "index.stats.timeout";
public static final String INDICES_STATS_TIMEOUT = PREFIX + "indices.stats.timeout";
public static final String INDICES = PREFIX + "indices";
public static final String CLUSTER_STATE_TIMEOUT = PREFIX + "cluster.state.timeout";
public static final String CLUSTER_STATS_TIMEOUT = PREFIX + "cluster.stats.timeout";
@ -44,7 +46,9 @@ public class MarvelSettings extends AbstractComponent implements NodeSettingsSer
map.put(STARTUP_DELAY, timeSetting(STARTUP_DELAY, null,
"Waiting time before the agent start to collect data (default to sampling interval)", false));
map.put(INDEX_STATS_TIMEOUT, timeoutSetting(INDEX_STATS_TIMEOUT, TimeValue.timeValueMinutes(10),
"Timeout value when collecting indices statistics (default to 10m)", true));
"Timeout value when collecting index statistics (default to 10m)", true));
map.put(INDICES_STATS_TIMEOUT, timeoutSetting(INDICES_STATS_TIMEOUT, TimeValue.timeValueMinutes(10),
"Timeout value when collecting total indices statistics (default to 10m)", true));
map.put(INDICES, arraySetting(INDICES, Strings.EMPTY_ARRAY,
"List of indices names whose stats will be exported (default to all indices)", true));
map.put(CLUSTER_STATE_TIMEOUT, timeoutSetting(CLUSTER_STATE_TIMEOUT, TimeValue.timeValueMinutes(10),
@ -54,7 +58,7 @@ public class MarvelSettings extends AbstractComponent implements NodeSettingsSer
map.put(INDEX_RECOVERY_TIMEOUT, timeoutSetting(INDEX_RECOVERY_TIMEOUT, TimeValue.timeValueMinutes(10),
"Timeout value when collecting the recovery information (default to 10m)", true));
map.put(INDEX_RECOVERY_ACTIVE_ONLY, booleanSetting(INDEX_RECOVERY_ACTIVE_ONLY, Boolean.FALSE,
"INDEX_RECOVERY_ACTIVE_ONLY to indicate if only active recoveries should be collected (default to false: all recoveries are collected)", true));
"Flag to indicate if only active recoveries should be collected (default to false: all recoveries are collected)", true));
map.put(COLLECTORS, arraySetting(COLLECTORS, Strings.EMPTY_ARRAY,
"List of collectors allowed to collect data (default to all)", false));
map.put(LICENSE_GRACE_PERIOD, timeSetting(LICENSE_GRACE_PERIOD, MAX_LICENSE_GRACE_PERIOD,
@ -145,6 +149,10 @@ public class MarvelSettings extends AbstractComponent implements NodeSettingsSer
return getSettingValue(INDEX_STATS_TIMEOUT);
}
public TimeValue indicesStatsTimeout() {
return getSettingValue(INDICES_STATS_TIMEOUT);
}
public String[] indices() {
return getSettingValue(INDICES);
}

36
marvel/src/main/resources/marvel_index_template.json
View File

@ -25,6 +25,40 @@
}
}
},
"marvel_indices_stats": {
"properties": {
"indices_stats": {
"properties": {
"_all": {
"properties": {
"primaries": {
"properties": {
"docs": {
"properties": {
"count": {
"type": "long"
}
}
}
}
},
"total": {
"properties": {
"docs": {
"properties": {
"count": {
"type": "long"
}
}
}
}
}
}
}
}
}
}
},
"marvel_index_stats": {
"properties": {
"index_stats": {
@ -152,7 +186,7 @@
}
}
},
"cluster_licenses": {
"cluster_info": {
"enabled": false
},
"marvel_node_stats": {

31
marvel/src/test/java/org/elasticsearch/marvel/MarvelLauncher.java
View File

@ -5,8 +5,14 @@
*/
package org.elasticsearch.marvel;
import org.elasticsearch.bootstrap.Elasticsearch;
import org.elasticsearch.Version;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.license.plugin.LicensePlugin;
import org.elasticsearch.node.MockNode;
import org.elasticsearch.node.Node;
import java.util.Arrays;
import java.util.concurrent.CountDownLatch;
/**
* Main class to easily run Marvel from a IDE.
@ -17,13 +23,24 @@ import org.elasticsearch.license.plugin.LicensePlugin;
public class MarvelLauncher {
public static void main(String[] args) throws Throwable {
System.setProperty("es.script.inline", "on");
System.setProperty("es.security.manager.enabled", "false");
System.setProperty("es.plugins.load_classpath_plugins", "false");
System.setProperty("es.plugin.types", MarvelPlugin.class.getName() + "," + LicensePlugin.class.getName());
System.setProperty("es.cluster.name", MarvelLauncher.class.getSimpleName());
Settings.Builder settings = Settings.builder();
settings.put("script.inline", "on");
settings.put("security.manager.enabled", "false");
settings.put("plugins.load_classpath_plugins", "false");
settings.put("cluster.name", MarvelLauncher.class.getSimpleName());
Elasticsearch.main(new String[]{"start"});
final CountDownLatch latch = new CountDownLatch(1);
final Node node = new MockNode(settings.build(), Version.CURRENT, Arrays.asList(MarvelPlugin.class, LicensePlugin.class));
Runtime.getRuntime().addShutdownHook(new Thread() {
@Override
public void run() {
node.close();
latch.countDown();
}
});
node.start();
latch.await();
}
}

16
marvel/src/test/java/org/elasticsearch/marvel/MarvelPluginTests.java
View File

@ -10,12 +10,16 @@ import org.elasticsearch.action.admin.cluster.node.info.NodesInfoResponse;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.license.plugin.LicensePlugin;
import org.elasticsearch.marvel.agent.AgentService;
import org.elasticsearch.plugins.Plugin;
import org.elasticsearch.plugins.PluginInfo;
import org.elasticsearch.test.ESIntegTestCase;
import org.elasticsearch.test.ESIntegTestCase.ClusterScope;
import org.elasticsearch.tribe.TribeService;
import org.junit.Test;
import java.util.Arrays;
import java.util.Collection;
import static org.elasticsearch.test.ESIntegTestCase.Scope.TEST;
import static org.hamcrest.Matchers.equalTo;
@ -23,11 +27,13 @@ import static org.hamcrest.Matchers.equalTo;
public class MarvelPluginTests extends ESIntegTestCase {
@Override
protected Settings nodeSettings(int nodeOrdinal) {
return Settings.settingsBuilder()
.put(super.nodeSettings(nodeOrdinal))
.put("plugin.types", MarvelPlugin.class.getName() + "," + LicensePlugin.class.getName())
.build();
protected Collection<Class<? extends Plugin>> nodePlugins() {
return Arrays.asList(LicensePlugin.class, MarvelPlugin.class);
}
@Override
protected Collection<Class<? extends Plugin>> transportClientPlugins() {
return nodePlugins();
}
@Test

69
marvel/src/test/java/org/elasticsearch/marvel/agent/collector/AbstractCollectorTestCase.java
View File

@ -7,6 +7,8 @@ package org.elasticsearch.marvel.agent.collector;
import com.carrotsearch.randomizedtesting.RandomizedTest;
import com.carrotsearch.randomizedtesting.SysGlobals;
import org.elasticsearch.action.ActionListener;
import org.elasticsearch.cluster.ack.ClusterStateUpdateResponse;
import org.elasticsearch.cluster.block.ClusterBlocks;
import org.elasticsearch.common.component.AbstractComponent;
import org.elasticsearch.common.inject.AbstractModule;
@ -16,19 +18,17 @@ import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.license.core.License;
import org.elasticsearch.license.plugin.core.LicensesClientService;
import org.elasticsearch.license.plugin.core.LicensesManagerService;
import org.elasticsearch.license.plugin.core.LicensesService;
import org.elasticsearch.marvel.MarvelPlugin;
import org.elasticsearch.marvel.agent.settings.MarvelSettings;
import org.elasticsearch.marvel.license.LicenseIntegrationTests;
import org.elasticsearch.marvel.license.LicenseService;
import org.elasticsearch.plugins.Plugin;
import org.elasticsearch.test.ESIntegTestCase;
import org.elasticsearch.test.ESIntegTestCase.ClusterScope;
import org.junit.Before;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.*;
import java.util.concurrent.Callable;
import java.util.concurrent.TimeUnit;
@ -36,11 +36,13 @@ import java.util.concurrent.TimeUnit;
public class AbstractCollectorTestCase extends ESIntegTestCase {
@Override
protected Settings nodeSettings(int nodeOrdinal) {
return Settings.settingsBuilder()
.put(super.nodeSettings(nodeOrdinal))
.put("plugin.types", MarvelPlugin.class.getName() + "," + LicensePluginForCollectors.class.getName())
.build();
protected Collection<Class<? extends Plugin>> nodePlugins() {
return Arrays.asList(LicensePluginForCollectors.class, MarvelPlugin.class);
}
@Override
protected Collection<Class<? extends Plugin>> transportClientPlugins() {
return nodePlugins();
}
@Before
@ -52,7 +54,7 @@ public class AbstractCollectorTestCase extends ESIntegTestCase {
assertNotNull(collector);
assertTrue("collector [" + collector.name() + "] should be able to collect data", collector.canCollect());
Collection results = collector.collect();
assertTrue(results != null && !results.isEmpty());
assertNotNull(results);
}
protected void assertCannotCollect(AbstractCollector collector) {
@ -73,7 +75,7 @@ public class AbstractCollectorTestCase extends ESIntegTestCase {
.signature("_signature")
.type("standard")
.subscriptionType("all_is_good")
.uid(String.valueOf(RandomizedTest.systemPropertyAsInt(SysGlobals.CHILDVM_SYSPROP_JVM_ID, 0)) + System.identityHashCode(LicenseIntegrationTests.class))
.uid(String.valueOf(RandomizedTest.systemPropertyAsInt(SysGlobals.CHILDVM_SYSPROP_JVM_ID, 0)) + System.identityHashCode(AbstractCollectorTestCase.class))
.build();
}
@ -85,6 +87,9 @@ public class AbstractCollectorTestCase extends ESIntegTestCase {
for (LicenseServiceForCollectors service : internalCluster().getInstances(LicenseServiceForCollectors.class)) {
service.enable(license);
}
for (LicensesManagerServiceForCollectors service : internalCluster().getInstances(LicensesManagerServiceForCollectors.class)) {
service.update(license);
}
}
protected static void beginGracefulPeriod() {
@ -95,6 +100,9 @@ public class AbstractCollectorTestCase extends ESIntegTestCase {
for (LicenseServiceForCollectors service : internalCluster().getInstances(LicenseServiceForCollectors.class)) {
service.disable(license);
}
for (LicensesManagerServiceForCollectors service : internalCluster().getInstances(LicensesManagerServiceForCollectors.class)) {
service.update(license);
}
}
protected static void endGracefulPeriod() {
@ -105,6 +113,9 @@ public class AbstractCollectorTestCase extends ESIntegTestCase {
for (LicenseServiceForCollectors service : internalCluster().getInstances(LicenseServiceForCollectors.class)) {
service.disable(license);
}
for (LicensesManagerServiceForCollectors service : internalCluster().getInstances(LicensesManagerServiceForCollectors.class)) {
service.update(license);
}
}
protected static void disableLicense() {
@ -115,6 +126,9 @@ public class AbstractCollectorTestCase extends ESIntegTestCase {
for (LicenseServiceForCollectors service : internalCluster().getInstances(LicenseServiceForCollectors.class)) {
service.disable(license);
}
for (LicensesManagerServiceForCollectors service : internalCluster().getInstances(LicensesManagerServiceForCollectors.class)) {
service.update(license);
}
}
private static long randomDaysInMillis() {
@ -164,12 +178,14 @@ public class AbstractCollectorTestCase extends ESIntegTestCase {
@Override
public Collection<Module> nodeModules() {
return Collections.<Module>singletonList(new AbstractModule(){
return Collections.<Module>singletonList(new AbstractModule() {
@Override
protected void configure() {
bind(LicenseServiceForCollectors.class).asEagerSingleton();
bind(LicensesClientService.class).to(LicenseServiceForCollectors.class);
bind(LicensesManagerServiceForCollectors.class).asEagerSingleton();
bind(LicensesManagerService.class).to(LicensesManagerServiceForCollectors.class);
}
});
}
@ -201,4 +217,31 @@ public class AbstractCollectorTestCase extends ESIntegTestCase {
}
}
}
public static class LicensesManagerServiceForCollectors implements LicensesManagerService {
private final Map<String, License> licenses = Collections.synchronizedMap(new HashMap<String, License>());
@Override
public void registerLicenses(LicensesService.PutLicenseRequestHolder requestHolder, ActionListener<LicensesService.LicensesUpdateResponse> listener) {
}
@Override
public void removeLicenses(LicensesService.DeleteLicenseRequestHolder requestHolder, ActionListener<ClusterStateUpdateResponse> listener) {
}
@Override
public Set<String> enabledFeatures() {
return null;
}
@Override
public List<License> getLicenses() {
return new ArrayList<>(licenses.values());
}
public void update(License license) {
licenses.put(license.uid(), license);
}
}
}

103
marvel/src/test/java/org/elasticsearch/marvel/agent/collector/cluster/ClusterInfoCollectorTests.java Normal file
View File

@ -0,0 +1,103 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.marvel.agent.collector.cluster;
import org.elasticsearch.cluster.ClusterName;
import org.elasticsearch.cluster.ClusterService;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.marvel.agent.collector.AbstractCollectorTestCase;
import org.elasticsearch.marvel.agent.exporter.MarvelDoc;
import org.elasticsearch.marvel.agent.settings.MarvelSettings;
import org.elasticsearch.marvel.license.LicenseService;
import org.junit.Test;
import java.util.Collection;
import static org.hamcrest.Matchers.*;
public class ClusterInfoCollectorTests extends AbstractCollectorTestCase {
@Test
public void testClusterInfoCollector() throws Exception {
Collection<MarvelDoc> results = newClusterInfoCollector().doCollect();
assertThat(results, hasSize(1));
MarvelDoc marvelDoc = results.iterator().next();
assertNotNull(marvelDoc);
assertThat(marvelDoc, instanceOf(ClusterInfoMarvelDoc.class));
ClusterInfoMarvelDoc clusterInfoMarvelDoc = (ClusterInfoMarvelDoc) marvelDoc;
assertThat(clusterInfoMarvelDoc.clusterUUID(), equalTo(client().admin().cluster().prepareState().setMetaData(true).get().getState().metaData().clusterUUID()));
assertThat(clusterInfoMarvelDoc.timestamp(), greaterThan(0L));
assertThat(clusterInfoMarvelDoc.type(), equalTo(ClusterInfoCollector.TYPE));
assertThat(clusterInfoMarvelDoc.getClusterName(), equalTo(client().admin().cluster().prepareState().setMetaData(true).get().getClusterName().value()));
assertThat(clusterInfoMarvelDoc.getVersion(), equalTo(client().admin().cluster().prepareNodesInfo().get().getNodes()[0].getVersion().toString()));
assertNotNull(clusterInfoMarvelDoc.getLicenses());
assertThat(clusterInfoMarvelDoc.getLicenses(), hasSize(1));
assertNotNull(clusterInfoMarvelDoc.getClusterStats());
assertThat(clusterInfoMarvelDoc.getClusterStats().getNodesStats().getCounts().getTotal(), equalTo(internalCluster().getNodeNames().length));
}
@Test
public void testClusterInfoCollectorWithLicensing() {
String[] nodes = internalCluster().getNodeNames();
for (String node : nodes) {
logger.debug("--> creating a new instance of the collector");
ClusterInfoCollector collector = newClusterInfoCollector(node);
assertNotNull(collector);
logger.debug("--> enabling license and checks that the collector can collect data (if node is master)");
enableLicense();
if (node.equals(internalCluster().getMasterName())) {
assertCanCollect(collector);
} else {
assertCannotCollect(collector);
}
logger.debug("--> starting graceful period and checks that the collector can still collect data (if node is master)");
beginGracefulPeriod();
if (node.equals(internalCluster().getMasterName())) {
assertCanCollect(collector);
} else {
assertCannotCollect(collector);
}
logger.debug("--> ending graceful period and checks that the collector can still collect data (if node is master)");
endGracefulPeriod();
if (node.equals(internalCluster().getMasterName())) {
assertCanCollect(collector);
} else {
assertCannotCollect(collector);
}
logger.debug("--> disabling license and checks that the collector can still collect data (if node is master)");
disableLicense();
if (node.equals(internalCluster().getMasterName())) {
assertCanCollect(collector);
} else {
assertCannotCollect(collector);
}
}
}
private ClusterInfoCollector newClusterInfoCollector() {
// This collector runs on master node only
return newClusterInfoCollector(internalCluster().getMasterName());
}
private ClusterInfoCollector newClusterInfoCollector(String nodeId) {
assertNotNull(nodeId);
return new ClusterInfoCollector(internalCluster().getInstance(Settings.class, nodeId),
internalCluster().getInstance(ClusterService.class, nodeId),
internalCluster().getInstance(MarvelSettings.class, nodeId),
internalCluster().getInstance(LicenseService.class, nodeId),
internalCluster().getInstance(ClusterName.class, nodeId),
client(nodeId));
}
}

10
marvel/src/test/java/org/elasticsearch/marvel/agent/collector/cluster/ClusterStateCollectorTests.java
View File

@ -8,7 +8,6 @@ package org.elasticsearch.marvel.agent.collector.cluster;
import org.elasticsearch.cluster.ClusterService;
import org.elasticsearch.cluster.ClusterState;
import org.elasticsearch.cluster.metadata.IndexMetaData;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.marvel.agent.collector.AbstractCollectorTestCase;
import org.elasticsearch.marvel.agent.exporter.MarvelDoc;
@ -118,7 +117,7 @@ public class ClusterStateCollectorTests extends AbstractCollectorTestCase {
}
@Test
public void tesClusterStateCollectorWithLicensing() {
public void testClusterStateCollectorWithLicensing() {
String[] nodes = internalCluster().getNodeNames();
for (String node : nodes) {
logger.debug("--> creating a new instance of the collector");
@ -152,13 +151,12 @@ public class ClusterStateCollectorTests extends AbstractCollectorTestCase {
}
private ClusterStateCollector newClusterStateCollector() {
return newClusterStateCollector(null);
// This collector runs on master node only
return newClusterStateCollector(internalCluster().getMasterName());
}
private ClusterStateCollector newClusterStateCollector(String nodeId) {
if (!Strings.hasText(nodeId)) {
nodeId = randomFrom(internalCluster().getNodeNames());
}
assertNotNull(nodeId);
return new ClusterStateCollector(internalCluster().getInstance(Settings.class, nodeId),
internalCluster().getInstance(ClusterService.class, nodeId),
internalCluster().getInstance(MarvelSettings.class, nodeId),

10
marvel/src/test/java/org/elasticsearch/marvel/agent/collector/cluster/ClusterStatsCollectorTests.java
View File

@ -6,7 +6,6 @@
package org.elasticsearch.marvel.agent.collector.cluster;
import org.elasticsearch.cluster.ClusterService;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.marvel.agent.collector.AbstractCollectorTestCase;
import org.elasticsearch.marvel.agent.exporter.MarvelDoc;
@ -39,7 +38,7 @@ public class ClusterStatsCollectorTests extends AbstractCollectorTestCase {
}
@Test
public void tesClusterStatsCollectorWithLicensing() {
public void testClusterStatsCollectorWithLicensing() {
String[] nodes = internalCluster().getNodeNames();
for (String node : nodes) {
logger.debug("--> creating a new instance of the collector");
@ -73,13 +72,12 @@ public class ClusterStatsCollectorTests extends AbstractCollectorTestCase {
}
private ClusterStatsCollector newClusterStatsCollector() {
return newClusterStatsCollector(null);
// This collector runs on master node only
return newClusterStatsCollector(internalCluster().getMasterName());
}
private ClusterStatsCollector newClusterStatsCollector(String nodeId) {
if (!Strings.hasText(nodeId)) {
nodeId = randomFrom(internalCluster().getNodeNames());
}
assertNotNull(nodeId);
return new ClusterStatsCollector(internalCluster().getInstance(Settings.class, nodeId),
internalCluster().getInstance(ClusterService.class, nodeId),
internalCluster().getInstance(MarvelSettings.class, nodeId),

15
marvel/src/test/java/org/elasticsearch/marvel/agent/collector/indices/IndexRecoveryCollectorTests.java
View File

@ -6,7 +6,6 @@
package org.elasticsearch.marvel.agent.collector.indices;
import org.elasticsearch.action.admin.indices.recovery.RecoveryResponse;
import org.elasticsearch.action.admin.indices.recovery.ShardRecoveryResponse;
import org.elasticsearch.cluster.ClusterService;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.settings.Settings;
@ -102,23 +101,23 @@ public class IndexRecoveryCollectorTests extends AbstractCollectorTestCase {
RecoveryResponse recovery = indexRecoveryMarvelDoc.getRecoveryResponse();
assertNotNull(recovery);
Map<String, List<ShardRecoveryResponse>> shards = recovery.shardResponses();
Map<String, List<RecoveryState>> shards = recovery.shardRecoveryStates();
assertThat(shards.size(), greaterThan(0));
for (Map.Entry<String, List<ShardRecoveryResponse>> shard : shards.entrySet()) {
List<ShardRecoveryResponse> shardRecoveries = shard.getValue();
for (Map.Entry<String, List<RecoveryState>> shard : shards.entrySet()) {
List<RecoveryState> shardRecoveries = shard.getValue();
assertNotNull(shardRecoveries);
assertThat(shardRecoveries.size(), greaterThan(0));
for (ShardRecoveryResponse shardRecovery : shardRecoveries) {
assertThat(shardRecovery.getIndex(), equalTo(indexName));
assertThat(shardRecovery.recoveryState().getType(), anyOf(equalTo(RecoveryState.Type.RELOCATION), equalTo(RecoveryState.Type.STORE), equalTo(RecoveryState.Type.REPLICA)));
for (RecoveryState shardRecovery : shardRecoveries) {
assertThat(shard.getKey(), equalTo(indexName));
assertThat(shardRecovery.getType(), anyOf(equalTo(RecoveryState.Type.RELOCATION), equalTo(RecoveryState.Type.STORE), equalTo(RecoveryState.Type.REPLICA)));
}
}
}
@Test
public void tesIndexRecoveryCollectorWithLicensing() {
public void testIndexRecoveryCollectorWithLicensing() {
String[] nodes = internalCluster().getNodeNames();
for (String node : nodes) {
logger.debug("--> creating a new instance of the collector");

101
marvel/src/test/java/org/elasticsearch/marvel/agent/collector/indices/IndexStatsCollectorTests.java
View File

@ -6,9 +6,11 @@
package org.elasticsearch.marvel.agent.collector.indices;
import org.elasticsearch.action.admin.indices.stats.IndexStats;
import org.elasticsearch.action.admin.indices.stats.IndicesStatsResponse;
import org.elasticsearch.cluster.ClusterService;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.marvel.agent.collector.AbstractCollectorTestCase;
import org.elasticsearch.marvel.agent.collector.cluster.ClusterStatsCollector;
import org.elasticsearch.marvel.agent.exporter.MarvelDoc;
import org.elasticsearch.marvel.agent.settings.MarvelSettings;
import org.elasticsearch.marvel.license.LicenseService;
@ -16,6 +18,8 @@ import org.junit.Test;
import java.util.Collection;
import java.util.Iterator;
import java.util.concurrent.Callable;
import java.util.concurrent.TimeUnit;
import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertHitCount;
import static org.hamcrest.Matchers.*;
@ -31,22 +35,40 @@ public class IndexStatsCollectorTests extends AbstractCollectorTestCase {
}
@Test
@AwaitsFix(bugUrl = "https://github.com/elastic/x-plugins/pull/538")
public void testIndexStatsCollectorOneIndex() throws Exception {
waitForNoBlocksOnNodes();
final String indexName = "test_" + randomInt();
final String indexName = "one-index";
final int nbDocs = randomIntBetween(1, 20);
for (int i = 0; i < nbDocs; i++) {
client().prepareIndex(indexName, "test").setSource("num", i).get();
}
client().admin().indices().prepareRefresh().get();
assertHitCount(client().prepareCount().get(), nbDocs);
refresh();
waitForRelocation();
Collection<MarvelDoc> results = newIndexStatsCollector().doCollect();
assertThat(results, hasSize(1));
assertHitCount(client().prepareCount().get(), nbDocs);
logger.debug("--> wait for index stats to report data about indices");
assertBusy(new Runnable() {
@Override
public void run() {
IndicesStatsResponse response = client(internalCluster().getMasterName()).admin().indices().prepareStats().setRefresh(true).get();
assertNotNull(response.getIndices());
assertThat(response.getIndices().size(), greaterThan(0));
}
}, 30L, TimeUnit.SECONDS);
Collection<MarvelDoc> results = assertBusy(new Callable<Collection<MarvelDoc>>() {
@Override
public Collection<MarvelDoc> call() throws Exception {
Collection<MarvelDoc> results = newIndexStatsCollector().doCollect();
assertThat(results, hasSize(1));
return results;
}
}, 30L, TimeUnit.SECONDS);
MarvelDoc marvelDoc = results.iterator().next();
assertNotNull(marvelDoc);
@ -70,11 +92,12 @@ public class IndexStatsCollectorTests extends AbstractCollectorTestCase {
}
@Test
@AwaitsFix(bugUrl = "https://github.com/elastic/x-plugins/pull/538")
public void testIndexStatsCollectorMultipleIndices() throws Exception {
waitForNoBlocksOnNodes();
final String indexPrefix = "test_" + randomInt() + "_";
int nbIndices = randomIntBetween(1, 5);
final String indexPrefix = "multi-indices-";
final int nbIndices = randomIntBetween(1, 5);
int[] docsPerIndex = new int[nbIndices];
for (int i = 0; i < nbIndices; i++) {
@ -90,10 +113,27 @@ public class IndexStatsCollectorTests extends AbstractCollectorTestCase {
assertHitCount(client().prepareCount(indexPrefix + i).get(), docsPerIndex[i]);
}
refresh();
waitForRelocation();
Collection<MarvelDoc> results = newIndexStatsCollector().doCollect();
assertThat(results, hasSize(nbIndices));
logger.debug("--> wait for index stats to report data about indices");
assertBusy(new Runnable() {
@Override
public void run() {
IndicesStatsResponse response = client().admin().indices().prepareStats().setRefresh(true).get();
assertNotNull(response.getIndices());
assertThat(response.getIndices().size(), greaterThan(0));
}
}, 30L, TimeUnit.SECONDS);
Collection<MarvelDoc> results = assertBusy(new Callable<Collection<MarvelDoc>>() {
@Override
public Collection<MarvelDoc> call() throws Exception {
Collection<MarvelDoc> results = newIndexStatsCollector().doCollect();
assertThat(results, hasSize(nbIndices));
return results;
}
}, 30L, TimeUnit.SECONDS);
for (int i = 0; i < nbIndices; i++) {
String indexName = indexPrefix + i;
@ -117,7 +157,7 @@ public class IndexStatsCollectorTests extends AbstractCollectorTestCase {
assertNotNull(indexStats.getTotal().getDocs());
assertThat(indexStats.getPrimaries().getDocs().getCount(), equalTo((long) docsPerIndex[i]));
assertNotNull(indexStats.getTotal().getStore());
assertThat(indexStats.getTotal().getStore().getSizeInBytes(), greaterThan(0L));
assertThat(indexStats.getTotal().getStore().getSizeInBytes(), greaterThanOrEqualTo(0L));
assertThat(indexStats.getTotal().getStore().getThrottleTime().millis(), equalTo(0L));
assertNotNull(indexStats.getTotal().getIndexing());
assertThat(indexStats.getTotal().getIndexing().getTotal().getThrottleTimeInMillis(), equalTo(0L));
@ -128,8 +168,47 @@ public class IndexStatsCollectorTests extends AbstractCollectorTestCase {
}
}
@Test
public void testIndexStatsCollectorWithLicensing() {
String[] nodes = internalCluster().getNodeNames();
for (String node : nodes) {
logger.debug("--> creating a new instance of the collector");
IndexStatsCollector collector = newIndexStatsCollector(node);
assertNotNull(collector);
logger.debug("--> enabling license and checks that the collector can collect data if node is master");
enableLicense();
if (node.equals(internalCluster().getMasterName())) {
assertCanCollect(collector);
} else {
assertCannotCollect(collector);
}
logger.debug("--> starting graceful period and checks that the collector can still collect data if node is master");
beginGracefulPeriod();
if (node.equals(internalCluster().getMasterName())) {
assertCanCollect(collector);
} else {
assertCannotCollect(collector);
}
logger.debug("--> ending graceful period and checks that the collector cannot collect data");
endGracefulPeriod();
assertCannotCollect(collector);
logger.debug("--> disabling license and checks that the collector cannot collect data");
disableLicense();
assertCannotCollect(collector);
}
}
private IndexStatsCollector newIndexStatsCollector() {
String nodeId = randomFrom(internalCluster().getNodeNames());
// This collector runs on master node only
return newIndexStatsCollector(internalCluster().getMasterName());
}
private IndexStatsCollector newIndexStatsCollector(String nodeId) {
assertNotNull(nodeId);
return new IndexStatsCollector(internalCluster().getInstance(Settings.class, nodeId),
internalCluster().getInstance(ClusterService.class, nodeId),
internalCluster().getInstance(MarvelSettings.class, nodeId),

177
marvel/src/test/java/org/elasticsearch/marvel/agent/exporter/HttpESExporterTests.java
View File

@ -5,30 +5,36 @@
*/
package org.elasticsearch.marvel.agent.exporter;
import com.google.common.base.Predicate;
import com.google.common.collect.ImmutableList;
import org.elasticsearch.action.admin.indices.stats.IndexStats;
import org.elasticsearch.action.admin.cluster.health.ClusterHealthStatus;
import org.elasticsearch.action.admin.indices.recovery.RecoveryResponse;
import org.elasticsearch.cluster.ClusterState;
import org.elasticsearch.cluster.metadata.IndexTemplateMetaData;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.license.plugin.LicensePlugin;
import org.elasticsearch.marvel.MarvelPlugin;
import org.elasticsearch.marvel.agent.AgentService;
import org.elasticsearch.marvel.agent.collector.indices.IndexStatsMarvelDoc;
import org.elasticsearch.marvel.agent.collector.cluster.ClusterStateCollector;
import org.elasticsearch.marvel.agent.collector.cluster.ClusterStateMarvelDoc;
import org.elasticsearch.marvel.agent.collector.indices.IndexRecoveryCollector;
import org.elasticsearch.marvel.agent.collector.indices.IndexRecoveryMarvelDoc;
import org.elasticsearch.marvel.agent.settings.MarvelSettings;
import org.elasticsearch.node.Node;
import org.elasticsearch.plugins.Plugin;
import org.elasticsearch.test.ESIntegTestCase;
import org.elasticsearch.test.ESIntegTestCase.ClusterScope;
import org.hamcrest.Matchers;
import org.joda.time.format.DateTimeFormat;
import org.junit.Test;
import java.io.IOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.concurrent.atomic.AtomicLong;
import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked;
import static org.hamcrest.Matchers.equalTo;
// Transport Client instantiation also calls the marvel plugin, which then fails to find modules
@ -39,13 +45,15 @@ public class HttpESExporterTests extends ESIntegTestCase {
final static AtomicLong timeStampGenerator = new AtomicLong();
@Override
protected Settings nodeSettings(int nodeOrdinal) {
return Settings.settingsBuilder()
.put(super.nodeSettings(nodeOrdinal))
.put("plugin.types", MarvelPlugin.class.getName() + "," + LicensePlugin.class.getName())
.build();
protected Collection<Class<? extends Plugin>> nodePlugins() {
return Arrays.asList(LicensePlugin.class, MarvelPlugin.class);
}
@Override
protected Collection<Class<? extends Plugin>> transportClientPlugins() {
return nodePlugins();
}
@Test
public void testHttpServerOff() {
Settings.Builder builder = Settings.builder()
@ -54,36 +62,9 @@ public class HttpESExporterTests extends ESIntegTestCase {
internalCluster().startNode(builder);
HttpESExporter httpEsExporter = getEsExporter();
logger.info("trying exporting despite of no target");
httpEsExporter.export(ImmutableList.of(newRandomMarvelDoc()));
httpEsExporter.export(Collections.singletonList(newRandomMarvelDoc()));
}
/*
@Test
public void testLargeClusterStateSerialization() throws InterruptedException {
// make sure no other exporting is done (quicker)..
internalCluster().startNode(Settings.builder().put(AgentService.SETTINGS_INTERVAL, "200m").put(Node.HTTP_ENABLED, true));
ESExporter esExporter = internalCluster().getInstance(ESExporter.class);
DiscoveryNodes.Builder nodesBuilder = new DiscoveryNodes.Builder();
int nodeCount = randomIntBetween(10, 200);
for (int i = 0; i < nodeCount; i++) {
nodesBuilder.put(new DiscoveryNode("node_" + i, new LocalTransportAddress("node_" + i), Version.CURRENT));
}
// get the current cluster state rather then construct one because the constructors have changed across ES versions
ClusterService clusterService = internalCluster().getInstance(ClusterService.class);
ClusterState state = ClusterState.builder(clusterService.state()).nodes(nodesBuilder).build();
logger.info("exporting cluster state with {} nodes", state.nodes().size());
esExporter.exportEvents(new Event[]{
new ClusterEvent.ClusterStateChange(1234l, state, "test", ClusterHealthStatus.GREEN, "testing_1234", "test_source_unique")
});
logger.info("done exporting");
ensureYellow();
client().admin().indices().prepareRefresh(".marvel-*").get();
assertHitCount(client().prepareSearch().setQuery(QueryBuilders.termQuery("event_source", "test_source_unique")).get(), 1);
}
*/
@Test
public void testTemplateAdditionDespiteOfLateClusterForming() {
Settings.Builder builder = Settings.builder()
@ -95,20 +76,21 @@ public class HttpESExporterTests extends ESIntegTestCase {
.put("discovery.zen.minimum_master_nodes", 2)
.put(HttpESExporter.SETTINGS_BULK_TIMEOUT, "1s")
.put(HttpESExporter.SETTINGS_CHECK_TEMPLATE_TIMEOUT, "1s");
internalCluster().startNode(builder);
HttpESExporter httpEsExporter = getEsExporter();
logger.info("exporting events while there is no cluster");
httpEsExporter.export(ImmutableList.of(newRandomMarvelDoc()));
httpEsExporter.export(Collections.singletonList(newRandomMarvelDoc()));
logger.info("bringing up a second node");
internalCluster().startNode(builder);
ensureGreen();
logger.info("exporting a second event");
httpEsExporter.export(ImmutableList.of(newRandomMarvelDoc()));
httpEsExporter.export(Collections.singletonList(newRandomMarvelDoc()));
logger.info("verifying template is inserted");
assertMarvelTemplate();
logger.info("verifying that template has been created");
assertMarvelTemplateExists();
}
@Test
@ -120,7 +102,7 @@ public class HttpESExporterTests extends ESIntegTestCase {
HttpESExporter httpEsExporter = getEsExporter();
assertAcked(client().admin().cluster().prepareUpdateSettings().setTransientSettings(Settings.builder().put(HttpESExporter.SETTINGS_HOSTS, "test1")));
assertAcked(client().admin().cluster().prepareUpdateSettings().setTransientSettings(Settings.builder().putArray(HttpESExporter.SETTINGS_HOSTS, "test1")));
assertThat(httpEsExporter.getHosts(), Matchers.arrayContaining("test1"));
// wipes the non array settings
@ -142,36 +124,38 @@ public class HttpESExporterTests extends ESIntegTestCase {
HttpESExporter httpEsExporter = getEsExporter();
logger.info("exporting an event");
httpEsExporter.export(ImmutableList.of(newRandomMarvelDoc()));
httpEsExporter.export(Collections.singletonList(newRandomMarvelDoc()));
logger.info("removing the marvel template");
assertAcked(client().admin().indices().prepareDeleteTemplate("marvel").get());
assertMarvelTemplateNotExists();
assertAcked(client().admin().cluster().prepareUpdateSettings().setTransientSettings(
Settings.builder().putArray(HttpESExporter.SETTINGS_HOSTS, httpEsExporter.getHosts())).get());
logger.info("exporting a second event");
httpEsExporter.export(ImmutableList.of(newRandomMarvelDoc()));
httpEsExporter.export(Collections.singletonList(newRandomMarvelDoc()));
logger.info("verifying template is inserted");
assertMarvelTemplate();
logger.info("verifying that template has been created");
assertMarvelTemplateExists();
}
@Test
public void testHostFailureChecksTemplate() throws InterruptedException, IOException {
public void testHostFailureChecksTemplate() throws Exception {
Settings.Builder builder = Settings.builder()
.put(MarvelSettings.STARTUP_DELAY, "200m")
.put(Node.HTTP_ENABLED, true);
final String node0 = internalCluster().startNode(builder);
String node1 = internalCluster().startNode(builder);
HttpESExporter httpEsExporter0 = getEsExporter(node0);
final String node0 = internalCluster().startNode(builder);
final HttpESExporter httpEsExporter0 = getEsExporter(node0);
assertThat(node0, equalTo(internalCluster().getMasterName()));
final String node1 = internalCluster().startNode(builder);
final HttpESExporter httpEsExporter1 = getEsExporter(node1);
logger.info("--> exporting events to force host resolution");
httpEsExporter0.export(ImmutableList.of(newRandomMarvelDoc()));
httpEsExporter1.export(ImmutableList.of(newRandomMarvelDoc()));
httpEsExporter0.export(Collections.singletonList(newRandomMarvelDoc()));
httpEsExporter1.export(Collections.singletonList(newRandomMarvelDoc()));
logger.info("--> setting exporting hosts to {} + {}", httpEsExporter0.getHosts(), httpEsExporter1.getHosts());
ArrayList<String> mergedHosts = new ArrayList<String>();
@ -182,32 +166,60 @@ public class HttpESExporterTests extends ESIntegTestCase {
Settings.builder().putArray(HttpESExporter.SETTINGS_HOSTS, mergedHosts.toArray(Strings.EMPTY_ARRAY))).get());
logger.info("--> exporting events to have new settings take effect");
httpEsExporter0.export(ImmutableList.of(newRandomMarvelDoc()));
httpEsExporter1.export(ImmutableList.of(newRandomMarvelDoc()));
httpEsExporter0.export(Collections.singletonList(newRandomMarvelDoc()));
httpEsExporter1.export(Collections.singletonList(newRandomMarvelDoc()));
assertMarvelTemplate();
logger.info("verifying that template has been created");
assertMarvelTemplateExists();
logger.info("--> removing the marvel template");
assertAcked(client().admin().indices().prepareDeleteTemplate("marvel").get());
assertMarvelTemplateNotExists();
logger.info("--> shutting down node0");
internalCluster().stopRandomNode(new Predicate<Settings>() {
@Override
public boolean apply(Settings settings) {
return settings.get("name").equals(node0);
}
});
internalCluster().stopCurrentMasterNode();
logger.info("--> exporting events from node1");
// we use assert busy node because url caching may cause the node failure to be only detected while sending the event
assertTrue("failed to find a template named 'marvel'", awaitBusy(new Predicate<Object>() {
assertBusy(new Runnable() {
@Override
public boolean apply(Object o) {
httpEsExporter1.export(ImmutableList.of(newRandomMarvelDoc()));
public void run() {
httpEsExporter1.export(Collections.singletonList(newRandomMarvelDoc()));
logger.debug("--> checking for template");
return findMarvelTemplate();
assertMarvelTemplateExists();
}
}));
});
}
@Test
public void testDynamicIndexFormatChange() {
Settings.Builder builder = Settings.builder()
.put(MarvelSettings.STARTUP_DELAY, "200m")
.put(Node.HTTP_ENABLED, true);
String nodeId = internalCluster().startNode(builder);
logger.info("exporting a first event");
HttpESExporter httpEsExporter = getEsExporter(nodeId);
httpEsExporter.export(Collections.singletonList(newRandomMarvelDoc()));
logger.info("checks that the index [{}] is created", httpEsExporter.getIndexName());
assertTrue(client().admin().indices().prepareExists(httpEsExporter.getIndexName()).get().isExists());
String newTimeFormat = randomFrom("YY", "YYYY", "YYYY.MM", "YYYY-MM", "MM.YYYY", "MM");
logger.info("updating index time format setting to {}", newTimeFormat);
assertAcked(client().admin().cluster().prepareUpdateSettings().setTransientSettings(Settings.builder().put(HttpESExporter.SETTINGS_INDEX_TIME_FORMAT, newTimeFormat)));
logger.info("exporting a second event");
httpEsExporter.export(Collections.singletonList(newRandomMarvelDoc()));
String expectedMarvelIndex = MarvelSettings.MARVEL_INDICES_PREFIX
+ DateTimeFormat.forPattern(newTimeFormat).withZoneUTC().print(System.currentTimeMillis());
logger.info("checks that the index [{}] is created", expectedMarvelIndex);
assertTrue(client().admin().indices().prepareExists(expectedMarvelIndex).get().isExists());
logger.info("verifying that template has been created");
assertMarvelTemplateExists();
}
private HttpESExporter getEsExporter() {
@ -221,19 +233,26 @@ public class HttpESExporterTests extends ESIntegTestCase {
}
private MarvelDoc newRandomMarvelDoc() {
return new IndexStatsMarvelDoc(internalCluster().getClusterName(), "test_marvelDoc", timeStampGenerator.incrementAndGet(),
new IndexStats("test_index", null));
if (randomBoolean()) {
return new IndexRecoveryMarvelDoc(internalCluster().getClusterName(),
IndexRecoveryCollector.TYPE, timeStampGenerator.incrementAndGet(), new RecoveryResponse());
} else {
return new ClusterStateMarvelDoc(internalCluster().getClusterName(),
ClusterStateCollector.TYPE, timeStampGenerator.incrementAndGet(), ClusterState.PROTO, ClusterHealthStatus.GREEN);
}
}
private void assertMarvelTemplate() {
boolean found;
found = findMarvelTemplate();
assertTrue("failed to find a template named `marvel`", found);
private void assertMarvelTemplateExists() {
assertTrue("marvel template must exists", isTemplateExists("marvel"));
}
private boolean findMarvelTemplate() {
for (IndexTemplateMetaData template : client().admin().indices().prepareGetTemplates("marvel").get().getIndexTemplates()) {
if (template.getName().equals("marvel")) {
private void assertMarvelTemplateNotExists() {
assertFalse("marvel template must not exists", isTemplateExists("marvel"));
}
private boolean isTemplateExists(String templateName) {
for (IndexTemplateMetaData template : client().admin().indices().prepareGetTemplates(templateName).get().getIndexTemplates()) {
if (template.getName().equals(templateName)) {
return true;
}
}

125
marvel/src/test/java/org/elasticsearch/marvel/agent/renderer/AbstractRendererTestCase.java Normal file
View File

@ -0,0 +1,125 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.marvel.agent.renderer;
import org.elasticsearch.action.admin.indices.template.get.GetIndexTemplatesResponse;
import org.elasticsearch.cluster.metadata.IndexTemplateMetaData;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.license.plugin.LicensePlugin;
import org.elasticsearch.marvel.MarvelPlugin;
import org.elasticsearch.marvel.agent.settings.MarvelSettings;
import org.elasticsearch.node.Node;
import org.elasticsearch.plugins.Plugin;
import org.elasticsearch.test.ESIntegTestCase;
import org.elasticsearch.test.ESIntegTestCase.ClusterScope;
import java.util.Arrays;
import java.util.Collection;
import java.util.Map;
import java.util.concurrent.TimeUnit;
import static org.hamcrest.Matchers.*;
@ClusterScope(scope = ESIntegTestCase.Scope.SUITE, randomDynamicTemplates = false, transportClientRatio = 0.0)
public abstract class AbstractRendererTestCase extends ESIntegTestCase {
@Override
protected Settings nodeSettings(int nodeOrdinal) {
return Settings.builder()
.put(super.nodeSettings(nodeOrdinal))
.put(Node.HTTP_ENABLED, true)
.put(MarvelSettings.STARTUP_DELAY, "3s")
.put(MarvelSettings.INTERVAL, "1s")
.put(MarvelSettings.COLLECTORS, Strings.collectionToCommaDelimitedString(collectors()))
.build();
}
@Override
protected Collection<Class<? extends Plugin>> nodePlugins() {
return Arrays.asList(LicensePlugin.class, MarvelPlugin.class);
}
@Override
protected Collection<Class<? extends Plugin>> transportClientPlugins() {
return nodePlugins();
}
protected abstract Collection<String> collectors ();
protected void waitForMarvelDocs(final String type) throws Exception {
waitForMarvelDocs(type, 0L);
}
protected void waitForMarvelDocs(final String type, final long minCount) throws Exception {
logger.debug("--> waiting for at least [{}] marvel docs of type [{}] to be collected", minCount, type);
assertBusy(new Runnable() {
@Override
public void run() {
try {
refresh();
assertThat(client().prepareCount().setTypes(type).get().getCount(), greaterThan(minCount));
} catch (Throwable t) {
fail("exception when waiting for marvel docs: " + t.getMessage());
}
}
}, 30L, TimeUnit.SECONDS);
}
/**
* Checks if a field exist in a map of values. If the field contains a dot like 'foo.bar'
* it checks that 'foo' exists in the map of values and that it points to a sub-map. Then
* it recurses to check if 'bar' exists in the sub-map.
*/
protected void assertContains(String field, Map<String, Object> values) {
assertNotNull(field);
assertNotNull(values);
int point = field.indexOf('.');
if (point > -1) {
assertThat(point, allOf(greaterThan(0), lessThan(field.length())));
String segment = field.substring(0, point);
assertTrue(Strings.hasText(segment));
boolean fieldExists = values.containsKey(segment);
assertTrue("expecting field [" + segment + "] to be present in marvel document", fieldExists);
Object value = values.get(segment);
String next = field.substring(point + 1);
if (next.length() > 0) {
assertTrue(value instanceof Map);
assertContains(next, (Map<String, Object>) value);
} else {
assertFalse(value instanceof Map);
}
} else {
assertNotNull(values.get(field));
}
}
protected void assertMarvelTemplateExists() throws Exception {
final String marvelTemplate = "marvel";
assertBusy(new Runnable() {
@Override
public void run() {
GetIndexTemplatesResponse response = client().admin().indices().prepareGetTemplates(marvelTemplate).get();
assertNotNull(response);
boolean found = false;
for (IndexTemplateMetaData template : response.getIndexTemplates()) {
if (marvelTemplate.equals(template.getName())) {
found = true;
break;
}
}
assertTrue("Template [" + marvelTemplate + "] not found", found);
}
});
}
}

114
marvel/src/test/java/org/elasticsearch/marvel/agent/renderer/cluster/ClusterInfoIT.java Normal file
View File

@ -0,0 +1,114 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.marvel.agent.renderer.cluster;
import org.elasticsearch.Version;
import org.elasticsearch.action.get.GetResponse;
import org.elasticsearch.common.Strings;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.license.core.License;
import org.elasticsearch.marvel.agent.collector.cluster.ClusterInfoCollector;
import org.elasticsearch.marvel.agent.renderer.AbstractRendererTestCase;
import org.elasticsearch.marvel.agent.settings.MarvelSettings;
import org.junit.Test;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.Map;
import java.util.concurrent.Callable;
import java.util.concurrent.TimeUnit;
import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertHitCount;
import static org.hamcrest.Matchers.*;
public class ClusterInfoIT extends AbstractRendererTestCase {
@Override
protected Collection<String> collectors() {
return Collections.singletonList(ClusterInfoCollector.NAME);
}
@Test
public void testLicenses() throws Exception {
final String clusterUUID = client().admin().cluster().prepareState().setMetaData(true).get().getState().metaData().clusterUUID();
assertTrue(Strings.hasText(clusterUUID));
logger.debug("--> waiting for licenses collector to collect data (ie, the trial marvel license)");
GetResponse response = assertBusy(new Callable<GetResponse>() {
@Override
public GetResponse call() throws Exception {
// Checks if the marvel data index exists (it should have been created by the LicenseCollector)
assertTrue(MarvelSettings.MARVEL_DATA_INDEX_NAME + " index does not exist", client().admin().indices().prepareExists(MarvelSettings.MARVEL_DATA_INDEX_NAME).get().isExists());
ensureYellow(MarvelSettings.MARVEL_DATA_INDEX_NAME);
client().admin().indices().prepareRefresh(MarvelSettings.MARVEL_DATA_INDEX_NAME).get();
GetResponse response = client().prepareGet(MarvelSettings.MARVEL_DATA_INDEX_NAME, ClusterInfoCollector.TYPE, clusterUUID).get();
assertTrue(MarvelSettings.MARVEL_DATA_INDEX_NAME + " document does not exist", response.isExists());
return response;
}
}, 30L, TimeUnit.SECONDS);
logger.debug("--> checking that the document contains license information");
assertThat(response.getIndex(), equalTo(MarvelSettings.MARVEL_DATA_INDEX_NAME));
assertThat(response.getType(), equalTo(ClusterInfoCollector.TYPE));
assertThat(response.getId(), equalTo(clusterUUID));
Map<String, Object> source = response.getSource();
assertThat((String) source.get(ClusterInfoRenderer.Fields.CLUSTER_NAME.underscore().toString()), equalTo(cluster().getClusterName()));
assertThat((String) source.get(ClusterInfoRenderer.Fields.VERSION.underscore().toString()), equalTo(Version.CURRENT.toString()));
Object licensesList = source.get(ClusterInfoRenderer.Fields.LICENSES.underscore().toString());
assertThat(licensesList, instanceOf(List.class));
List licenses = (List) licensesList;
assertThat(licenses.size(), equalTo(1));
Map license = (Map) licenses.iterator().next();
assertThat(license, instanceOf(Map.class));
String uid = (String) license.get(ClusterInfoRenderer.Fields.UID.underscore().toString());
assertThat(uid, not(isEmptyOrNullString()));
String type = (String) license.get(ClusterInfoRenderer.Fields.TYPE.underscore().toString());
assertThat(type, not(isEmptyOrNullString()));
String status = (String) license.get(ClusterInfoRenderer.Fields.STATUS.underscore().toString());
assertThat(status, not(isEmptyOrNullString()));
Long expiryDate = (Long) license.get(ClusterInfoRenderer.Fields.EXPIRY_DATE_IN_MILLIS.underscore().toString());
assertThat(expiryDate, greaterThan(0L));
// We basically recompute the hash here
String hkey = (String) license.get(ClusterInfoRenderer.Fields.HKEY.underscore().toString());
String recalculated = ClusterInfoRenderer.hash(status, uid, type, String.valueOf(expiryDate), clusterUUID);
assertThat(hkey, equalTo(recalculated));
assertThat((String) license.get(ClusterInfoRenderer.Fields.FEATURE.underscore().toString()), not(isEmptyOrNullString()));
assertThat((String) license.get(ClusterInfoRenderer.Fields.ISSUER.underscore().toString()), not(isEmptyOrNullString()));
assertThat((String) license.get(ClusterInfoRenderer.Fields.ISSUED_TO.underscore().toString()), not(isEmptyOrNullString()));
assertThat((Long) license.get(ClusterInfoRenderer.Fields.ISSUE_DATE_IN_MILLIS.underscore().toString()), greaterThan(0L));
assertThat((Integer) license.get(ClusterInfoRenderer.Fields.MAX_NODES.underscore().toString()), greaterThan(0));
Object clusterStats = source.get(ClusterStatsRenderer.Fields.CLUSTER_STATS.underscore().toString());
assertNotNull(clusterStats);
assertThat(clusterStats, instanceOf(Map.class));
assertThat(((Map) clusterStats).size(), greaterThan(0));
logger.debug("--> check that the cluster_info is not indexed");
refresh();
assertHitCount(client().prepareCount()
.setIndices(MarvelSettings.MARVEL_DATA_INDEX_NAME)
.setTypes(ClusterInfoCollector.TYPE)
.setQuery(QueryBuilders.boolQuery()
.should(QueryBuilders.matchQuery(ClusterInfoRenderer.Fields.STATUS.underscore().toString(), License.Status.ACTIVE.label()))
.should(QueryBuilders.matchQuery(ClusterInfoRenderer.Fields.STATUS.underscore().toString(), License.Status.INVALID.label()))
.should(QueryBuilders.matchQuery(ClusterInfoRenderer.Fields.STATUS.underscore().toString(), License.Status.EXPIRED.label()))
.minimumNumberShouldMatch(1)
).get(), 0L);
}
}

47
marvel/src/test/java/org/elasticsearch/marvel/agent/renderer/cluster/ClusterStateIT.java Normal file
View File

@ -0,0 +1,47 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.marvel.agent.renderer.cluster;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.marvel.agent.collector.cluster.ClusterStateCollector;
import org.elasticsearch.marvel.agent.renderer.AbstractRendererTestCase;
import org.elasticsearch.search.SearchHit;
import org.junit.Test;
import java.util.Collection;
import java.util.Collections;
import java.util.Map;
import static org.hamcrest.Matchers.greaterThan;
public class ClusterStateIT extends AbstractRendererTestCase {
@Override
protected Collection<String> collectors() {
return Collections.singletonList(ClusterStateCollector.NAME);
}
@Test
public void testClusterState() throws Exception {
waitForMarvelDocs(ClusterStateCollector.TYPE);
logger.debug("--> searching for marvel documents of type [{}]", ClusterStateCollector.TYPE);
SearchResponse response = client().prepareSearch().setTypes(ClusterStateCollector.TYPE).get();
assertThat(response.getHits().getTotalHits(), greaterThan(0L));
logger.debug("--> checking that every document contains the expected fields");
String[] filters = ClusterStateRenderer.FILTERS;
for (SearchHit searchHit : response.getHits().getHits()) {
Map<String, Object> fields = searchHit.sourceAsMap();
for (String filter : filters) {
assertContains(filter, fields);
}
}
logger.debug("--> cluster state successfully collected");
}
}

69
marvel/src/test/java/org/elasticsearch/marvel/agent/renderer/cluster/ClusterStatsIT.java Normal file
View File

@ -0,0 +1,69 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.marvel.agent.renderer.cluster;
import org.elasticsearch.action.admin.cluster.stats.ClusterStatsResponse;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.marvel.agent.collector.cluster.ClusterStatsCollector;
import org.elasticsearch.marvel.agent.renderer.AbstractRendererTestCase;
import org.elasticsearch.marvel.agent.settings.MarvelSettings;
import org.elasticsearch.search.SearchHit;
import org.junit.Test;
import java.util.Collection;
import java.util.Collections;
import java.util.Map;
import java.util.concurrent.TimeUnit;
import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked;
import static org.hamcrest.Matchers.greaterThan;
public class ClusterStatsIT extends AbstractRendererTestCase {
@Override
protected Collection<String> collectors() {
return Collections.singletonList(ClusterStatsCollector.NAME);
}
@Test
public void testClusterStats() throws Exception {
logger.debug("--> creating some indices so that cluster stats reports data about shards");
for (int i = 0; i < randomIntBetween(1, 5); i++) {
createIndex("test-" + i);
}
logger.debug("--> wait for cluster stats to report data about shards");
assertBusy(new Runnable() {
@Override
public void run() {
ClusterStatsResponse response = client().admin().cluster().prepareClusterStats().get();
assertNotNull(response.getIndicesStats().getShards());
assertThat(response.getIndicesStats().getShards().getTotal(), greaterThan(0));
}
}, 30L, TimeUnit.SECONDS);
logger.debug("--> delete all indices in case of cluster stats documents have been indexed with no shards data");
assertAcked(client().admin().indices().prepareDelete(MarvelSettings.MARVEL_INDICES_PREFIX + "*"));
waitForMarvelDocs(ClusterStatsCollector.TYPE);
logger.debug("--> searching for marvel documents of type [{}]", ClusterStatsCollector.TYPE);
SearchResponse response = client().prepareSearch().setTypes(ClusterStatsCollector.TYPE).get();
assertThat(response.getHits().getTotalHits(), greaterThan(0L));
logger.debug("--> checking that every document contains the expected fields");
String[] filters = ClusterStatsRenderer.FILTERS;
for (SearchHit searchHit : response.getHits().getHits()) {
Map<String, Object> fields = searchHit.sourceAsMap();
for (String filter : filters) {
assertContains(filter, fields);
}
}
logger.debug("--> cluster stats successfully collected");
}
}

56
marvel/src/test/java/org/elasticsearch/marvel/agent/renderer/indices/IndexRecoveryIT.java Normal file
View File

@ -0,0 +1,56 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.marvel.agent.renderer.indices;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.marvel.agent.collector.indices.IndexRecoveryCollector;
import org.elasticsearch.marvel.agent.renderer.AbstractRendererTestCase;
import org.elasticsearch.search.SearchHit;
import org.junit.Test;
import java.util.Collection;
import java.util.Collections;
import java.util.Map;
import static org.hamcrest.Matchers.greaterThan;
public class IndexRecoveryIT extends AbstractRendererTestCase {
@Override
protected Collection<String> collectors() {
return Collections.singletonList(IndexRecoveryCollector.NAME);
}
@Test
public void testIndexRecovery() throws Exception {
logger.debug("--> creating some indices so that index recovery collector reports data");
for (int i = 0; i < randomIntBetween(1, 5); i++) {
client().prepareIndex("test-" + i, "foo").setRefresh(true).setSource("field1", "value1").get();
}
waitForMarvelDocs(IndexRecoveryCollector.TYPE);
logger.debug("--> searching for marvel documents of type [{}]", IndexRecoveryCollector.TYPE);
SearchResponse response = client().prepareSearch().setTypes(IndexRecoveryCollector.TYPE).get();
assertThat(response.getHits().getTotalHits(), greaterThan(0L));
logger.debug("--> checking that every document contains the expected fields");
String[] filters = {
IndexRecoveryRenderer.Fields.INDEX_RECOVERY.underscore().toString(),
IndexRecoveryRenderer.Fields.INDEX_RECOVERY.underscore().toString() + "." + IndexRecoveryRenderer.Fields.SHARDS.underscore().toString(),
};
for (SearchHit searchHit : response.getHits().getHits()) {
Map<String, Object> fields = searchHit.sourceAsMap();
for (String filter : filters) {
assertContains(filter, fields);
}
}
logger.debug("--> index recovery successfully collected");
}
}

11
marvel/src/test/java/org/elasticsearch/marvel/agent/renderer/indices/IndexRecoveryRendererTests.java
View File

@ -7,7 +7,6 @@ package org.elasticsearch.marvel.agent.renderer.indices;
import org.elasticsearch.Version;
import org.elasticsearch.action.admin.indices.recovery.RecoveryResponse;
import org.elasticsearch.action.admin.indices.recovery.ShardRecoveryResponse;
import org.elasticsearch.cluster.node.DiscoveryNode;
import org.elasticsearch.common.transport.DummyTransportAddress;
import org.elasticsearch.index.shard.ShardId;
@ -36,19 +35,17 @@ public class IndexRecoveryRendererTests extends ESTestCase {
DiscoveryNode source = new DiscoveryNode("node-src", DummyTransportAddress.INSTANCE, Version.CURRENT);
DiscoveryNode target = new DiscoveryNode("node-tgt", DummyTransportAddress.INSTANCE, Version.CURRENT);
List<ShardRecoveryResponse> shards = new ArrayList<>();
List<RecoveryState> shards = new ArrayList<>();
// Shard 0
ShardRecoveryResponse shard0 = new ShardRecoveryResponse();
shard0.recoveryState(new RecoveryState(new ShardId(indexName, 0), true, RecoveryState.Type.RELOCATION, source, target));
RecoveryState shard0 = new RecoveryState(new ShardId(indexName, 0), true, RecoveryState.Type.RELOCATION, source, target);
shards.add(shard0);
// Shard 1
ShardRecoveryResponse shard1 = new ShardRecoveryResponse();
shard1.recoveryState(new RecoveryState(new ShardId(indexName, 1), true, RecoveryState.Type.STORE, source, target));
RecoveryState shard1 = new RecoveryState(new ShardId(indexName, 1), true, RecoveryState.Type.STORE, source, target);
shards.add(shard1);
Map<String, List<ShardRecoveryResponse>> shardResponses = new HashMap<>(1);
Map<String, List<RecoveryState>> shardResponses = new HashMap<>(1);
shardResponses.put(indexName, shards);
RecoveryResponse recoveryResponse = new RecoveryResponse(2, 2, 2, false, shardResponses, null);

77
marvel/src/test/java/org/elasticsearch/marvel/agent/renderer/indices/IndexStatsIT.java Normal file
View File

@ -0,0 +1,77 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.marvel.agent.renderer.indices;
import org.elasticsearch.action.count.CountResponse;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.marvel.agent.collector.indices.IndexStatsCollector;
import org.elasticsearch.marvel.agent.renderer.AbstractRendererTestCase;
import org.elasticsearch.search.SearchHit;
import org.junit.Test;
import java.util.Collection;
import java.util.Collections;
import java.util.Map;
import static org.hamcrest.Matchers.greaterThan;
public class IndexStatsIT extends AbstractRendererTestCase {
@Override
protected Collection<String> collectors() {
return Collections.singletonList(IndexStatsCollector.NAME);
}
@Test
public void testIndexStats() throws Exception {
logger.debug("--> creating some indices for future index stats");
final int nbIndices = randomIntBetween(1, 5);
for (int i = 0; i < nbIndices; i++) {
createIndex("stat" + i);
}
final long[] nbDocsPerIndex = new long[nbIndices];
for (int i = 0; i < nbIndices; i++) {
nbDocsPerIndex[i] = randomIntBetween(1, 50);
for (int j = 0; j < nbDocsPerIndex[i]; j++) {
client().prepareIndex("stat" + i, "type1").setSource("num", i).get();
}
}
waitForMarvelDocs(IndexStatsCollector.TYPE);
logger.debug("--> wait for index stats collector to collect stat for each index");
assertBusy(new Runnable() {
@Override
public void run() {
for (int i = 0; i < nbIndices; i++) {
CountResponse count = client().prepareCount()
.setTypes(IndexStatsCollector.TYPE)
.setQuery(QueryBuilders.termQuery("index_stats.index", "stat" + i))
.get();
assertThat(count.getCount(), greaterThan(0L));
}
}
});
logger.debug("--> searching for marvel documents of type [{}]", IndexStatsCollector.TYPE);
SearchResponse response = client().prepareSearch().setTypes(IndexStatsCollector.TYPE).get();
assertThat(response.getHits().getTotalHits(), greaterThan(0L));
logger.debug("--> checking that every document contains the expected fields");
String[] filters = IndexStatsRenderer.FILTERS;
for (SearchHit searchHit : response.getHits().getHits()) {
Map<String, Object> fields = searchHit.sourceAsMap();
for (String filter : filters) {
assertContains(filter, fields);
}
}
logger.debug("--> index stats successfully collected");
}
}

116
marvel/src/test/java/org/elasticsearch/marvel/agent/renderer/indices/IndexStatsRendererIT.java
View File

@ -1,116 +0,0 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.marvel.agent.renderer.indices;
import org.elasticsearch.action.count.CountResponse;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.license.plugin.LicensePlugin;
import org.elasticsearch.marvel.MarvelPlugin;
import org.elasticsearch.marvel.agent.collector.indices.IndexStatsCollector;
import org.elasticsearch.marvel.agent.settings.MarvelSettings;
import org.elasticsearch.node.Node;
import org.elasticsearch.search.SearchHit;
import org.elasticsearch.test.ESIntegTestCase;
import org.elasticsearch.test.ESIntegTestCase.ClusterScope;
import org.junit.Test;
import java.util.Map;
import static org.hamcrest.Matchers.*;
@ClusterScope(scope = ESIntegTestCase.Scope.SUITE, numDataNodes = 1)
public class IndexStatsRendererIT extends ESIntegTestCase {
@Override
protected Settings nodeSettings(int nodeOrdinal) {
return Settings.builder()
.put(super.nodeSettings(nodeOrdinal))
.put("plugin.types", MarvelPlugin.class.getName() + "," + LicensePlugin.class.getName())
.put(Node.HTTP_ENABLED, true)
.put(MarvelSettings.STARTUP_DELAY, "1s")
.put(MarvelSettings.INTERVAL, "30s")
.put(MarvelSettings.COLLECTORS, IndexStatsCollector.NAME)
.build();
}
@Test
public void testIndexStats() throws Exception {
final int nbIndices = randomIntBetween(1, 5);
for (int i = 0; i < nbIndices; i++) {
createIndex("test" + i);
}
final long[] nbDocsPerIndex = new long[nbIndices];
for (int i = 0; i < nbIndices; i++) {
nbDocsPerIndex[i] = randomIntBetween(1, 50);
for (int j = 0; j < nbDocsPerIndex[i]; j++) {
client().prepareIndex("test" + i, "type1").setSource("num", i).get();
}
}
refresh();
logger.debug("--> wait for index stats collector to collect stat for each index");
assertBusy(new Runnable() {
@Override
public void run() {
for (int i = 0; i < nbIndices; i++) {
CountResponse count = client().prepareCount()
.setTypes(IndexStatsCollector.TYPE)
.setQuery(QueryBuilders.termQuery("index_stats.index", "test" + i))
.get();
assertThat(count.getCount(), greaterThan(0L));
}
}
});
logger.debug("--> get the list of filters used by the renderer");
String[] filters = IndexStatsRenderer.FILTERS;
logger.debug("--> checking that every document contains the expected fields");
SearchResponse response = client().prepareSearch().setTypes(IndexStatsCollector.TYPE).get();
for (SearchHit searchHit : response.getHits().getHits()) {
Map<String, Object> fields = searchHit.sourceAsMap();
for (String filter : filters) {
assertContains(filter, fields);
}
}
}
/**
* Checks if a field exist in a map of values. If the field contains a dot like 'foo.bar'
* it checks that 'foo' exists in the map of values and that it points to a sub-map. Then
* it recurses to check if 'bar' exists in the sub-map.
*/
private void assertContains(String field, Map<String, Object> values) {
assertNotNull(field);
assertNotNull(values);
int point = field.indexOf('.');
if (point > -1) {
assertThat(point, allOf(greaterThan(0), lessThan(field.length())));
String segment = field.substring(0, point);
assertTrue(Strings.hasText(segment));
Object value = values.get(segment);
assertNotNull(value);
String next = field.substring(point + 1);
if (next.length() > 0) {
assertTrue(value instanceof Map);
assertContains(next, (Map<String, Object>) value);
} else {
assertFalse(value instanceof Map);
}
} else {
assertNotNull(values.get(field));
}
}
}

4
marvel/src/test/java/org/elasticsearch/marvel/agent/renderer/indices/IndexStatsRendererTests.java
View File

@ -36,7 +36,7 @@ public class IndexStatsRendererTests extends ESTestCase {
CommonStats stats = new CommonStats();
stats.docs = new DocsStats(345678L, 123L);
stats.store = new StoreStats(5761573L, 0L);
stats.indexing = new IndexingStats(new IndexingStats.Stats(3L, 71L, 0L, 0L, 0L, 0L, 0L, true, 302L), null);
stats.indexing = new IndexingStats(new IndexingStats.Stats(3L, 71L, 0L, 0L, 0L, 0L, 0L, 0L, true, 302L), null);
stats.search = new SearchStats(new SearchStats.Stats(1L, 7L, 0L, 0L, 0L, 0L, 0L, 0L, 0L), 0L, null);
stats.merge = new MergeStats();
stats.merge.add(0L, 0L, 0L, 42L, 0L, 0L, 0L, 0L, 0L, 0L);
@ -52,7 +52,7 @@ public class IndexStatsRendererTests extends ESTestCase {
CommonStats stats = new CommonStats();
stats.docs = new DocsStats(345678L, randomLong());
stats.store = new StoreStats(randomLong(), randomLong());
stats.indexing = new IndexingStats(new IndexingStats.Stats(0L, 0L, 0L, 0L, 0L, 0L, 0L, true, randomLong()), null);
stats.indexing = new IndexingStats(new IndexingStats.Stats(0L, 0L, 0L, 0L, 0L, 0L, 0L, 0L, true, randomLong()), null);
return stats;
}
});

75
marvel/src/test/java/org/elasticsearch/marvel/agent/renderer/indices/IndicesStatsIT.java Normal file
View File

@ -0,0 +1,75 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.marvel.agent.renderer.indices;
import org.elasticsearch.action.count.CountResponse;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.marvel.agent.collector.indices.IndicesStatsCollector;
import org.elasticsearch.marvel.agent.renderer.AbstractRendererTestCase;
import org.elasticsearch.search.SearchHit;
import org.junit.Test;
import java.util.Collection;
import java.util.Collections;
import java.util.Map;
import static org.hamcrest.Matchers.greaterThan;
public class IndicesStatsIT extends AbstractRendererTestCase {
@Override
protected Collection<String> collectors() {
return Collections.singletonList(IndicesStatsCollector.NAME);
}
@Test
public void testIndicesStats() throws Exception {
logger.debug("--> creating some indices for future indices stats");
final int nbIndices = randomIntBetween(1, 5);
for (int i = 0; i < nbIndices; i++) {
createIndex("stat" + i);
}
final long[] nbDocsPerIndex = new long[nbIndices];
for (int i = 0; i < nbIndices; i++) {
nbDocsPerIndex[i] = randomIntBetween(1, 50);
for (int j = 0; j < nbDocsPerIndex[i]; j++) {
client().prepareIndex("stat" + i, "type1").setSource("num", i).get();
}
}
waitForMarvelDocs(IndicesStatsCollector.TYPE);
logger.debug("--> wait for indicesx stats collector to collect global stat");
assertBusy(new Runnable() {
@Override
public void run() {
for (int i = 0; i < nbIndices; i++) {
CountResponse count = client().prepareCount()
.setTypes(IndicesStatsCollector.TYPE)
.get();
assertThat(count.getCount(), greaterThan(0L));
}
}
});
logger.debug("--> searching for marvel documents of type [{}]", IndicesStatsCollector.TYPE);
SearchResponse response = client().prepareSearch().setTypes(IndicesStatsCollector.TYPE).get();
assertThat(response.getHits().getTotalHits(), greaterThan(0L));
logger.debug("--> checking that every document contains the expected fields");
String[] filters = IndicesStatsRenderer.FILTERS;
for (SearchHit searchHit : response.getHits().getHits()) {
Map<String, Object> fields = searchHit.sourceAsMap();
for (String filter : filters) {
assertContains(filter, fields);
}
}
logger.debug("--> indices stats successfully collected");
}
}

42
marvel/src/test/java/org/elasticsearch/marvel/agent/renderer/indices/IndicesStatsRendererTests.java Normal file
View File

@ -0,0 +1,42 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.marvel.agent.renderer.indices;
import org.elasticsearch.action.admin.indices.stats.IndicesStatsResponse;
import org.elasticsearch.cluster.metadata.IndexMetaData;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.marvel.agent.collector.indices.IndicesStatsMarvelDoc;
import org.elasticsearch.marvel.agent.renderer.Renderer;
import org.elasticsearch.marvel.agent.renderer.RendererTestUtils;
import org.elasticsearch.test.ESSingleNodeTestCase;
import org.elasticsearch.test.StreamsUtils;
import org.junit.Test;
public class IndicesStatsRendererTests extends ESSingleNodeTestCase {
private static final String SAMPLE_FILE = "/samples/marvel_indices_stats.json";
@Test
public void testIndexStatsRenderer() throws Exception {
createIndex("index-0");
logger.debug("--> retrieving indices stats response");
IndicesStatsResponse indicesStats = client().admin().indices().prepareStats().get();
logger.debug("--> creating the indices stats marvel document");
IndicesStatsMarvelDoc marvelDoc = new IndicesStatsMarvelDoc("test", "marvel_indices_stats", 1437580442979L, indicesStats);
logger.debug("--> rendering the document");
Renderer renderer = new IndicesStatsRenderer();
String result = RendererTestUtils.renderAsJSON(marvelDoc, renderer);
logger.debug("--> loading sample document from file {}", SAMPLE_FILE);
String expected = StreamsUtils.copyToStringFromClasspath(SAMPLE_FILE);
logger.debug("--> comparing both documents, they must have the same structure");
RendererTestUtils.assertJSONStructure(result, expected);
}
}

121
marvel/src/test/java/org/elasticsearch/marvel/agent/renderer/licenses/LicensesRendererIT.java
View File

@ -1,121 +0,0 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.marvel.agent.renderer.licenses;
import org.elasticsearch.Version;
import org.elasticsearch.action.get.GetResponse;
import org.elasticsearch.cluster.metadata.IndexMetaData;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.license.core.License;
import org.elasticsearch.license.plugin.LicensePlugin;
import org.elasticsearch.marvel.MarvelPlugin;
import org.elasticsearch.marvel.agent.collector.licenses.LicensesCollector;
import org.elasticsearch.marvel.agent.settings.MarvelSettings;
import org.elasticsearch.node.Node;
import org.elasticsearch.test.ESIntegTestCase;
import org.junit.Test;
import java.util.List;
import java.util.Map;
import java.util.concurrent.Callable;
import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertHitCount;
import static org.hamcrest.Matchers.*;
@ESIntegTestCase.ClusterScope(scope = ESIntegTestCase.Scope.SUITE, transportClientRatio = 0.0)
public class LicensesRendererIT extends ESIntegTestCase {
@Override
protected Settings nodeSettings(int nodeOrdinal) {
return Settings.builder()
.put(super.nodeSettings(nodeOrdinal))
.put("plugin.types", MarvelPlugin.class.getName() + "," + LicensePlugin.class.getName())
.put(Node.HTTP_ENABLED, true)
.put(IndexMetaData.SETTING_NUMBER_OF_SHARDS, 1)
.put(IndexMetaData.SETTING_NUMBER_OF_REPLICAS, 0)
.put(MarvelSettings.STARTUP_DELAY, "1s")
.put(MarvelSettings.COLLECTORS, LicensesCollector.NAME)
.build();
}
@Test
@AwaitsFix(bugUrl = "https://github.com/elastic/elasticsearch/issues/13017")
public void testLicenses() throws Exception {
final String clusterUUID = client().admin().cluster().prepareState().setMetaData(true).get().getState().metaData().clusterUUID();
assertTrue(Strings.hasText(clusterUUID));
logger.debug("--> waiting for licenses collector to collect data (ie, the trial marvel license)");
GetResponse response = assertBusy(new Callable<GetResponse>() {
@Override
public GetResponse call() throws Exception {
// Checks if the marvel data index exists (it should have been created by the LicenseCollector)
assertTrue(MarvelSettings.MARVEL_DATA_INDEX_NAME + " index does not exist", client().admin().indices().prepareExists(MarvelSettings.MARVEL_DATA_INDEX_NAME).get().isExists());
ensureYellow(MarvelSettings.MARVEL_DATA_INDEX_NAME);
GetResponse response = client().prepareGet(MarvelSettings.MARVEL_DATA_INDEX_NAME, LicensesCollector.TYPE, clusterUUID).get();
assertTrue(MarvelSettings.MARVEL_DATA_INDEX_NAME + " document does not exist", response.isExists());
return response;
}
});
logger.debug("--> checking that the document contains license information");
assertThat(response.getIndex(), equalTo(MarvelSettings.MARVEL_DATA_INDEX_NAME));
assertThat(response.getType(), equalTo(LicensesCollector.TYPE));
assertThat(response.getId(), equalTo(clusterUUID));
Map<String, Object> source = response.getSource();
assertThat((String) source.get(LicensesRenderer.Fields.CLUSTER_NAME.underscore().toString()), equalTo(cluster().getClusterName()));
assertThat((String) source.get(LicensesRenderer.Fields.VERSION.underscore().toString()), equalTo(Version.CURRENT.toString()));
Object licensesList = source.get(LicensesRenderer.Fields.LICENSES.underscore().toString());
assertThat(licensesList, instanceOf(List.class));
List licenses = (List) licensesList;
assertThat(licenses.size(), equalTo(1));
Map license = (Map) licenses.iterator().next();
assertThat(license, instanceOf(Map.class));
String uid = (String) ((Map) license).get(LicensesRenderer.Fields.UID.underscore().toString());
assertThat(uid, not(isEmptyOrNullString()));
String type = (String) ((Map) license).get(LicensesRenderer.Fields.TYPE.underscore().toString());
assertThat(type, not(isEmptyOrNullString()));
String status = (String) ((Map) license).get(LicensesRenderer.Fields.STATUS.underscore().toString());
assertThat(status, not(isEmptyOrNullString()));
Long expiryDate = (Long) ((Map) license).get(LicensesRenderer.Fields.EXPIRY_DATE_IN_MILLIS.underscore().toString());
assertThat(expiryDate, greaterThan(0L));
// We basically recompute the hash here
String hkey = (String) ((Map) license).get(LicensesRenderer.Fields.HKEY.underscore().toString());
String recalculated = LicensesRenderer.hash(status, uid, type, String.valueOf(expiryDate), clusterUUID);
assertThat(hkey, equalTo(recalculated));
assertThat((String) ((Map) license).get(LicensesRenderer.Fields.FEATURE.underscore().toString()), not(isEmptyOrNullString()));
assertThat((String) ((Map) license).get(LicensesRenderer.Fields.ISSUER.underscore().toString()), not(isEmptyOrNullString()));
assertThat((String) ((Map) license).get(LicensesRenderer.Fields.ISSUED_TO.underscore().toString()), not(isEmptyOrNullString()));
assertThat((Long) ((Map) license).get(LicensesRenderer.Fields.ISSUE_DATE_IN_MILLIS.underscore().toString()), greaterThan(0L));
assertThat((Integer) ((Map) license).get(LicensesRenderer.Fields.MAX_NODES.underscore().toString()), greaterThan(0));
logger.debug("--> check that the cluster_licenses is not indexed");
refresh();
assertHitCount(client().prepareCount()
.setIndices(MarvelSettings.MARVEL_DATA_INDEX_NAME)
.setTypes(LicensesCollector.TYPE)
.setQuery(QueryBuilders.boolQuery()
.should(QueryBuilders.matchQuery(LicensesRenderer.Fields.STATUS.underscore().toString(), License.Status.ACTIVE.label()))
.should(QueryBuilders.matchQuery(LicensesRenderer.Fields.STATUS.underscore().toString(), License.Status.INVALID.label()))
.should(QueryBuilders.matchQuery(LicensesRenderer.Fields.STATUS.underscore().toString(), License.Status.EXPIRED.label()))
.minimumNumberShouldMatch(1)
).get(), 0L);
}
}

53
marvel/src/test/java/org/elasticsearch/marvel/agent/renderer/node/NodeStatsIT.java Normal file
View File

@ -0,0 +1,53 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.marvel.agent.renderer.node;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.marvel.agent.collector.node.NodeStatsCollector;
import org.elasticsearch.marvel.agent.renderer.AbstractRendererTestCase;
import org.elasticsearch.search.SearchHit;
import org.junit.Test;
import java.util.Collection;
import java.util.Collections;
import java.util.Map;
import static org.hamcrest.Matchers.greaterThan;
public class NodeStatsIT extends AbstractRendererTestCase {
@Override
protected Collection<String> collectors() {
return Collections.singletonList(NodeStatsCollector.NAME);
}
@Test
public void testNodeStats() throws Exception {
logger.debug("--> creating some indices for future node stats");
final int numDocs = between(50, 150);
for (int i = 0; i < numDocs; i++) {
client().prepareIndex("test", "foo").setSource("value", randomInt()).get();
}
waitForMarvelDocs(NodeStatsCollector.TYPE);
logger.debug("--> searching for marvel documents of type [{}]", NodeStatsCollector.TYPE);
SearchResponse response = client().prepareSearch().setTypes(NodeStatsCollector.TYPE).get();
assertThat(response.getHits().getTotalHits(), greaterThan(0L));
logger.debug("--> checking that every document contains the expected fields");
String[] filters = NodeStatsRenderer.FILTERS;
for (SearchHit searchHit : response.getHits().getHits()) {
Map<String, Object> fields = searchHit.sourceAsMap();
for (String filter : filters) {
assertContains(filter, fields);
}
}
logger.debug("--> node stats successfully collected");
}
}

1
marvel/src/test/java/org/elasticsearch/marvel/agent/renderer/node/NodeStatsRendererTests.java
View File

@ -19,7 +19,6 @@ public class NodeStatsRendererTests extends ESSingleNodeTestCase {
private static final String SAMPLE_FILE = "/samples/marvel_node_stats.json";
@Test
@AwaitsFix(bugUrl = "https://github.com/elastic/x-plugins/issues/367")
public void testNodeStatsRenderer() throws Exception {
createIndex("index-0");

22
marvel/src/test/java/org/elasticsearch/marvel/agent/settings/MarvelSettingsTests.java
View File

@ -11,9 +11,13 @@ import org.elasticsearch.common.unit.TimeValue;
import org.elasticsearch.license.plugin.LicensePlugin;
import org.elasticsearch.marvel.MarvelPlugin;
import org.elasticsearch.node.Node;
import org.elasticsearch.plugins.Plugin;
import org.elasticsearch.test.ESIntegTestCase;
import org.junit.Test;
import java.util.Arrays;
import java.util.Collection;
import static org.elasticsearch.test.hamcrest.ElasticsearchAssertions.assertAcked;
import static org.hamcrest.Matchers.*;
@ -35,12 +39,21 @@ public class MarvelSettingsTests extends ESIntegTestCase {
protected Settings nodeSettings(int nodeOrdinal) {
return Settings.builder()
.put(super.nodeSettings(nodeOrdinal))
.put("plugin.types", MarvelPlugin.class.getName() + "," + LicensePlugin.class.getName())
.put(Node.HTTP_ENABLED, true)
.put(marvelSettings())
.build();
}
@Override
protected Collection<Class<? extends Plugin>> nodePlugins() {
return Arrays.asList(LicensePlugin.class, MarvelPlugin.class);
}
@Override
protected Collection<Class<? extends Plugin>> transportClientPlugins() {
return nodePlugins();
}
private Settings marvelSettings() {
return Settings.builder()
.put(MarvelSettings.STARTUP_DELAY, startUp)
@ -58,6 +71,11 @@ public class MarvelSettingsTests extends ESIntegTestCase {
@Test
public void testMarvelSettingService() throws Exception {
logger.info("--> printing marvel settings values");
for (MarvelSetting setting : MarvelSettings.settings()) {
logger.info("\t{}", setting);
}
logger.info("--> testing marvel settings service initialization");
for (final MarvelSettings marvelSettings : internalCluster().getInstances(MarvelSettings.class)) {
assertThat(marvelSettings.startUpDelay().millis(), equalTo(startUp.millis()));
@ -139,7 +157,7 @@ public class MarvelSettingsTests extends ESIntegTestCase {
}
private TimeValue randomTimeValue() {
return TimeValue.parseTimeValue(randomFrom("1s", "10s", "30s", "1m", "30m", "1h"), null, getClass().getSimpleName());
return TimeValue.parseTimeValue(randomFrom("30m", "1h", "3h", "5h", "7h", "10h", "1d"), null, getClass().getSimpleName());
}
private String[] randomStringArray() {

22
marvel/src/test/java/org/elasticsearch/marvel/license/LicenseIntegrationTests.java
View File

@ -7,7 +7,6 @@ package org.elasticsearch.marvel.license;
import com.carrotsearch.randomizedtesting.RandomizedTest;
import com.carrotsearch.randomizedtesting.SysGlobals;
import com.google.common.collect.ImmutableSet;
import org.elasticsearch.common.component.AbstractComponent;
import org.elasticsearch.common.inject.AbstractModule;
import org.elasticsearch.common.inject.Inject;
@ -23,25 +22,22 @@ import org.elasticsearch.test.ESIntegTestCase;
import org.elasticsearch.test.ESIntegTestCase.ClusterScope;
import org.junit.Test;
import java.util.ArrayList;
import java.util.Collection;
import java.util.Collections;
import java.util.List;
import java.util.*;
import static org.elasticsearch.test.ESIntegTestCase.Scope.SUITE;
import static org.hamcrest.Matchers.equalTo;
import static org.hamcrest.Matchers.greaterThan;
import static org.hamcrest.Matchers.is;
import static org.hamcrest.Matchers.*;
@ClusterScope(scope = SUITE, transportClientRatio = 0, numClientNodes = 0)
public class LicenseIntegrationTests extends ESIntegTestCase {
@Override
protected Settings nodeSettings(int nodeOrdinal) {
return Settings.settingsBuilder()
.put(super.nodeSettings(nodeOrdinal))
.put("plugin.types", MarvelPlugin.class.getName() + "," + MockLicensePlugin.class.getName())
.build();
protected Collection<Class<? extends Plugin>> nodePlugins() {
return Arrays.asList(MockLicensePlugin.class, MarvelPlugin.class);
}
@Override
protected Collection<Class<? extends Plugin>> transportClientPlugins() {
return nodePlugins();
}
@Test

1
marvel/src/test/resources/samples/marvel_cluster_stats.json
View File

@ -3,6 +3,7 @@
"timestamp": "2015-07-22T15:54:02.979Z",
"cluster_stats": {
"indices": {
"count": 10,
"shards": {
"total": 1,
"index": {

44
marvel/src/test/resources/samples/marvel_indices_stats.json Normal file
View File

@ -0,0 +1,44 @@
{
"cluster_uuid": "dsFPzYRyQCe6cq48a0wxkQ",
"timestamp": "2015-07-22T15:54:02.979Z",
"indices_stats": {
"_all": {
"primaries": {
"docs": {
"count": 0
},
"store": {
"size_in_bytes": 127
},
"indexing": {
"index_total": 1,
"index_time_in_millis": 286,
"is_throttled": false,
"throttle_time_in_millis": 0
},
"search": {
"query_total": 0,
"query_time_in_millis": 0
}
},
"total": {
"docs": {
"count": 0
},
"store": {
"size_in_bytes": 127
},
"indexing": {
"index_total": 1,
"index_time_in_millis": 286,
"is_throttled": false,
"throttle_time_in_millis": 0
},
"search": {
"query_total": 0,
"query_time_in_millis": 0
}
}
}
}
}

6
marvel/src/test/resources/samples/marvel_node_stats.json
View File

@ -48,14 +48,14 @@
}
},
"thread_pool": {
"bulk": {
"rejected": 0
},
"index": {
"rejected": 0
},
"search": {
"rejected": 0
},
"bulk": {
"rejected": 0
}
},
"fs": {

6
pom.xml
View File

@ -8,14 +8,14 @@
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>x-plugins</artifactId>
<version>2.1.0-SNAPSHOT</version>
<version>3.0.0-SNAPSHOT</version>
<packaging>pom</packaging>
<name>X-Plugins: Parent POM</name>
<parent>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>plugins</artifactId>
<version>2.1.0-SNAPSHOT</version>
<version>3.0.0-SNAPSHOT</version>
</parent>
<properties>
@ -73,7 +73,7 @@
<dependency>
<groupId>org.elasticsearch</groupId>
<artifactId>securemock</artifactId>
<version>1.0-SNAPSHOT</version>
<version>1.1</version>
<scope>test</scope>
</dependency>
</dependencies>

7
qa/pom.xml
View File

@ -14,7 +14,7 @@
<parent>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>x-plugins</artifactId>
<version>2.1.0-SNAPSHOT</version>
<version>3.0.0-SNAPSHOT</version>
</parent>
<properties>
@ -302,6 +302,7 @@
<systemProperties>
<!-- use external cluster -->
<tests.cluster>127.0.0.1:${integ.transport.port}</tests.cluster>
<tests.rest.cluster>127.0.0.1:${integ.http.port}</tests.rest.cluster>
</systemProperties>
</configuration>
</execution>
@ -314,10 +315,14 @@
<modules>
<module>smoke-test-plugins</module>
<!-- Disabled until 'openssl' is available on the Windows build machines
See: https://github.com/elastic/infra/issues/331
<module>smoke-test-plugins-ssl</module>-->
<module>shield-core-rest-tests</module>
<module>smoke-test-watcher-with-shield</module>
<module>shield-example-realm</module>
<module>shield-tribe-node-tests</module>
<module>shield-client-tests</module>
<module>shield-audit-tests</module>
</modules>
<profiles>

90
qa/shield-audit-tests/integration-tests.xml Normal file
View File

@ -0,0 +1,90 @@
<?xml version="1.0"?>
<!--
~ ELASTICSEARCH CONFIDENTIAL
~ __________________
~
~ [2014] Elasticsearch Incorporated. All Rights Reserved.
~
~ NOTICE: All information contained herein is, and remains
~ the property of Elasticsearch Incorporated and its suppliers,
~ if any. The intellectual and technical concepts contained
~ herein are proprietary to Elasticsearch Incorporated
~ and its suppliers and may be covered by U.S. and Foreign Patents,
~ patents in process, and are protected by trade secret or copyright law.
~ Dissemination of this information or reproduction of this material
~ is strictly forbidden unless prior written permission is obtained
~ from Elasticsearch Incorporated.
-->
<project name="shield-client-tests"
xmlns:ac="antlib:net.sf.antcontrib">
<import file="${elasticsearch.integ.antfile.default}"/>
<!-- redefined to work with auth -->
<macrodef name="waitfor-elasticsearch">
<attribute name="port"/>
<attribute name="timeoutproperty"/>
<sequential>
<echo>Waiting for elasticsearch to become available on port @{port}...</echo>
<waitfor maxwait="30" maxwaitunit="second"
checkevery="500" checkeveryunit="millisecond"
timeoutproperty="@{timeoutproperty}">
<socket server="127.0.0.1" port="@{port}"/>
</waitfor>
</sequential>
</macrodef>
<target name="start-external-cluster-with-plugin" depends="setup-workspace">
<ac:for list="${xplugins.list}" param="xplugin.name">
<sequential>
<fail message="Expected @{xplugin.name}-${version}.zip as a dependency, but could not be found in ${integ.deps}/plugins}">
<condition>
<not>
<available file="${integ.deps}/plugins/@{xplugin.name}-${elasticsearch.version}.zip" />
</not>
</condition>
</fail>
</sequential>
</ac:for>
<ac:for param="file">
<path>
<fileset dir="${integ.deps}/plugins"/>
</path>
<sequential>
<local name="plugin.name"/>
<convert-plugin-name file="@{file}" outputproperty="plugin.name"/>
<install-plugin name="${plugin.name}" file="@{file}"/>
</sequential>
</ac:for>
<local name="home"/>
<property name="home" location="${integ.scratch}/elasticsearch-${elasticsearch.version}"/>
<echo>Adding shield users...</echo>
<run-script script="${home}/bin/shield/esusers">
<nested>
<arg value="useradd"/>
<arg value="test_user"/>
<arg value="-p"/>
<arg value="changeme"/>
<arg value="-r"/>
<arg value="admin"/>
</nested>
</run-script>
<startup-elasticsearch>
<additional-args>
<arg value="-Des.shield.audit.enabled=true"/>
<arg value="-Des.shield.audit.outputs=index"/>
</additional-args>
</startup-elasticsearch>
<echo>Checking we can connect with basic auth on port ${integ.http.port}...</echo>
<local name="temp.file"/>
<tempfile property="temp.file" destdir="${java.io.tmpdir}"/>
<get src="http://127.0.0.1:${integ.http.port}" dest="${temp.file}"
username="test_user" password="changeme" verbose="true" retries="10"/>
</target>
</project>

145
qa/shield-audit-tests/pom.xml Normal file
View File

@ -0,0 +1,145 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>x-plugins-qa</artifactId>
<groupId>org.elasticsearch.qa</groupId>
<version>3.0.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>shield-audit-tests</artifactId>
<name>QA: Shield index audit tests</name>
<description>Tests the Shield Index Audit that works in a cluster properly</description>
<properties>
<skip.unit.tests>true</skip.unit.tests>
<tests.rest.load_packaged>false</tests.rest.load_packaged>
<elasticsearch.integ.antfile>${project.basedir}/integration-tests.xml</elasticsearch.integ.antfile>
<xplugins.list>license,shield</xplugins.list>
</properties>
<dependencies>
<dependency>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>shield</artifactId>
<version>${elasticsearch.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>integ-setup-dependencies</id>
<phase>pre-integration-test</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
<skip>${skip.integ.tests}</skip>
<useBaseVersion>true</useBaseVersion>
<outputDirectory>${integ.deps}/plugins</outputDirectory>
<artifactItems>
<!-- elasticsearch distribution -->
<artifactItem>
<groupId>org.elasticsearch.distribution.zip</groupId>
<artifactId>elasticsearch</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
<outputDirectory>${integ.deps}</outputDirectory>
</artifactItem>
<!-- commercial plugins -->
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>license</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>shield</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
<!-- integration tests -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<!-- start up external cluster -->
<execution>
<id>integ-setup</id>
<phase>pre-integration-test</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<target>
<ant antfile="${elasticsearch.integ.antfile}" target="start-external-cluster-with-plugin">
<property name="tests.jvm.argline" value="${tests.jvm.argline}"/>
<property name="plugins.dir" value="${plugins.dir}"/>
<property name="xplugins.list" value="${xplugins.list}"/>
</ant>
</target>
<skip>${skip.integ.tests}</skip>
</configuration>
</execution>
<!-- shut down external cluster -->
<execution>
<id>integ-teardown</id>
<phase>post-integration-test</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<target>
<ant antfile="${elasticsearch.integ.antfile}" target="stop-external-cluster"/>
</target>
<skip>${skip.integ.tests}</skip>
</configuration>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>ant-contrib</groupId>
<artifactId>ant-contrib</artifactId>
<version>1.0b3</version>
<exclusions>
<exclusion>
<groupId>ant</groupId>
<artifactId>ant</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.ant</groupId>
<artifactId>ant-nodeps</artifactId>
<version>1.8.1</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
</project>

65
qa/shield-audit-tests/src/test/java/org/elasticsearch/shield/audit/IndexAuditIT.java Normal file
View File

@ -0,0 +1,65 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.shield.audit;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.plugins.Plugin;
import org.elasticsearch.shield.ShieldPlugin;
import org.elasticsearch.shield.authc.support.SecuredString;
import org.elasticsearch.shield.authc.support.UsernamePasswordToken;
import org.elasticsearch.test.ESIntegTestCase;
import org.elasticsearch.test.rest.client.http.HttpResponse;
import org.junit.Test;
import java.util.Collection;
import java.util.Collections;
import java.util.concurrent.TimeUnit;
import static org.hamcrest.Matchers.greaterThan;
import static org.hamcrest.Matchers.is;
public class IndexAuditIT extends ESIntegTestCase {
private static final String USER = "test_user";
private static final String PASS = "changeme";
@Test
public void testShieldIndexAuditTrailWorking() throws Exception {
HttpResponse response = httpClient().path("/_cluster/health")
.addHeader("Authorization", UsernamePasswordToken.basicAuthHeaderValue(USER, new SecuredString(PASS.toCharArray())))
.execute();
assertThat(response.getStatusCode(), is(200));
boolean found = awaitBusy(() -> {
if (client().admin().cluster().prepareState().get().getState().getMetaData().getIndices().size() < 1) {
return false;
}
client().admin().indices().prepareRefresh().get();
return client().prepareSearch(".shield_audit_log*").setQuery(QueryBuilders.matchQuery("principal", USER)).get().getHits().totalHits() > 0;
}, 5L, TimeUnit.SECONDS);
assertThat(found, is(true));
SearchResponse searchResponse = client().prepareSearch(".shield_audit_log*").setQuery(QueryBuilders.matchQuery("principal", USER)).addField("principal").get();
assertThat(searchResponse.getHits().getHits().length, greaterThan(0));
assertThat((String) searchResponse.getHits().getAt(0).field("principal").getValue(), is(USER));
}
@Override
protected Settings externalClusterClientSettings() {
return Settings.builder()
.put("shield.user", USER + ":" + PASS)
.build();
}
@Override
protected Collection<Class<? extends Plugin>> transportClientPlugins() {
return Collections.<Class<? extends Plugin>>singleton(ShieldPlugin.class);
}
}

95
qa/shield-client-tests/integration-tests.xml Normal file
View File

@ -0,0 +1,95 @@
<?xml version="1.0"?>
<!--
~ ELASTICSEARCH CONFIDENTIAL
~ __________________
~
~ [2014] Elasticsearch Incorporated. All Rights Reserved.
~
~ NOTICE: All information contained herein is, and remains
~ the property of Elasticsearch Incorporated and its suppliers,
~ if any. The intellectual and technical concepts contained
~ herein are proprietary to Elasticsearch Incorporated
~ and its suppliers and may be covered by U.S. and Foreign Patents,
~ patents in process, and are protected by trade secret or copyright law.
~ Dissemination of this information or reproduction of this material
~ is strictly forbidden unless prior written permission is obtained
~ from Elasticsearch Incorporated.
-->
<project name="shield-client-tests"
xmlns:ac="antlib:net.sf.antcontrib">
<import file="${elasticsearch.integ.antfile.default}"/>
<!-- redefined to work with auth -->
<macrodef name="waitfor-elasticsearch">
<attribute name="port"/>
<attribute name="timeoutproperty"/>
<sequential>
<echo>Waiting for elasticsearch to become available on port @{port}...</echo>
<waitfor maxwait="30" maxwaitunit="second"
checkevery="500" checkeveryunit="millisecond"
timeoutproperty="@{timeoutproperty}">
<socket server="127.0.0.1" port="@{port}"/>
</waitfor>
</sequential>
</macrodef>
<target name="start-external-cluster-with-plugin" depends="setup-workspace">
<ac:for list="${xplugins.list}" param="xplugin.name">
<sequential>
<fail message="Expected @{xplugin.name}-${version}.zip as a dependency, but could not be found in ${integ.deps}/plugins}">
<condition>
<not>
<available file="${integ.deps}/plugins/@{xplugin.name}-${elasticsearch.version}.zip" />
</not>
</condition>
</fail>
</sequential>
</ac:for>
<ac:for param="file">
<path>
<fileset dir="${integ.deps}/plugins"/>
</path>
<sequential>
<local name="plugin.name"/>
<convert-plugin-name file="@{file}" outputproperty="plugin.name"/>
<install-plugin name="${plugin.name}" file="@{file}"/>
</sequential>
</ac:for>
<local name="home"/>
<property name="home" location="${integ.scratch}/elasticsearch-${elasticsearch.version}"/>
<echo>Adding shield users...</echo>
<run-script script="${home}/bin/shield/esusers">
<nested>
<arg value="useradd"/>
<arg value="test_user"/>
<arg value="-p"/>
<arg value="changeme"/>
<arg value="-r"/>
<arg value="admin"/>
</nested>
</run-script>
<run-script script="${home}/bin/shield/esusers">
<nested>
<arg value="useradd"/>
<arg value="transport"/>
<arg value="-p"/>
<arg value="changeme"/>
<arg value="-r"/>
<arg value="transport_client"/>
</nested>
</run-script>
<startup-elasticsearch/>
<echo>Checking we can connect with basic auth on port ${integ.http.port}...</echo>
<local name="temp.file"/>
<tempfile property="temp.file" destdir="${java.io.tmpdir}"/>
<get src="http://127.0.0.1:${integ.http.port}" dest="${temp.file}"
username="test_user" password="changeme" verbose="true" retries="10"/>
</target>
</project>

144
qa/shield-client-tests/pom.xml Normal file
View File

@ -0,0 +1,144 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>x-plugins-qa</artifactId>
<groupId>org.elasticsearch.qa</groupId>
<version>3.0.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<artifactId>shield-client-tests</artifactId>
<name>QA: Shield transport client tests</name>
<description>Run tests with a Transport Client for communication with a Shield enabled cluster</description>
<properties>
<skip.unit.tests>true</skip.unit.tests>
<tests.rest.load_packaged>false</tests.rest.load_packaged>
<elasticsearch.integ.antfile>${project.basedir}/integration-tests.xml</elasticsearch.integ.antfile>
<xplugins.list>license,shield</xplugins.list>
</properties>
<dependencies>
<dependency>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>shield</artifactId>
<version>${elasticsearch.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>integ-setup-dependencies</id>
<phase>pre-integration-test</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
<skip>${skip.integ.tests}</skip>
<useBaseVersion>true</useBaseVersion>
<outputDirectory>${integ.deps}/plugins</outputDirectory>
<artifactItems>
<!-- elasticsearch distribution -->
<artifactItem>
<groupId>org.elasticsearch.distribution.zip</groupId>
<artifactId>elasticsearch</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
<outputDirectory>${integ.deps}</outputDirectory>
</artifactItem>
<!-- commercial plugins -->
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>license</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>shield</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
<!-- integration tests -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<!-- start up external cluster -->
<execution>
<id>integ-setup</id>
<phase>pre-integration-test</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<target>
<ant antfile="${elasticsearch.integ.antfile}" target="start-external-cluster-with-plugin">
<property name="tests.jvm.argline" value="${tests.jvm.argline}"/>
<property name="plugins.dir" value="${plugins.dir}"/>
<property name="xplugins.list" value="${xplugins.list}"/>
</ant>
</target>
<skip>${skip.integ.tests}</skip>
</configuration>
</execution>
<!-- shut down external cluster -->
<execution>
<id>integ-teardown</id>
<phase>post-integration-test</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<target>
<ant antfile="${elasticsearch.integ.antfile}" target="stop-external-cluster"/>
</target>
<skip>${skip.integ.tests}</skip>
</configuration>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>ant-contrib</groupId>
<artifactId>ant-contrib</artifactId>
<version>1.0b3</version>
<exclusions>
<exclusion>
<groupId>ant</groupId>
<artifactId>ant</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.ant</groupId>
<artifactId>ant-nodeps</artifactId>
<version>1.8.1</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
</project>

120
qa/shield-client-tests/src/test/java/org/elasticsearch/shield/qa/ShieldTransportClientIT.java Normal file
View File

@ -0,0 +1,120 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.shield.qa;
import org.elasticsearch.ElasticsearchSecurityException;
import org.elasticsearch.action.admin.cluster.health.ClusterHealthResponse;
import org.elasticsearch.action.admin.cluster.node.info.NodeInfo;
import org.elasticsearch.action.admin.cluster.node.info.NodesInfoResponse;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.TransportAddress;
import org.elasticsearch.plugins.Plugin;
import org.elasticsearch.shield.ShieldPlugin;
import org.elasticsearch.shield.authc.support.SecuredString;
import org.elasticsearch.test.ESIntegTestCase;
import org.junit.Test;
import java.util.Collection;
import java.util.Collections;
import java.util.concurrent.TimeUnit;
import static org.elasticsearch.shield.authc.support.UsernamePasswordToken.basicAuthHeaderValue;
import static org.hamcrest.Matchers.*;
/**
* Integration tests that test a transport client with Shield being loaded that connect to an external cluster
*/
public class ShieldTransportClientIT extends ESIntegTestCase {
static final String ADMIN_USER_PW = "test_user:changeme";
static final String TRANSPORT_USER_PW = "transport:changeme";
@Override
protected Settings externalClusterClientSettings() {
return Settings.builder()
.put("shield.user", ADMIN_USER_PW)
.build();
}
@Override
protected Collection<Class<? extends Plugin>> transportClientPlugins() {
return Collections.singletonList(ShieldPlugin.class);
}
@Test
public void testThatTransportClientWithoutAuthenticationDoesNotWork() throws Exception {
try (TransportClient client = transportClient(Settings.EMPTY)) {
boolean connected = awaitBusy(() -> {
return client.connectedNodes().size() > 0;
}, 5L, TimeUnit.SECONDS);
assertThat(connected, is(false));
}
}
@Test
public void testThatTransportClientAuthenticationWithTransportClientRole() throws Exception {
Settings settings = Settings.builder()
.put("shield.user", TRANSPORT_USER_PW)
.build();
try (TransportClient client = transportClient(settings)) {
boolean connected = awaitBusy(() -> {
return client.connectedNodes().size() > 0;
}, 5L, TimeUnit.SECONDS);
assertThat(connected, is(true));
// this checks that the transport client is really running in a limited state
try {
client.admin().cluster().prepareHealth().get();
fail("the transport user should not be be able to get health!");
} catch (ElasticsearchSecurityException e) {
assertThat(e.toString(), containsString("unauthorized"));
}
}
}
@Test
public void testTransportClientWithAdminUser() throws Exception {
final boolean useTransportUser = randomBoolean();
Settings settings = Settings.builder()
.put("shield.user", useTransportUser ? TRANSPORT_USER_PW : ADMIN_USER_PW)
.build();
try (TransportClient client = transportClient(settings)) {
boolean connected = awaitBusy(() -> {
return client.connectedNodes().size() > 0;
}, 5L, TimeUnit.SECONDS);
assertThat(connected, is(true));
// this checks that the transport client is really running in a limited state
ClusterHealthResponse response;
if (useTransportUser) {
response = client.admin().cluster().prepareHealth().putHeader("Authorization", basicAuthHeaderValue("test_user", new SecuredString("changeme".toCharArray()))).get();
} else {
response = client.admin().cluster().prepareHealth().get();
}
assertThat(response.isTimedOut(), is(false));
}
}
TransportClient transportClient(Settings extraSettings) {
NodesInfoResponse nodeInfos = client().admin().cluster().prepareNodesInfo().get();
NodeInfo[] nodes = nodeInfos.getNodes();
assertTrue(nodes.length > 0);
TransportAddress publishAddress = randomFrom(nodes).getTransport().address().publishAddress();
String clusterName = nodeInfos.getClusterNameAsString();
Settings settings = Settings.builder()
.put(extraSettings)
.put("cluster.name", clusterName)
.build();
return TransportClient.builder().settings(settings).addPlugin(ShieldPlugin.class).build().addTransportAddress(publishAddress);
}
}

2
qa/shield-core-rest-tests/pom.xml
View File

@ -8,7 +8,7 @@
<parent>
<groupId>org.elasticsearch.qa</groupId>
<artifactId>x-plugins-qa</artifactId>
<version>2.1.0-SNAPSHOT</version>
<version>3.0.0-SNAPSHOT</version>
</parent>
<!--

10
qa/shield-core-rest-tests/src/test/java/org/elasticsearch/shield/RestIT.java
View File

@ -10,13 +10,15 @@ import com.carrotsearch.randomizedtesting.annotations.ParametersFactory;
import org.elasticsearch.client.support.Headers;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.shield.ShieldPlugin;
import org.elasticsearch.plugins.Plugin;
import org.elasticsearch.shield.authc.support.SecuredString;
import org.elasticsearch.test.rest.ESRestTestCase;
import org.elasticsearch.test.rest.RestTestCandidate;
import org.elasticsearch.test.rest.parser.RestTestParseException;
import java.io.IOException;
import java.util.Collection;
import java.util.Collections;
import static org.elasticsearch.shield.authc.support.UsernamePasswordToken.basicAuthHeaderValue;
@ -46,9 +48,13 @@ public class RestIT extends ESRestTestCase {
protected Settings externalClusterClientSettings() {
return Settings.builder()
.put("shield.user", USER + ":" + PASS)
.put("plugin.types", ShieldPlugin.class.getName())
.build();
}
@Override
protected Collection<Class<? extends Plugin>> transportClientPlugins() {
return Collections.<Class<? extends Plugin>>singleton(ShieldPlugin.class);
}
}

5
qa/shield-example-realm/pom.xml
View File

@ -5,7 +5,7 @@
<parent>
<artifactId>x-plugins-qa</artifactId>
<groupId>org.elasticsearch.qa</groupId>
<version>2.1.0-SNAPSHOT</version>
<version>3.0.0-SNAPSHOT</version>
</parent>
<modelVersion>4.0.0</modelVersion>
@ -18,6 +18,7 @@
<elasticsearch.plugin.isolated>false</elasticsearch.plugin.isolated>
<elasticsearch.integ.antfile>${project.basedir}/integration-tests.xml</elasticsearch.integ.antfile>
<xplugins.list>license,shield,shield-example-realm</xplugins.list>
<xlint.options>-Xlint:-rawtypes</xlint.options>
</properties>
<dependencies>
@ -120,4 +121,4 @@
</plugin>
</plugins>
</build>
</project>
</project>

10
qa/shield-example-realm/src/main/java/org/elasticsearch/example/realm/CustomRealm.java
View File

@ -66,4 +66,14 @@ public class CustomRealm extends Realm<UsernamePasswordToken> {
}
return null;
}
@Override
public User lookupUser(String username) {
return null;
}
@Override
public boolean userLookupSupported() {
return false;
}
}

18
qa/shield-example-realm/src/test/java/org/elasticsearch/example/realm/CustomRealmIT.java
View File

@ -13,11 +13,15 @@ import org.elasticsearch.client.transport.NoNodeAvailableException;
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.TransportAddress;
import org.elasticsearch.plugins.Plugin;
import org.elasticsearch.shield.ShieldPlugin;
import org.elasticsearch.test.ESIntegTestCase;
import org.elasticsearch.test.rest.client.http.HttpResponse;
import org.junit.Test;
import java.util.Collection;
import java.util.Collections;
import static org.hamcrest.Matchers.*;
/**
@ -28,12 +32,16 @@ public class CustomRealmIT extends ESIntegTestCase {
@Override
protected Settings externalClusterClientSettings() {
return Settings.builder()
.put("plugin.types", ShieldPlugin.class.getName())
.put(Headers.PREFIX + "." + CustomRealm.USER_HEADER, CustomRealm.KNOWN_USER)
.put(Headers.PREFIX + "." + CustomRealm.PW_HEADER, CustomRealm.KNOWN_PW)
.build();
}
@Override
protected Collection<Class<? extends Plugin>> transportClientPlugins() {
return Collections.<Class<? extends Plugin>>singleton(ShieldPlugin.class);
}
@Test
public void testHttpConnectionWithNoAuthentication() throws Exception {
HttpResponse response = httpClient().path("/").execute();
@ -60,13 +68,11 @@ public class CustomRealmIT extends ESIntegTestCase {
String clusterName = nodeInfos.getClusterNameAsString();
Settings settings = Settings.builder()
.put("path.home", createTempDir())
.put("plugin.types", ShieldPlugin.class.getName())
.put("cluster.name", clusterName)
.put(Headers.PREFIX + "." + CustomRealm.USER_HEADER, CustomRealm.KNOWN_USER)
.put(Headers.PREFIX + "." + CustomRealm.PW_HEADER, CustomRealm.KNOWN_PW)
.build();
try (TransportClient client = TransportClient.builder().settings(settings).build()) {
try (TransportClient client = TransportClient.builder().settings(settings).addPlugin(ShieldPlugin.class).build()) {
client.addTransportAddress(publishAddress);
ClusterHealthResponse response = client.admin().cluster().prepareHealth().execute().actionGet();
assertThat(response.isTimedOut(), is(false));
@ -82,13 +88,11 @@ public class CustomRealmIT extends ESIntegTestCase {
String clusterName = nodeInfos.getClusterNameAsString();
Settings settings = Settings.builder()
.put("path.home", createTempDir())
.put("plugin.types", ShieldPlugin.class.getName())
.put("cluster.name", clusterName)
.put(Headers.PREFIX + "." + CustomRealm.USER_HEADER, CustomRealm.KNOWN_USER + randomAsciiOfLength(1))
.put(Headers.PREFIX + "." + CustomRealm.PW_HEADER, CustomRealm.KNOWN_PW)
.build();
try (TransportClient client = TransportClient.builder().settings(settings).build()) {
try (TransportClient client = TransportClient.builder().addPlugin(ShieldPlugin.class).settings(settings).build()) {
client.addTransportAddress(publishAddress);
client.admin().cluster().prepareHealth().execute().actionGet();
fail("authentication failure should have resulted in a NoNodesAvailableException");

176
qa/shield-tribe-node-tests/integration-tests.xml Normal file
View File

@ -0,0 +1,176 @@
<?xml version="1.0"?>
<!--
~ ELASTICSEARCH CONFIDENTIAL
~ __________________
~
~ [2014] Elasticsearch Incorporated. All Rights Reserved.
~
~ NOTICE: All information contained herein is, and remains
~ the property of Elasticsearch Incorporated and its suppliers,
~ if any. The intellectual and technical concepts contained
~ herein are proprietary to Elasticsearch Incorporated
~ and its suppliers and may be covered by U.S. and Foreign Patents,
~ patents in process, and are protected by trade secret or copyright law.
~ Dissemination of this information or reproduction of this material
~ is strictly forbidden unless prior written permission is obtained
~ from Elasticsearch Incorporated.
-->
<project name="smoke-test-tribe-node-with-shield"
xmlns:ac="antlib:net.sf.antcontrib">
<taskdef name="xhttp" classname="org.elasticsearch.ant.HttpTask" classpath="${test_classpath}" />
<typedef name="xhttp" classname="org.elasticsearch.ant.HttpCondition" classpath="${test_classpath}"/>
<import file="${elasticsearch.integ.antfile.default}"/>
<import file="${elasticsearch.tools.directory}/ant/shield-overrides.xml"/>
<property name="tribe_node.pidfile" location="${integ.scratch}/tribe-node.pid"/>
<available property="tribe_node.pidfile.exists" file="${tribe_node.pidfile}"/>
<property name="cluster1.pidfile" location="${integ.scratch}/cluster1.pid"/>
<available property="cluster1.pidfile.exists" file="${cluster1.pidfile}"/>
<property name="cluster2.pidfile" location="${integ.scratch}/cluster2.pid"/>
<available property="cluster2.pidfile.exists" file="${cluster2.pidfile}"/>
<macrodef name="create-index">
<attribute name="name" />
<attribute name="port" />
<sequential>
<xhttp uri="http://127.0.0.1:@{port}/@{name}" method="PUT" username="test_admin" password="changeme" />
<waitfor maxwait="30" maxwaitunit="second"
checkevery="500" checkeveryunit="millisecond"
timeoutproperty="@{timeoutproperty}">
<xhttp uri="http://127.0.0.1:@{port}/_cluster/health/@{name}?wait_for_status=yellow" username="test_admin" password="changeme" />
</waitfor>
</sequential>
</macrodef>
<target name="start-tribe-node-and-2-clusters-with-shield" depends="setup-workspace">
<ac:for list="${xplugins.list}" param="xplugin.name">
<sequential>
<fail message="Expected @{xplugin.name}-${version}.zip as a dependency, but could not be found in ${integ.deps}/plugins}">
<condition>
<not>
<available file="${integ.deps}/plugins/@{xplugin.name}-${elasticsearch.version}.zip"/>
</not>
</condition>
</fail>
</sequential>
</ac:for>
<ac:for param="file">
<path>
<fileset dir="${integ.deps}/plugins"/>
</path>
<sequential>
<local name="plugin.name"/>
<convert-plugin-name file="@{file}" outputproperty="plugin.name"/>
<install-plugin name="${plugin.name}" file="@{file}"/>
</sequential>
</ac:for>
<local name="home"/>
<property name="home" location="${integ.scratch}/elasticsearch-${elasticsearch.version}"/>
<echo>Adding roles.yml</echo>
<copy file="shield-roles.yml" tofile="${home}/config/shield/roles.yml" overwrite="true"/>
<echo>Adding shield users...</echo>
<run-script script="${home}/bin/shield/esusers">
<nested>
<arg value="useradd"/>
<arg value="test_admin"/>
<arg value="-p"/>
<arg value="changeme"/>
<arg value="-r"/>
<arg value="admin"/>
</nested>
</run-script>
<echo>Starting two nodes, each node in a different cluster</echo>
<ac:trycatch property="failure.message">
<ac:try>
<startup-elasticsearch es.transport.tcp.port="9600"
es.http.port="9700"
es.pidfile="${cluster1.pidfile}"
es.unicast.hosts="127.0.0.1:9600"
es.cluster.name="cluster1"/>
</ac:try>
<ac:catch>
<echo>Failed to start cluster1 with message: ${failure.message}</echo>
<stop-node es.pidfile="${cluster1.pidfile}"/>
</ac:catch>
</ac:trycatch>
<ac:trycatch property="failure.message">
<ac:try>
<startup-elasticsearch es.transport.tcp.port="9800"
es.http.port="9900"
es.pidfile="${cluster2.pidfile}"
es.unicast.hosts="127.0.0.1:9800"
es.cluster.name="cluster2"/>
</ac:try>
<ac:catch>
<echo>Failed to start cluster2 with message: ${failure.message}</echo>
<stop-node es.pidfile="${cluster1.pidfile}"/>
<stop-node es.pidfile="${cluster2.pidfile}"/>
</ac:catch>
</ac:trycatch>
<ac:trycatch property="failure.message">
<ac:try>
<echo>Starting a tribe node, configured to connect to cluster1 and cluster2</echo>
<startup-elasticsearch es.pidfile="${tribe_node.pidfile}">
<additional-args>
<arg value="-Des.tribe.cluster1.cluster.name=cluster1"/>
<arg value="-Des.tribe.cluster1.discovery.zen.ping.unicast.hosts=127.0.0.1:9600"/>
<arg value="-Des.tribe.cluster2.cluster.name=cluster2"/>
<arg value="-Des.tribe.cluster2.discovery.zen.ping.unicast.hosts=127.0.0.1:9800"/>
</additional-args>
</startup-elasticsearch>
<xhttp uri="http://127.0.0.1:${integ.http.port}/_cluster/health?wait_for_nodes=5" username="test_admin" password="changeme" />
<!--
From the rest tests we only connect to the tribe node, so we need create the indices externally:
By creating the index after the tribe node has started we can be sure that the tribe node knows
about it. See: https://github.com/elastic/elasticsearch/issues/13292
-->
<echo>Creating index1 in cluster1</echo>
<create-index name="index1" port="9700"/>
<!-- TODO: remove this after we know why on CI the shards of index1 don't get into a started state -->
<loadfile property="cluster1-logs" srcFile="${integ.scratch}/elasticsearch-${elasticsearch.version}/logs/cluster1.log" />
<echo>post index1 creation es logs: ${cluster1-logs}</echo>
<echo>Creating index2 in cluster2</echo>
<create-index name="index2" port="9900"/>
<!-- TODO: remove this after we know why on CI the shards of index2 don't get into a started state -->
<loadfile property="cluster2-logs" srcFile="${integ.scratch}/elasticsearch-${elasticsearch.version}/logs/cluster2.log" />
<echo>post index2 creation es logs: ${cluster2-logs}</echo>
</ac:try>
<ac:catch>
<echo>Failed to start tribe node with message: ${failure.message}</echo>
<stop-node es.pidfile="${tribe_node.pidfile}"/>
<stop-node es.pidfile="${cluster1.pidfile}"/>
<stop-node es.pidfile="${cluster2.pidfile}"/>
</ac:catch>
</ac:trycatch>
</target>
<target name="stop-tribe-node" if="tribe_node.pidfile.exists">
<stop-node es.pidfile="${tribe_node.pidfile}"/>
</target>
<target name="stop-cluster1" if="cluster1.pidfile.exists">
<stop-node es.pidfile="${cluster1.pidfile}"/>
</target>
<target name="stop-cluster2" if="cluster2.pidfile.exists">
<stop-node es.pidfile="${cluster2.pidfile}"/>
</target>
<target name="stop-tribe-node-and-all-clusters" depends="stop-tribe-node,stop-cluster1,stop-cluster2"/>
</project>

202
qa/shield-tribe-node-tests/pom.xml Normal file
View File

@ -0,0 +1,202 @@
<?xml version="1.0" encoding="UTF-8"?>
<!--
~ ELASTICSEARCH CONFIDENTIAL
~ __________________
~
~ [2014] Elasticsearch Incorporated. All Rights Reserved.
~
~ NOTICE: All information contained herein is, and remains
~ the property of Elasticsearch Incorporated and its suppliers,
~ if any. The intellectual and technical concepts contained
~ herein are proprietary to Elasticsearch Incorporated
~ and its suppliers and may be covered by U.S. and Foreign Patents,
~ patents in process, and are protected by trade secret or copyright law.
~ Dissemination of this information or reproduction of this material
~ is strictly forbidden unless prior written permission is obtained
~ from Elasticsearch Incorporated.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.elasticsearch.qa</groupId>
<artifactId>x-plugins-qa</artifactId>
<version>3.0.0-SNAPSHOT</version>
</parent>
<artifactId>smoke-test-tribe-node-with-shield</artifactId>
<name>QA: Smoke Test tribe node with Shield</name>
<description>Start a tribe node and two nodes both with a different cluster name and verifies if all data accessable via the tribe node</description>
<properties>
<skip.unit.tests>true</skip.unit.tests>
<elasticsearch.integ.antfile>${project.basedir}/integration-tests.xml</elasticsearch.integ.antfile>
<tests.rest.load_packaged>false</tests.rest.load_packaged>
<xplugins.list>license,shield</xplugins.list>
</properties>
<dependencies>
<dependency>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>shield</artifactId>
<version>${elasticsearch.version}</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>license</artifactId>
<version>${project.version}</version>
<scope>test</scope>
</dependency>
</dependencies>
<build>
<testResources>
<!-- REST API specifications copied from main Elasticsearch specs
because they are required to execute the Watcher REST tests -->
<testResource>
<directory>${elasticsearch.tools.directory}/rest-api-spec</directory>
<targetPath>rest-api-spec</targetPath>
<includes>
<!-- required by the test framework -->
<include>api/info.json</include>
<include>api/cluster.health.json</include>
<include>api/cluster.state.json</include>
<!-- used by Watcher REST tests -->
<include>api/index.json</include>
<include>api/get.json</include>
<include>api/delete.json</include>
<include>api/delete-by-query.json</include>
<include>api/bulk.json</include>
<include>api/update.json</include>
<include>api/search.json</include>
<include>api/indices.delete.json</include>
<include>api/indices.refresh.json</include>
</includes>
</testResource>
<testResource>
<directory>${basedir}/rest-api-spec</directory>
<filtering>true</filtering>
<targetPath>rest-api-spec</targetPath>
<includes>
<include>api/*.json</include>
<include>test/**/*.yaml</include>
</includes>
</testResource>
</testResources>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-dependency-plugin</artifactId>
<executions>
<execution>
<id>integ-setup-dependencies</id>
<phase>pre-integration-test</phase>
<goals>
<goal>copy</goal>
</goals>
<configuration>
<skip>${skip.integ.tests}</skip>
<useBaseVersion>true</useBaseVersion>
<outputDirectory>${integ.deps}/plugins</outputDirectory>
<artifactItems>
<!-- elasticsearch distribution -->
<artifactItem>
<groupId>org.elasticsearch.distribution.zip</groupId>
<artifactId>elasticsearch</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
<outputDirectory>${integ.deps}</outputDirectory>
</artifactItem>
<!-- commercial plugins -->
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>license</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>shield</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
</artifactItems>
</configuration>
</execution>
</executions>
</plugin>
<!-- integration tests -->
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-antrun-plugin</artifactId>
<executions>
<!-- start up external cluster -->
<execution>
<id>integ-setup</id>
<phase>pre-integration-test</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<target>
<property name="test_classpath" refid="maven.test.classpath"/>
<ant antfile="${elasticsearch.integ.antfile}" target="start-tribe-node-and-2-clusters-with-shield">
<property name="tests.jvm.argline" value="${tests.jvm.argline}"/>
<property name="plugins.dir" value="${plugins.dir}"/>
<property name="xplugins.list" value="${xplugins.list}"/>
</ant>
</target>
<skip>${skip.integ.tests}</skip>
</configuration>
</execution>
<!-- shut down external cluster -->
<execution>
<id>integ-teardown</id>
<phase>post-integration-test</phase>
<goals>
<goal>run</goal>
</goals>
<configuration>
<target>
<property name="test_classpath" refid="maven.test.classpath"/>
<ant antfile="${elasticsearch.integ.antfile}" target="stop-tribe-node-and-all-clusters"/>
</target>
<skip>${skip.integ.tests}</skip>
</configuration>
</execution>
</executions>
<dependencies>
<dependency>
<groupId>ant-contrib</groupId>
<artifactId>ant-contrib</artifactId>
<version>1.0b3</version>
<exclusions>
<exclusion>
<groupId>ant</groupId>
<artifactId>ant</artifactId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.apache.ant</groupId>
<artifactId>ant-nodeps</artifactId>
<version>1.8.1</version>
</dependency>
</dependencies>
</plugin>
</plugins>
</build>
</project>

27
qa/shield-tribe-node-tests/rest-api-spec/test/tribe_node/10_basic.yaml Normal file
View File

@ -0,0 +1,27 @@
---
"Tribe node search":
- do:
index:
index: index1
type: test
id: 1
body: { foo: bar }
- do:
index:
index: index2
type: test
id: 1
body: { foo: bar }
- do:
indices.refresh: {}
- do:
search:
index: index1,index2
body:
query: { term: { foo: bar }}
- match: { hits.total: 2 }

4
qa/shield-tribe-node-tests/shield-roles.yml Normal file
View File

@ -0,0 +1,4 @@
admin:
cluster: all
indices:
'*': all

25
qa/shield-tribe-node-tests/src/test/java/org/elasticsearch/ant/HttpCondition.java Normal file
View File

@ -0,0 +1,25 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.ant;
import org.apache.tools.ant.BuildException;
import org.apache.tools.ant.taskdefs.condition.Condition;
public class HttpCondition extends HttpTask implements Condition {
private int expectedResponseCode = 200;
@Override
public boolean eval() throws BuildException {
int responseCode = executeHttpRequest();
getProject().log("response code=" + responseCode);
return responseCode == expectedResponseCode;
}
public void setExpectedResponseCode(int expectedResponseCode) {
this.expectedResponseCode = expectedResponseCode;
}
}

82
qa/shield-tribe-node-tests/src/test/java/org/elasticsearch/ant/HttpTask.java Normal file
View File

@ -0,0 +1,82 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.ant;
import org.apache.tools.ant.BuildException;
import org.apache.tools.ant.Task;
import org.elasticsearch.common.Base64;
import java.net.HttpURLConnection;
import java.net.URI;
import java.net.URL;
import java.nio.charset.StandardCharsets;
public class HttpTask extends Task {
private String uri;
private String method;
private String body;
private String username;
private String password;
@Override
public void execute() throws BuildException {
int responseCode = executeHttpRequest();
getProject().log("response code=" + responseCode);
}
protected int executeHttpRequest() {
try {
URI uri = new URI(this.uri);
URL url = uri.toURL();
getProject().log("url=" + url);
HttpURLConnection urlConnection = (HttpURLConnection) url.openConnection();
if (method != null) {
urlConnection.setRequestMethod(method);
}
if (username != null) {
String basicAuth = "Basic " + Base64.encodeBytes((username + ":" + password).getBytes(StandardCharsets.UTF_8));
urlConnection.setRequestProperty("Authorization", basicAuth);
}
if (body != null) {
urlConnection.setDoOutput(true);
urlConnection.setRequestProperty("Accept-Charset", StandardCharsets.UTF_8.name());
byte[] bytes = body.getBytes(StandardCharsets.UTF_8.name());
urlConnection.setRequestProperty("Content-Length", String.valueOf(bytes.length));
urlConnection.getOutputStream().write(bytes);
urlConnection.getOutputStream().close();
}
urlConnection.connect();
int responseCode = urlConnection.getResponseCode();
urlConnection.disconnect();
return responseCode;
} catch (Exception e) {
throw new BuildException(e);
}
}
public void setUri(String uri) {
this.uri = uri;
}
public void setMethod(String method) {
this.method = method;
}
public void setBody(String body) {
this.body = body;
}
public void setUsername(String username) {
this.username = username;
}
public void setPassword(String password) {
this.password = password;
}
}

44
qa/shield-tribe-node-tests/src/test/java/org/elasticsearch/shield/RestIT.java Normal file
View File

@ -0,0 +1,44 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.shield;
import com.carrotsearch.randomizedtesting.annotations.Name;
import com.carrotsearch.randomizedtesting.annotations.ParametersFactory;
import org.elasticsearch.client.support.Headers;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.shield.authc.support.SecuredString;
import org.elasticsearch.test.rest.ESRestTestCase;
import org.elasticsearch.test.rest.RestTestCandidate;
import org.elasticsearch.test.rest.parser.RestTestParseException;
import java.io.IOException;
import static org.elasticsearch.shield.authc.support.UsernamePasswordToken.basicAuthHeaderValue;
public class RestIT extends TribeRestTestCase {
private static final String USER = "test_admin";
private static final String PASS = "changeme";
public RestIT(@Name("yaml") RestTestCandidate testCandidate) {
super(testCandidate);
}
@ParametersFactory
public static Iterable<Object[]> parameters() throws IOException, RestTestParseException {
return ESRestTestCase.createParameters(0, 1);
}
@Override
protected Settings restClientSettings() {
String token = basicAuthHeaderValue(USER, new SecuredString(PASS.toCharArray()));
return Settings.builder()
.put(Headers.PREFIX + ".Authorization", token)
.build();
}
}

373
qa/shield-tribe-node-tests/src/test/java/org/elasticsearch/shield/TribeRestTestCase.java Normal file
View File

@ -0,0 +1,373 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License;
* you may not use this file except in compliance with the Elastic License.
*/
package org.elasticsearch.shield;
import com.carrotsearch.randomizedtesting.RandomizedTest;
import com.carrotsearch.randomizedtesting.annotations.TestGroup;
import com.carrotsearch.randomizedtesting.annotations.TimeoutSuite;
import org.apache.lucene.util.IOUtils;
import org.apache.lucene.util.LuceneTestCase.SuppressCodecs;
import org.apache.lucene.util.LuceneTestCase.SuppressFsync;
import org.apache.lucene.util.TimeUnits;
import org.elasticsearch.common.Strings;
import org.elasticsearch.common.SuppressForbidden;
import org.elasticsearch.common.io.PathUtils;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.InetSocketTransportAddress;
import org.elasticsearch.common.transport.TransportAddress;
import org.elasticsearch.common.xcontent.XContentHelper;
import org.elasticsearch.node.Node;
import org.elasticsearch.repositories.uri.URLRepository;
import org.elasticsearch.test.ESIntegTestCase;
import org.elasticsearch.test.ESIntegTestCase.ClusterScope;
import org.elasticsearch.test.ESTestCase;
import org.elasticsearch.test.rest.ESRestTestCase;
import org.elasticsearch.test.rest.RestTestCandidate;
import org.elasticsearch.test.rest.RestTestExecutionContext;
import org.elasticsearch.test.rest.client.RestException;
import org.elasticsearch.test.rest.parser.RestTestParseException;
import org.elasticsearch.test.rest.parser.RestTestSuiteParser;
import org.elasticsearch.test.rest.section.DoSection;
import org.elasticsearch.test.rest.section.ExecutableSection;
import org.elasticsearch.test.rest.section.RestTestSuite;
import org.elasticsearch.test.rest.section.SkipSection;
import org.elasticsearch.test.rest.section.TestSection;
import org.elasticsearch.test.rest.spec.RestApi;
import org.elasticsearch.test.rest.spec.RestSpec;
import org.elasticsearch.test.rest.support.FileUtils;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
import java.io.IOException;
import java.io.InputStream;
import java.lang.annotation.ElementType;
import java.lang.annotation.Inherited;
import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;
import java.net.*;
import java.nio.file.FileSystem;
import java.nio.file.FileSystems;
import java.nio.file.Files;
import java.nio.file.Path;
import java.nio.file.PathMatcher;
import java.nio.file.StandardCopyOption;
import java.util.ArrayList;
import java.util.Collections;
import java.util.Comparator;
import java.util.List;
import java.util.Map;
import java.util.Set;
/**
* Forked from RestTestCase with changes required to run rest tests via a tribe node
*
* Reasons for forking:
* 1) Always communicate via the tribe node from the tests. The original class in core connects to any endpoint it can see via the nodes info api and that would mean also the nodes part of the other clusters would be just as entry point. This should not happen for the tribe tests
* 2) The original class in core executes delete calls after each test, but the tribe node can't handle master level write operations. These api calls hang for 1m and then just fail.
* 3) The indices in cluster1 and cluster2 are created from the ant integ file and because of that the original class in core would just remove that in between tests.
* 4) extends ESTestCase instead if ESIntegTestCase and doesn't setup a test cluster and just connects to the one endpoint defined in the tests.rest.cluster.
*/
@ESRestTestCase.Rest
@SuppressFsync // we aren't trying to test this here, and it can make the test slow
@SuppressCodecs("*") // requires custom completion postings format
@ClusterScope(randomDynamicTemplates = false)
@TimeoutSuite(millis = 40 * TimeUnits.MINUTE) // timeout the suite after 40min and fail the test.
public abstract class TribeRestTestCase extends ESTestCase {
/**
* Property that allows to control whether the REST tests are run (default) or not
*/
public static final String TESTS_REST = "tests.rest";
public static final String TESTS_REST_CLUSTER = "tests.rest.cluster";
/**
* Annotation for REST tests
*/
@Inherited
@Retention(RetentionPolicy.RUNTIME)
@Target(ElementType.TYPE)
@TestGroup(enabled = true, sysProperty = ESRestTestCase.TESTS_REST)
public @interface Rest {
}
/**
* Property that allows to control which REST tests get run. Supports comma separated list of tests
* or directories that contain tests e.g. -Dtests.rest.suite=index,get,create/10_with_id
*/
public static final String REST_TESTS_SUITE = "tests.rest.suite";
/**
* Property that allows to blacklist some of the REST tests based on a comma separated list of globs
* e.g. -Dtests.rest.blacklist=get/10_basic/*
*/
public static final String REST_TESTS_BLACKLIST = "tests.rest.blacklist";
/**
* Property that allows to control whether spec validation is enabled or not (default true).
*/
public static final String REST_TESTS_VALIDATE_SPEC = "tests.rest.validate_spec";
/**
* Property that allows to control where the REST spec files need to be loaded from
*/
public static final String REST_TESTS_SPEC = "tests.rest.spec";
public static final String REST_LOAD_PACKAGED_TESTS = "tests.rest.load_packaged";
private static final String DEFAULT_TESTS_PATH = "/rest-api-spec/test";
private static final String DEFAULT_SPEC_PATH = "/rest-api-spec/api";
private static final String PATHS_SEPARATOR = ",";
private final PathMatcher[] blacklistPathMatchers;
private static RestTestExecutionContext restTestExecutionContext;
private final RestTestCandidate testCandidate;
public TribeRestTestCase(RestTestCandidate testCandidate) {
this.testCandidate = testCandidate;
String[] blacklist = resolvePathsProperty(REST_TESTS_BLACKLIST, null);
if (blacklist != null) {
blacklistPathMatchers = new PathMatcher[blacklist.length];
int i = 0;
for (String glob : blacklist) {
blacklistPathMatchers[i++] = PathUtils.getDefaultFileSystem().getPathMatcher("glob:" + glob);
}
} else {
blacklistPathMatchers = new PathMatcher[0];
}
}
@Override
protected void afterIfFailed(List<Throwable> errors) {
logger.info("Stash dump on failure [{}]", XContentHelper.toString(restTestExecutionContext.stash()));
super.afterIfFailed(errors);
}
public static Iterable<Object[]> createParameters(int id, int count) throws IOException, RestTestParseException {
TestGroup testGroup = Rest.class.getAnnotation(TestGroup.class);
String sysProperty = TestGroup.Utilities.getSysProperty(Rest.class);
boolean enabled;
try {
enabled = RandomizedTest.systemPropertyAsBoolean(sysProperty, testGroup.enabled());
} catch (IllegalArgumentException e) {
// Ignore malformed system property, disable the group if malformed though.
enabled = false;
}
if (!enabled) {
return new ArrayList<>();
}
//parse tests only if rest test group is enabled, otherwise rest tests might not even be available on file system
List<RestTestCandidate> restTestCandidates = collectTestCandidates(id, count);
List<Object[]> objects = new ArrayList<>();
for (RestTestCandidate restTestCandidate : restTestCandidates) {
objects.add(new Object[]{restTestCandidate});
}
return objects;
}
private static List<RestTestCandidate> collectTestCandidates(int id, int count) throws RestTestParseException, IOException {
List<RestTestCandidate> testCandidates = new ArrayList<>();
FileSystem fileSystem = getFileSystem();
// don't make a try-with, getFileSystem returns null
// ... and you can't close() the default filesystem
try {
String[] paths = resolvePathsProperty(REST_TESTS_SUITE, DEFAULT_TESTS_PATH);
Map<String, Set<Path>> yamlSuites = FileUtils.findYamlSuites(fileSystem, DEFAULT_TESTS_PATH, paths);
RestTestSuiteParser restTestSuiteParser = new RestTestSuiteParser();
//yaml suites are grouped by directory (effectively by api)
for (String api : yamlSuites.keySet()) {
List<Path> yamlFiles = new ArrayList<>(yamlSuites.get(api));
for (Path yamlFile : yamlFiles) {
String key = api + yamlFile.getFileName().toString();
if (mustExecute(key, id, count)) {
RestTestSuite restTestSuite = restTestSuiteParser.parse(api, yamlFile);
for (TestSection testSection : restTestSuite.getTestSections()) {
testCandidates.add(new RestTestCandidate(restTestSuite, testSection));
}
}
}
}
} finally {
IOUtils.close(fileSystem);
}
//sort the candidates so they will always be in the same order before being shuffled, for repeatability
Collections.sort(testCandidates, new Comparator<RestTestCandidate>() {
@Override
public int compare(RestTestCandidate o1, RestTestCandidate o2) {
return o1.getTestPath().compareTo(o2.getTestPath());
}
});
return testCandidates;
}
private static boolean mustExecute(String test, int id, int count) {
int hash = (int) (Math.abs((long)test.hashCode()) % count);
return hash == id;
}
private static String[] resolvePathsProperty(String propertyName, String defaultValue) {
String property = System.getProperty(propertyName);
if (!Strings.hasLength(property)) {
return defaultValue == null ? null : new String[]{defaultValue};
} else {
return property.split(PATHS_SEPARATOR);
}
}
/**
* Returns a new FileSystem to read REST resources, or null if they
* are available from classpath.
*/
@SuppressForbidden(reason = "proper use of URL, hack around a JDK bug")
static FileSystem getFileSystem() throws IOException {
// REST suite handling is currently complicated, with lots of filtering and so on
// For now, to work embedded in a jar, return a ZipFileSystem over the jar contents.
URL codeLocation = FileUtils.class.getProtectionDomain().getCodeSource().getLocation();
boolean loadPackaged = RandomizedTest.systemPropertyAsBoolean(REST_LOAD_PACKAGED_TESTS, true);
if (codeLocation.getFile().endsWith(".jar") && loadPackaged) {
try {
// hack around a bug in the zipfilesystem implementation before java 9,
// its checkWritable was incorrect and it won't work without write permissions.
// if we add the permission, it will open jars r/w, which is too scary! so copy to a safe r-w location.
Path tmp = Files.createTempFile(null, ".jar");
try (InputStream in = codeLocation.openStream()) {
Files.copy(in, tmp, StandardCopyOption.REPLACE_EXISTING);
}
return FileSystems.newFileSystem(new URI("jar:" + tmp.toUri()), Collections.<String,Object>emptyMap());
} catch (URISyntaxException e) {
throw new IOException("couldn't open zipfilesystem: ", e);
}
} else {
return null;
}
}
@BeforeClass
public static void initExecutionContext() throws IOException, RestException {
String[] specPaths = resolvePathsProperty(REST_TESTS_SPEC, DEFAULT_SPEC_PATH);
RestSpec restSpec = null;
FileSystem fileSystem = getFileSystem();
// don't make a try-with, getFileSystem returns null
// ... and you can't close() the default filesystem
try {
restSpec = RestSpec.parseFrom(fileSystem, DEFAULT_SPEC_PATH, specPaths);
} finally {
IOUtils.close(fileSystem);
}
validateSpec(restSpec);
restTestExecutionContext = new RestTestExecutionContext(restSpec);
}
private static void validateSpec(RestSpec restSpec) {
boolean validateSpec = RandomizedTest.systemPropertyAsBoolean(REST_TESTS_VALIDATE_SPEC, true);
if (validateSpec) {
StringBuilder errorMessage = new StringBuilder();
for (RestApi restApi : restSpec.getApis()) {
if (restApi.getMethods().contains("GET") && restApi.isBodySupported()) {
if (!restApi.getMethods().contains("POST")) {
errorMessage.append("\n- ").append(restApi.getName()).append(" supports GET with a body but doesn't support POST");
}
}
}
if (errorMessage.length() > 0) {
throw new IllegalArgumentException(errorMessage.toString());
}
}
}
@AfterClass
public static void close() {
if (restTestExecutionContext != null) {
restTestExecutionContext.close();
restTestExecutionContext = null;
}
}
/**
* Used to obtain settings for the REST client that is used to send REST requests.
*/
protected Settings restClientSettings() {
return Settings.EMPTY;
}
protected InetSocketAddress[] httpAddresses() {
String clusterAddresses = System.getProperty(TESTS_REST_CLUSTER);
String[] stringAddresses = clusterAddresses.split(",");
InetSocketAddress[] transportAddresses = new InetSocketAddress[stringAddresses.length];
int i = 0;
for (String stringAddress : stringAddresses) {
String[] split = stringAddress.split(":");
if (split.length < 2) {
throw new IllegalArgumentException("address [" + clusterAddresses + "] not valid");
}
try {
transportAddresses[i++] = new InetSocketAddress(InetAddress.getByName(split[0]), Integer.valueOf(split[1]));
} catch (NumberFormatException e) {
throw new IllegalArgumentException("port is not valid, expected number but was [" + split[1] + "]");
} catch (UnknownHostException e) {
throw new IllegalArgumentException("unknown host [" + split[0] + "]", e);
}
}
return transportAddresses;
}
@Before
public void reset() throws IOException, RestException {
//skip test if it matches one of the blacklist globs
for (PathMatcher blacklistedPathMatcher : blacklistPathMatchers) {
//we need to replace a few characters otherwise the test section name can't be parsed as a path on windows
String testSection = testCandidate.getTestSection().getName().replace("*", "").replace("\\", "/").replaceAll("\\s+/", "/").replace(":", "").trim();
String testPath = testCandidate.getSuitePath() + "/" + testSection;
assumeFalse("[" + testCandidate.getTestPath() + "] skipped, reason: blacklisted", blacklistedPathMatcher.matches(PathUtils.get(testPath)));
}
//The client needs non static info to get initialized, therefore it can't be initialized in the before class
restTestExecutionContext.initClient(httpAddresses(), restClientSettings());
restTestExecutionContext.clear();
//skip test if the whole suite (yaml file) is disabled
assumeFalse(buildSkipMessage(testCandidate.getSuitePath(), testCandidate.getSetupSection().getSkipSection()),
testCandidate.getSetupSection().getSkipSection().skip(restTestExecutionContext.esVersion()));
//skip test if test section is disabled
assumeFalse(buildSkipMessage(testCandidate.getTestPath(), testCandidate.getTestSection().getSkipSection()),
testCandidate.getTestSection().getSkipSection().skip(restTestExecutionContext.esVersion()));
}
private static String buildSkipMessage(String description, SkipSection skipSection) {
StringBuilder messageBuilder = new StringBuilder();
if (skipSection.isVersionCheck()) {
messageBuilder.append("[").append(description).append("] skipped, reason: [").append(skipSection.getReason()).append("] ");
} else {
messageBuilder.append("[").append(description).append("] skipped, reason: features ").append(skipSection.getFeatures()).append(" not supported");
}
return messageBuilder.toString();
}
@Test
public void test() throws IOException {
//let's check that there is something to run, otherwise there might be a problem with the test section
if (testCandidate.getTestSection().getExecutableSections().size() == 0) {
throw new IllegalArgumentException("No executable sections loaded for [" + testCandidate.getTestPath() + "]");
}
if (!testCandidate.getSetupSection().isEmpty()) {
logger.info("start setup test [{}]", testCandidate.getTestPath());
for (DoSection doSection : testCandidate.getSetupSection().getDoSections()) {
doSection.execute(restTestExecutionContext);
}
logger.info("end setup test [{}]", testCandidate.getTestPath());
}
restTestExecutionContext.clear();
for (ExecutableSection executableSection : testCandidate.getTestSection().getExecutableSections()) {
executableSection.execute(restTestExecutionContext);
}
}
}

87
qa/smoke-test-plugins-ssl/pom.xml
View File

@ -8,7 +8,7 @@
<parent>
<groupId>org.elasticsearch.qa</groupId>
<artifactId>x-plugins-qa</artifactId>
<version>2.1.0-SNAPSHOT</version>
<version>3.0.0-SNAPSHOT</version>
</parent>
<!--
@ -101,6 +101,14 @@
</artifactItem>
<!-- open-source plugins -->
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>analysis-icu</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>analysis-kuromoji</artifactId>
@ -109,6 +117,14 @@
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>analysis-phonetic</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>analysis-smartcn</artifactId>
@ -127,15 +143,7 @@
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>analysis-phonetic</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>analysis-icu</artifactId>
<artifactId>cloud-azure</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
@ -149,22 +157,6 @@
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>cloud-azure</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>cloud-aws</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>delete-by-query</artifactId>
@ -175,7 +167,15 @@
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>lang-python</artifactId>
<artifactId>discovery-ec2</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>discovery-multicast</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
@ -189,6 +189,22 @@
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>lang-python</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>mapper-murmur3</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>mapper-size</artifactId>
@ -197,6 +213,23 @@
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>repository-s3</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<!-- Internal plugins for test purpose only -->
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>jvm-example</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>site-example</artifactId>

8
qa/smoke-test-plugins-ssl/rest-api-spec/test/smoke_test_plugins_ssl/10_basic.yaml
View File

@ -10,7 +10,7 @@
- do:
nodes.info: {}
- match: { nodes.$master.plugins.11.name: license }
- match: { nodes.$master.plugins.13.name: marvel }
- match: { nodes.$master.plugins.14.name: shield }
- match: { nodes.$master.plugins.16.name: watcher }
- match: { nodes.$master.plugins.13.name: license }
- match: { nodes.$master.plugins.16.name: marvel }
- match: { nodes.$master.plugins.18.name: shield }
- match: { nodes.$master.plugins.20.name: watcher }

6
qa/smoke-test-plugins-ssl/src/test/java/org/elasticsearch/smoketest/SmokeTestPluginsSslIT.java
View File

@ -80,8 +80,12 @@ public class SmokeTestPluginsSslIT extends ESRestTestCase {
.put("shield.transport.ssl", true)
.put("shield.ssl.keystore.path", keyStore)
.put("shield.ssl.keystore.password", KEYSTORE_PASS)
.put("plugin.types", ShieldPlugin.class.getName())
.build();
}
@Override
protected Collection<Class<? extends Plugin>> transportClientPlugins() {
return Collections.<Class<? extends Plugin>>singleton(ShieldPlugin.class);
}
}

87
qa/smoke-test-plugins/pom.xml
View File

@ -8,7 +8,7 @@
<parent>
<groupId>org.elasticsearch.qa</groupId>
<artifactId>x-plugins-qa</artifactId>
<version>2.1.0-SNAPSHOT</version>
<version>3.0.0-SNAPSHOT</version>
</parent>
<!--
@ -99,6 +99,14 @@
</artifactItem>
<!-- open-source plugins -->
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>analysis-icu</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>analysis-kuromoji</artifactId>
@ -107,6 +115,14 @@
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>analysis-phonetic</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>analysis-smartcn</artifactId>
@ -125,15 +141,7 @@
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>analysis-phonetic</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>analysis-icu</artifactId>
<artifactId>cloud-azure</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
@ -147,22 +155,6 @@
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>cloud-azure</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>cloud-aws</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>delete-by-query</artifactId>
@ -173,7 +165,15 @@
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>lang-python</artifactId>
<artifactId>discovery-ec2</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>discovery-multicast</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
@ -187,6 +187,22 @@
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>lang-python</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>mapper-murmur3</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>mapper-size</artifactId>
@ -195,6 +211,23 @@
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>repository-s3</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<!-- Internal plugins for test purpose only -->
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>jvm-example</artifactId>
<version>${elasticsearch.version}</version>
<type>zip</type>
<overWrite>true</overWrite>
</artifactItem>
<artifactItem>
<groupId>org.elasticsearch.plugin</groupId>
<artifactId>site-example</artifactId>

8
qa/smoke-test-plugins/rest-api-spec/test/smoke_test_plugins/10_basic.yaml
View File

@ -10,7 +10,7 @@
- do:
nodes.info: {}
- match: { nodes.$master.plugins.11.name: license }
- match: { nodes.$master.plugins.13.name: marvel }
- match: { nodes.$master.plugins.14.name: shield }
- match: { nodes.$master.plugins.16.name: watcher }
- match: { nodes.$master.plugins.13.name: license }
- match: { nodes.$master.plugins.16.name: marvel }
- match: { nodes.$master.plugins.18.name: shield }
- match: { nodes.$master.plugins.20.name: watcher }

8
qa/smoke-test-plugins/src/test/java/org/elasticsearch/smoketest/SmokeTestPluginsIT.java
View File

@ -10,6 +10,7 @@ import com.carrotsearch.randomizedtesting.annotations.ParametersFactory;
import org.elasticsearch.client.support.Headers;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.plugins.Plugin;
import org.elasticsearch.shield.ShieldPlugin;
import org.elasticsearch.shield.authc.support.SecuredString;
import org.elasticsearch.test.rest.ESRestTestCase;
@ -17,6 +18,8 @@ import org.elasticsearch.test.rest.RestTestCandidate;
import org.elasticsearch.test.rest.parser.RestTestParseException;
import java.io.IOException;
import java.util.Collection;
import java.util.Collections;
import static org.elasticsearch.shield.authc.support.UsernamePasswordToken.basicAuthHeaderValue;
@ -46,9 +49,12 @@ public class SmokeTestPluginsIT extends ESRestTestCase {
protected Settings externalClusterClientSettings() {
return Settings.builder()
.put("shield.user", USER + ":" + PASS)
.put("plugin.types", ShieldPlugin.class.getName())
.build();
}
@Override
protected Collection<Class<? extends Plugin>> transportClientPlugins() {
return Collections.<Class<? extends Plugin>>singleton(ShieldPlugin.class);
}
}

4
qa/smoke-test-watcher-with-shield/pom.xml
View File

@ -25,7 +25,7 @@
<parent>
<groupId>org.elasticsearch.qa</groupId>
<artifactId>x-plugins-qa</artifactId>
<version>2.1.0-SNAPSHOT</version>
<version>3.0.0-SNAPSHOT</version>
</parent>
<artifactId>smoke-test-watcher-with-shield</artifactId>
@ -37,7 +37,7 @@
<elasticsearch.integ.antfile>${project.basedir}/integration-tests.xml</elasticsearch.integ.antfile>
<tests.rest.load_packaged>false</tests.rest.load_packaged>
<xplugins.list>license,shield,watcher</xplugins.list>
<tests.rest.blacklist>hijack/10_basic/*</tests.rest.blacklist>
<tests.rest.blacklist>hijack/10_basic/*,array_compare_watch/10_basic/Basic array_compare watch</tests.rest.blacklist>
</properties>
<dependencies>

11
qa/smoke-test-watcher-with-shield/src/test/java/org/elasticsearch/smoketest/WatcherWithShieldIT.java
View File

@ -14,20 +14,21 @@ import org.apache.http.impl.conn.BasicHttpClientConnectionManager;
import org.elasticsearch.client.support.Headers;
import org.elasticsearch.common.network.NetworkAddress;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.license.plugin.LicensePlugin;
import org.elasticsearch.plugins.Plugin;
import org.elasticsearch.shield.ShieldPlugin;
import org.elasticsearch.shield.authc.support.SecuredString;
import org.elasticsearch.shield.authc.support.UsernamePasswordToken;
import org.elasticsearch.test.rest.ESRestTestCase;
import org.elasticsearch.test.rest.RestTestCandidate;
import org.elasticsearch.test.rest.parser.RestTestParseException;
import org.elasticsearch.watcher.WatcherPlugin;
import org.junit.After;
import org.junit.Before;
import java.io.IOException;
import java.net.InetSocketAddress;
import java.net.URI;
import java.util.Collection;
import java.util.Collections;
import static org.elasticsearch.shield.authc.support.UsernamePasswordToken.basicAuthHeaderValue;
@ -80,7 +81,6 @@ public class WatcherWithShieldIT extends ESRestTestCase {
protected Settings externalClusterClientSettings() {
return Settings.builder()
.put("shield.user", TEST_ADMIN_USERNAME + ":" + TEST_ADMIN_PASSWORD)
.put("plugin.types", WatcherPlugin.class.getName() + "," + ShieldPlugin.class.getName() + "," + LicensePlugin.class.getName())
.build();
}
@ -88,5 +88,10 @@ public class WatcherWithShieldIT extends ESRestTestCase {
return new String[]{"watcher_manager", "changeme"};
}
@Override
protected Collection<Class<? extends Plugin>> transportClientPlugins() {
return Collections.<Class<? extends Plugin>>singleton(ShieldPlugin.class);
}
}

6
shield/.settings/org.eclipse.jdt.core.prefs
View File

@ -3,8 +3,8 @@ org.eclipse.jdt.core.compiler.annotation.inheritNullAnnotations=enabled
org.eclipse.jdt.core.compiler.annotation.missingNonNullByDefaultAnnotation=ignore
org.eclipse.jdt.core.compiler.annotation.nullable=org.elasticsearch.common.Nullable
org.eclipse.jdt.core.compiler.annotation.nullanalysis=enabled
org.eclipse.jdt.core.compiler.codegen.targetPlatform=1.7
org.eclipse.jdt.core.compiler.compliance=1.7
org.eclipse.jdt.core.compiler.codegen.targetPlatform=1.8
org.eclipse.jdt.core.compiler.compliance=1.8
org.eclipse.jdt.core.compiler.problem.forbiddenReference=warning
org.eclipse.jdt.core.compiler.problem.nonnullParameterAnnotationDropped=warning
org.eclipse.jdt.core.compiler.problem.nullAnnotationInferenceConflict=warning
@ -12,7 +12,7 @@ org.eclipse.jdt.core.compiler.problem.nullReference=warning
org.eclipse.jdt.core.compiler.problem.nullSpecViolation=warning
org.eclipse.jdt.core.compiler.problem.nullUncheckedConversion=warning
org.eclipse.jdt.core.compiler.problem.potentialNullReference=warning
org.eclipse.jdt.core.compiler.source=1.7
org.eclipse.jdt.core.compiler.source=1.8
org.eclipse.jdt.core.formatter.lineSplit=140
org.eclipse.jdt.core.formatter.tabulation.char=space
org.eclipse.jdt.core.formatter.tabulation.size=4

4
shield/bin/shield/esusers
View File

@ -32,7 +32,7 @@ if [ "x$ES_INCLUDE" = "x" ]; then
/opt/elasticsearch/elasticsearch.in.sh \
~/.elasticsearch.in.sh \
"`dirname "$0"`"/../elasticsearch.in.sh \
$ES_HOME/bin/elasticsearch.in.sh; do
"$ES_HOME/bin/elasticsearch.in.sh"; do
if [ -r "$include" ]; then
. "$include"
break
@ -125,7 +125,7 @@ export HOSTNAME=`hostname -s`
# include shield jars in classpath
ES_CLASSPATH="$ES_CLASSPATH:$ES_HOME/plugins/shield/*"
cd $ES_HOME > /dev/null
cd "$ES_HOME" > /dev/null
"$JAVA" $ES_JAVA_OPTS -cp "$ES_CLASSPATH" -Des.path.home="$ES_HOME" $properties org.elasticsearch.shield.authc.esusers.tool.ESUsersTool "$@"
status=$?
cd - > /dev/null

4
shield/bin/shield/esusers.bat
View File

@ -4,6 +4,6 @@ rem Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
rem or more contributor license agreements. Licensed under the Elastic License;
rem you may not use this file except in compliance with the Elastic License.
PUSHD %~dp0
CALL %~dp0.in.bat org.elasticsearch.shield.authc.esusers.tool.ESUsersTool %*
PUSHD "%~dp0"
CALL "%~dp0.in.bat" org.elasticsearch.shield.authc.esusers.tool.ESUsersTool %*
POPD

4
shield/bin/shield/syskeygen
View File

@ -32,7 +32,7 @@ if [ "x$ES_INCLUDE" = "x" ]; then
/opt/elasticsearch/elasticsearch.in.sh \
~/.elasticsearch.in.sh \
"`dirname "$0"`"/../elasticsearch.in.sh \
$ES_HOME/bin/elasticsearch.in.sh; do
"$ES_HOME/bin/elasticsearch.in.sh"; do
if [ -r "$include" ]; then
. "$include"
break
@ -125,7 +125,7 @@ export HOSTNAME=`hostname -s`
# include shield jars in classpath
ES_CLASSPATH="$ES_CLASSPATH:$ES_HOME/plugins/shield/*"
cd $ES_HOME > /dev/null
cd "$ES_HOME" > /dev/null
$JAVA $ES_JAVA_OPTS -cp "$ES_CLASSPATH" -Des.path.home="$ES_HOME" $properties org.elasticsearch.shield.crypto.tool.SystemKeyTool "$@"
status=$?
cd - > /dev/null

4
shield/bin/shield/syskeygen.bat
View File

@ -4,6 +4,6 @@ rem Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
rem or more contributor license agreements. Licensed under the Elastic License;
rem you may not use this file except in compliance with the Elastic License.
PUSHD %~dp0
CALL %~dp0.in.bat org.elasticsearch.shield.crypto.tool.SystemKeyTool %*
PUSHD "%~dp0"
CALL "%~dp0.in.bat" org.elasticsearch.shield.crypto.tool.SystemKeyTool %*
POPD

69
shield/config/shield/roles.yml
View File

@ -1,33 +1,29 @@
admin:
cluster: all
indices:
'*': all
'*':
privileges: all
# monitoring cluster privileges
# All operations on all indices
power_user:
cluster: monitor
indices:
'*': all
'*':
privileges: all
# Read-only operations on indices
user:
indices:
'*': read
'*':
privileges: read
# Defines the required permissions for transport clients
transport_client:
cluster:
- cluster:monitor/nodes/info
#uncomment the following for sniffing
#- cluster:monitor/state
# The required role for kibana 3 users
kibana3:
cluster: cluster:monitor/nodes/info
indices:
'*': indices:data/read/search, indices:data/read/get, indices:admin/get
'kibana-int': indices:data/read/search, indices:data/read/get, indices:data/write/delete, indices:data/write/index, create_index
- cluster:monitor/nodes/liveness
#uncomment the following for sniffing
#- cluster:monitor/state
# The required permissions for kibana 4 users.
kibana4:
@ -36,24 +32,9 @@ kibana4:
- cluster:monitor/health
indices:
'*':
- indices:admin/mappings/fields/get
- indices:admin/validate/query
- indices:data/read/search
- indices:data/read/msearch
- indices:admin/get
privileges: indices:admin/mappings/fields/get, indices:admin/validate/query, indices:data/read/search, indices:data/read/msearch, indices:admin/get
'.kibana':
- indices:admin/exists
- indices:admin/mapping/put
- indices:admin/mappings/fields/get
- indices:admin/refresh
- indices:admin/validate/query
- indices:data/read/get
- indices:data/read/mget
- indices:data/read/search
- indices:data/write/delete
- indices:data/write/index
- indices:data/write/update
- indices:admin/create
privileges: indices:admin/exists, indices:admin/mapping/put, indices:admin/mappings/fields/get, indices:admin/refresh, indices:admin/validate/query, indices:data/read/get, indices:data/read/mget, indices:data/read/search, indices:data/write/delete, indices:data/write/index, indices:data/write/update, indices:admin/create
# The required permissions for the kibana 4 server
kibana4_server:
@ -62,33 +43,11 @@ kibana4_server:
- cluster:monitor/health
indices:
'.kibana':
- indices:admin/exists
- indices:admin/mapping/put
- indices:admin/mappings/fields/get
- indices:admin/refresh
- indices:admin/validate/query
- indices:data/read/get
- indices:data/read/mget
- indices:data/read/search
- indices:data/write/delete
- indices:data/write/index
- indices:data/write/update
privileges: indices:admin/exists, indices:admin/mapping/put, indices:admin/mappings/fields/get, indices:admin/refresh, indices:admin/validate/query, indices:data/read/get, indices:data/read/mget, indices:data/read/search, indices:data/write/delete, indices:data/write/index, indices:data/write/update
# The required role for logstash users
logstash:
cluster: indices:admin/template/get, indices:admin/template/put
indices:
'logstash-*': indices:data/write/bulk, indices:data/write/delete, indices:data/write/update, indices:data/read/search, indices:data/read/scroll, create_index
# Marvel role, allowing all operations
# on the marvel indices
marvel_user:
cluster: cluster:monitor/nodes/info, cluster:admin/plugin/license/get
indices:
'.marvel-*': all
# Marvel Agent users
marvel_agent:
cluster: indices:admin/template/get, indices:admin/template/put
indices:
'.marvel-*': indices:data/write/bulk, create_index
'logstash-*':
privileges: indices:data/write/bulk, indices:data/write/delete, indices:data/write/update, indices:data/read/search, indices:data/read/scroll, create_index

26
shield/docs/public/configuring-clients-integrations/java.asciidoc
View File

@ -15,7 +15,7 @@ To use the transport client with Shield, you need to:
`transport_client` role is defined in `roles.yml` that grants the `cluster:monitor/nodes/info` cluster permission. The transport client uses the node info API to fetch information about the nodes in the cluster. If the client is configured to use sniffing, you need to add the
`cluster:monitor/state` cluster permission to the `transport_client` role.
. Add the Shield JAR file (`shield-2.1.0.jar`) to your CLASSPATH. You can download the Shield distribution and extract the JAR file manually or you can get it from the http://maven.elasticsearch.org/releases/org/elasticsearch/plugin/shield/{version}/shield-{version}.jar[Elasticsearch Maven repository].
. Add the Shield JAR file (`shield-2.1.0.jar`) to your CLASSPATH. You can download the Shield distribution and extract the JAR file manually or you can get it from the https://maven.elasticsearch.org/releases/org/elasticsearch/plugin/shield/{version}/shield-{version}.jar[Elasticsearch Maven repository].
+
If you are using Maven, you need to add the Shield JAR file as a dependency in your project's `pom.xml` file:
+
@ -27,7 +27,7 @@ If you are using Maven, you need to add the Shield JAR file as a dependency in y
<!-- add the elasticsearch repo -->
<repository>
<id>elasticsearch-releases</id>
<url>http://maven.elasticsearch.org/releases</url>
<url>https://maven.elasticsearch.org/releases</url>
<releases>
<enabled>true</enabled>
</releases>
@ -62,7 +62,7 @@ repositories {
// Add the Elasticsearch Maven Repository
maven {
url "http://maven.elasticsearch.org/releases"
url "https://maven.elasticsearch.org/releases"
}
}
@ -80,14 +80,14 @@ dependencies {
[source,java]
-------------------------------------------------------------------------------------------------
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.shield.ShieldPlugin
...
TransportClient client = TransportClient.builder()
.addPlugin(ShieldPlugin.class)
.settings(Settings.builder()
.put("cluster.name", "myClusterName")
.put("shield.user", "transport_client_user:changeme")
.put("plugin.types", "org.elasticsearch.shield.ShieldPlugin")
.put("path.home", "/path/to/client/home")
...
.build())
.addTransportAddress(new InetSocketTransportAddress("localhost", 9300))
@ -106,16 +106,16 @@ import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.common.settings.Settings;
import org.elasticsearch.common.transport.InetSocketTransportAddress;
import org.elasticsearch.shield.authc.support.SecuredString;
import org.elasticsearch.shield.ShieldPlugin
import static org.elasticsearch.shield.authc.support.UsernamePasswordToken.basicAuthHeaderValue;
...
TransportClient client = TransportClient.builder()
.addPlugin(ShieldPlugin.class)
.settings(Settings.builder()
.put("cluster.name", "myClusterName")
.put("shield.user", "transport_client_user:changeme")
.put("plugin.types", "org.elasticsearch.shield.ShieldPlugin")
.put("path.home", "/path/to/client/home")
...
.build())
.build()
@ -137,16 +137,16 @@ NOTE: Client authentication is enabled by default. For information about disabli
[source,java]
--------------------------------------------------------------------------------------------------
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.shield.ShieldPlugin
...
TransportClient client = TransportClient.builder()
.addPlugin(ShieldPlugin.class)
.settings(Settings.builder()
.put("cluster.name", "myClusterName")
.put("plugin.types", "org.elasticsearch.shield.ShieldPlugin")
.put("shield.user", "transport_client_user:changeme")
.put("shield.ssl.keystore.path", "/path/to/client.jks") (1)
.put("shield.ssl.keystore.password", "password")
.put("path.home", "/path/to/client/home")
...
.build());
--------------------------------------------------------------------------------------------------
@ -158,17 +158,17 @@ TransportClient client = TransportClient.builder()
[source,java]
--------------------------------------------------------------------------------------------------
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.shield.ShieldPlugin
...
TransportClient client = TransportClient.builder()
.addPlugin(ShieldPlugin.class)
.settings(Settings.builder()
.put("cluster.name", "myClusterName")
.put("shield.user", "transport_client_user:changeme")
.put("shield.ssl.keystore.path", "/path/to/client.jks") (1)
.put("shield.ssl.keystore.password", "password")
.put("shield.transport.ssl", "true")
.put("plugin.types", "org.elasticsearch.shield.ShieldPlugin")
.put("path.home", "/path/to/client/home")
...
.build())
.addTransportAddress(new InetSocketTransportAddress("localhost", 9300))
@ -186,17 +186,17 @@ If you are not using client authentication and sign the Elasticsearch node certi
[source,java]
------------------------------------------------------------------------------------------------------
import org.elasticsearch.client.transport.TransportClient;
import org.elasticsearch.shield.ShieldPlugin
...
TransportClient client = TransportClient.builder()
.addPlugin(ShieldPlugin.class)
.settings(Settings.builder()
.put("cluster.name", "myClusterName")
.put("shield.user", "test_user:changeme")
.put("shield.ssl.truststore.path", "/path/to/truststore.jks") (1)
.put("shield.ssl.truststore.password", "password")
.put("shield.transport.ssl", "true")
.put("plugin.types", "org.elasticsearch.shield.ShieldPlugin")
.put("path.home", "/path/to/client/home")
...
.build())
.addTransportAddress(new InetSocketTransportAddress("localhost", 9300))

20
shield/docs/public/configuring-clients-integrations/kibana.asciidoc
View File

@ -148,8 +148,8 @@ file, `/config/kibana.yml`.
+
[source,yaml]
--------------------------------------------------------------------------------
kibana_elasticsearch_username: kibana4-server
kibana_elasticsearch_password: password
elasticsearch.username: kibana4-server
elasticsearch.password: password
--------------------------------------------------------------------------------
[[kibana4-roles]]
@ -229,31 +229,31 @@ kibana4_monitoring:
. If you have enabled SSL encryption in Shield, configure Kibana 4 to connect to Elasticsearch via HTTPS. To do this:
.. Specify the HTTPS protocol in the `elasticsearch` URL setting in the Kibana configuration file, `kibana.yml`:
.. Specify the HTTPS protocol in the `elasticsearch.url` setting in the Kibana configuration file, `kibana.yml`:
+
[source,yaml]
--------------------------------------------------------------------------------
elasticsearch: "https://<your_elasticsearch_host>.com:9200"
elasticsearch.url: "https://<your_elasticsearch_host>.com:9200"
--------------------------------------------------------------------------------
.. If you are using your own CA to sign certificates for Elasticsearch, set the `ca` property in `kibana.yml` to specify the location of the PEM file.
.. If you are using your own CA to sign certificates for Elasticsearch, set the `elasticsearch.ssl.ca` setting in `kibana.yml` to specify the location of the PEM file.
+
[source,yaml]
--------------------------------------------------------------------------------
ca: /path/to/your/ca/cacert.pem
elasticsearch.ssl.ca: /path/to/your/cacert.pem
--------------------------------------------------------------------------------
. Configure Kibana 4 to encrypt communications between the browser and the Kibana server. To do this, configure the `ssl_key_file` and `ssl_cert_file` properties in `kibana.yml` and restart Kibana:
. Configure Kibana 4 to encrypt communications between the browser and the Kibana server. To do this, configure the `server.ssl.key` and `server.ssl.cert` properties in `kibana.yml`:
+
[source,yaml]
--------------------------------------------------------------------------------
ssl_key_file: /path/to/your/server.key
ssl_cert_file: /path/to/your/server.crt
server.ssl.key: /path/to/your/server.key
server.ssl.cert: /path/to/your/server.crt
--------------------------------------------------------------------------------
+
Once you enable SSL encryption between the browser and the Kibana server, access Kibana via HTTPS. For example, `https://localhost:5601`.
+
NOTE: Enabling browser encryption is required to prevent passing user credentials in the clear in Kibana 4.0 and 4.1.
NOTE: Enabling browser encryption is required to prevent passing user credentials in the clear.
. Restart Kibana and verify that you can sign in as a user. If you are running Kibana locally,
go to `localhost:5601` and enter the credentials for a user you've assigned a Kibana user role. For example, you could log in as the `jacknich` user created in step 3.

33
shield/docs/public/configuring-rbac.asciidoc
View File

@ -47,24 +47,28 @@ The following snippet shows an example configuration:
admin:
cluster: all
indices:
'*': all
'*':
privileges: all
# Monitoring cluster privileges
# All operations on all indices
power_user:
cluster: monitor
indices:
'*': all
'*':
privileges: all
# Only read operations on indices
user:
indices:
'*': read
'*':
privileges: read
# Only read operations on indices named events_*
events_user:
indices:
'events_*': read
'events_*':
privileges: read
-----------------------------------
[[valid-role-name]]
@ -100,7 +104,8 @@ The role in the following example allows access to document `GET` actions for a
# Only GET read action on index named events_index
get_user:
indices:
'events_index': 'indices:data/read/get'
'events_index':
privileges: 'indices:data/read/get'
---------------------------------------------------
See the complete list of available <<ref-actions-list, cluster and indices actions>>.
@ -129,6 +134,22 @@ TIP: Once the roles are defined, users can then be associated with any number of
<<setting-up-authentication,Setting Up Authentication>> we'll learn more about authentication and see how users can be associated with the
configured roles.
The privileges can also directly be set on an index expression. This notation is useful if no other security features
are configured.
.Shorter privileges notation
[source,yaml]
---------------------------------------------------
# Only GET read action on index named events_index
get_user:
indices:
'events_index': 'indices:data/read/get'
---------------------------------------------------
include::granting-alias-privileges.asciidoc[]
include::mapping-roles.asciidoc[]
include::mapping-roles.asciidoc[]
include::setting-up-field-and-document-level-security.asciidoc[]
include::submitting-requests-for-other-users.asciidoc[]

15
shield/docs/public/getting-started.asciidoc
View File

@ -4,8 +4,9 @@
This getting started guide walks you through installing Shield, setting up basic authentication, and getting started with role-based
access control. You can install Shield on nodes running Elasticsearch {version}.
IMPORTANT: The Shield plugin must be installed on every node in the cluster and every
node must be restarted after installation. Plan for a complete cluster restart before beginning the installation process.
IMPORTANT: The Shield plugin must be installed on every node in the cluster. If you are installing
to a live cluster, you must stop all of the nodes, install Shield, and restart the nodes. You cannot
perform a rolling restart to install Shield.
To install and run Shield:
@ -32,7 +33,14 @@ NOTE: If you are using a <<deb-rpm-install, DEB/RPM distribution>> of Elasticsea
bin/elasticsearch
----------------------------------------------------------
. To verify that Shield is up and running, check the startup log entries. When Shield is operating normally, the log indicates that the network transports are using Shield:
. To verify that Shield is up and running, use the `_shield` API to get the Shield version:
+
[source,shell]
----------------------------------------------------------
curl -u es_admin -XGET 'http://localhost:9200/_shield'
----------------------------------------------------------
+
You can also check the startup log entries. When Shield is operating normally, the log indicates that the network transports are using Shield:
+
[source,shell]
----------------
@ -41,6 +49,7 @@ bin/elasticsearch
[2014-10-09 13:47:38,842][INFO ][http ] [Ezekiel Stane] Using [org.elasticsearch.shield.transport.netty.ShieldNettyHttpServerTransport] as http transport, overridden by [shield]
----------------
Now you're ready to secure your cluster! Here are a few things
you might want to do to start with:

12
shield/docs/public/limitations.asciidoc
View File

@ -62,17 +62,21 @@ when computing the aggregation results.
[[limitations-disable-cache]]
[float]
==== Elasticsearch 1.6+
Elasticsearch 1.6 removes all of the limitations below with queries and filters, *but* there is the possibility of
authorization being bypassed when using a terms filter with the
Elasticsearch 1.6 removes all of the limitations below with queries and filters with the exception of the {ref}/query-dsl-mlt-query.html[More Like This Query].
*But* there is the possibility of authorization being bypassed when using a terms filter with the
{ref}/query-dsl-terms-filter.html#_terms_lookup_mechanism[terms lookup mechanism]. The authorization that could be
bypassed is for the index containing the terms. In order to ensure that all requests are properly authorized when using
Shield 1.2.0 and 1.2.1, add the following setting to your `elasticsearch.yml` file:
bypassed is for the index containing the terms when using Shield 1.2.0 and 1.2.1. If using Shield 1.2.0 or 1.2.1,
add the following setting to your `elasticsearch.yml` file to ensure that requests are properly authorized:
[source,yaml]
--------------------------------------------------
indices.cache.filter.terms.size: 0
--------------------------------------------------
Shield 1.2.2+ does not require this setting to be specified.
Elasticsearch 1.6.2+ and 1.7.1+ remove the limitations on the {ref}/query-dsl-mlt-query.html[More Like This Query].
[float]
==== Elasticsearch pre-1.6.0
Certain Elasticsearch requests execute other requests as part of their implementation. Some of these requests do not

1
shield/docs/public/managing-users.asciidoc
View File

@ -1,3 +1,4 @@
[[managing-users]]
== Managing Users in an esusers Realm
The `esusers` command line tool is located in `ES_HOME/bin/shield` and enables several

85
shield/docs/public/mapping-roles.asciidoc
View File

@ -3,75 +3,54 @@
If you authenticate users with an `esusers` realm, you can assign roles when you <<esusers-add,add a user>> and use the <<esusers-roles,`roles`>> command to add or remove roles.
For other types of realms, you configure role mappings for users and groups in a YAML file. By default, role mappings are stored in `config/shield/role_mapping.yml`. You can specify
the name and location of the mapping file by configuring the appropriate `role_mapping` setting in `elasticsearch.yml`:
For other types of realms, you configure role mappings for users and groups in a YAML file
and copy it to each node in the cluster. Tools like Puppet or Chef can help with this.
`shield.authc.ldap.files.role_mapping` :: The location of the role mapping file for LDAP realms.
`shield.authc.active_directory.files.role_mapping` :: The location of the role mapping file for Active Directory realms.
`shield.authc.pki.files.role_mapping` :: The location of the role mapping file for PKI realms.
By default, role mappings are stored in `CONF_DIR/shield/users/role_mapping.yml`, where `CONF_DIR`
is `ES_HOME/config` (zip/tar installations) or `/etc/elasticsearch` (package installations).
To specify a different location, you configure the `role_mapping` settings in `elasticsearch.yml`.
The `role_mapping` settings enable you to use a different set of mappings for each realm type:
Within the role mapping file, Elasticsearch roles are keys and groups
and users are values. The mapping can have a many-to-many relationship.
When you map roles to groups, the roles of a user in that group are the combination of the
roles assigned to that group and the roles assigned to that user.
`shield.authc.ldap.files.role_mapping` :: The location of the role mappings for LDAP realms.
`shield.authc.active_directory.files.role_mapping` :: The location of the role mappings for Active Directory realms.
`shield.authc.pki.files.role_mapping` :: The location of the role mappings for PKI realms.
To map users and groups to a role, you create a mapping file and copy it to each node in the cluster. Tools like Puppet or Chef can help with this.
[[ldap-role-mapping]]
.Example LDAP Role Mapping File
[source, yaml]
------------------------------------------------------------
# Example LDAP group mapping configuration:
# roleA: <1>
# - groupA-DN <2>
# - groupB-DN
# - user1-DN <3>
monitoring:
- "cn=admins,dc=example,dc=com"
user:
- "cn=users,dc=example,dc=com"
- "cn=admins,dc=example,dc=com"
- "cn=John Doe,cn=contractors,dc=example,dc=com"
------------------------------------------------------------
<1> The name of the Elasticsearch role found in the <<defining-roles, roles file>>
<2> Example specifying the distinguished name of a LDAP group
<3> Example specifying the distinguished name of a LDAP user added[1.1.0]
IMPORTANT: For Shield to read the mapping file, it must be stored in the Elasticsearch `CONF_DIR`.
Within the role mapping file, Shield roles are keys and groups and users are values.
The mappings can have a many-to-many relationship. When you map roles to groups, the roles of a
user in that group are the combination of the roles assigned to that group and the roles assigned
to that user.
[[ad-role-mapping]]
.Example Active Directory Role Mapping File
The available roles are defined in the <<defining-roles, roles file>>. To specify users and
groups in the role mappings, you use their _Distinguished Names_ (DNs). A DN
is a string that uniquely identifies the user or group, for example
`"cn=John Doe,cn=contractors,dc=example,dc=com"`.
[[ldap-role-mapping]]
LDAP and Active Directory realms support mapping both users and groups to roles. For example:
[source, yaml]
------------------------------------------------------------
# Example Active Directory group mapping configuration:
# roleA: <1>
# - groupA-DN <2>
# - groupB-DN
# - user1-DN <3>
monitoring:
- "cn=admins,dc=example,dc=com"
monitoring: <1>
- "cn=admins,dc=example,dc=com" <2>
user:
- "cn=John Doe,cn=contractors,dc=example,dc=com" <3>
- "cn=users,dc=example,dc=com"
- "cn=admins,dc=example,dc=com"
- "cn=John Doe,cn=contractors,dc=example,dc=com"
------------------------------------------------------------
<1> The name of a Shield role defined in the <<defining-roles, roles file>>
<2> Example specifying the distinguished name of a Active Directory group
<3> Example specifying the distinguished name of a Active Directory user
<1> The name of a Shield role defined in the <<defining-roles, roles file>>.
<2> The distinguished name of an LDAP or Active Directory group.
<3> The distinguished name of an LDAP or Active Directory user. added[1.1.0]
[[pki-role-mapping]]
.Example PKI Role Mapping File
PKI realms only support mapping users to roles, as there is no notion of a group in PKI. For example:
[source, yaml]
------------------------------------------------------------
# Example user mapping configuration:
# roleA: <1>
# - user1-DN <2>
monitoring:
- "cn=Admin,ou=example,o=com"
monitoring:
- "cn=Admin,ou=example,o=com"
user:
- "cn=John Doe,ou=example,o=com"
------------------------------------------------------------
<1> The name of a Shield role defined in the <<defining-roles, roles file>>
<2> Example specifying the distinguished name of a PKI user
NOTE: For PKI realms, only the DN of a user can be mapped as there is no concept of a group in PKI

Some files were not shown because too many files have changed in this diff Show More